From 5eb18c3799bfbc6911b88e21732cee070687905c Mon Sep 17 00:00:00 2001 From: Anna Urbiztondo Date: Thu, 1 Feb 2024 15:31:31 +0100 Subject: [PATCH 1/6] Edits --- metrics-and-metadata/enablerelatedcontent.rst | 38 ++++++++++++------- metrics-and-metadata/relatedcontent.rst | 10 ++--- 2 files changed, 29 insertions(+), 19 deletions(-) diff --git a/metrics-and-metadata/enablerelatedcontent.rst b/metrics-and-metadata/enablerelatedcontent.rst index 97fc3fa6e..8fae8c03f 100644 --- a/metrics-and-metadata/enablerelatedcontent.rst +++ b/metrics-and-metadata/enablerelatedcontent.rst @@ -6,15 +6,27 @@ Enable Related Content in Splunk Observability Cloud Splunk Observability Cloud uses OpenTelemetry to correlate telemetry types. To enable this ability, your telemetry field names or metadata key names must exactly match the metadata key names used by OpenTelemetry and Splunk Observability Cloud. -When you deploy the Splunk Distribution of OpenTelemetry Collector to send your telemetry data to Observability Cloud, your metadata key names are automatically mapped correctly. If you do not use the Splunk Distribution of OpenTelemetry Collector, your telemetry data might have metadata key names that are not consistent with those used by Observability Cloud and OpenTelemetry, and Related Content might not work. In that case, you must change your metadata key names. +Related Content and the Splunk Distribution of the OpenTelemetry Collector compatibility +============================================================================================ + +When you deploy the Splunk Distribution of OpenTelemetry Collector to send your telemetry data to Splunk Observability Cloud, your metadata key names are automatically mapped correctly. + +.. caution:: If you don't use the Splunk Distribution of OpenTelemetry Collector, or you use a non-default configuration, your telemetry data might have metadata key names that are not consistent with those used by Splunk Observability Cloud and OpenTelemetry, and Related Content might not work. In that case, you must change your metadata key names. + +Example +----------------------------------------------------------------- For example, say Splunk Observability Cloud receives the following telemetry data: -- Splunk APM receives a trace with the metadata key ``trace_id: 2b78e7c951497655`` +- Splunk APM receives a trace with the metadata key ``trace_id:2b78e7c951497655`` - Splunk Log Observer receives a log with the metadata key ``trace.id:2b78e7c951497655`` -Although these refer to the same trace ID value, the log and the trace cannot be correlated in Splunk Observability Cloud because the field names, ``trace_id`` and ``trace.id`` do not match. In this case, rename your log metadata key ``trace.id`` to ``trace_id`` using the field copy processor in Logs Pipeline Management. Alternatively, you can re-instrument your log collection to make metadata key names align. When the field names in APM and Log Observer match, the trace and the log with the same trace ID value can be correlated in Observability Cloud. Then, when you are viewing the trace in APM, you can select directly into the log with the same trace ID value and view the correlated log in Log Observer. +Although these refer to the same trace ID value, the log and the trace cannot be correlated in Splunk Observability Cloud because the field names, ``trace_id`` and ``trace.id`` do not match. + +To solve this, rename your log metadata key ``trace.id`` to ``trace_id`` using the field copy processor in Logs Pipeline Management. Alternatively, you can re-instrument your log collection to make metadata key names align. + +When the field names in APM and Log Observer match, the trace and the log with the same trace ID value can be correlated in Splunk Observability Cloud. Then, when you are viewing the trace in APM, you can select directly into the log with the same trace ID value and view the correlated log in Log Observer. How to change your metadata key names ================================================================= @@ -49,10 +61,10 @@ The Splunk Distribution of OpenTelemetry Collector provides the following APM sp To learn more about deployment environments in Splunk APM, see :ref:`apm-environments`. -Leverage Related Content for pod-specific Kubernetes data ------------------------------------------------------------------ +Leverage Related Content for APM and pod-specific Kubernetes data +----------------------------------------------------------------------------- -For a Related Content tile in APM to link to data for a specific Kubernetes pod (k8s.pod.name), you must first filter on a specific Kubernetes cluster (k8s.cluster.name). APM cannot guarantee an accurate Related Content Kubernetes pod destination in Infrastructure Monitoring without both values because Kubernetes pod names are not required to be unique across clusters. +For a Related Content tile in APM to link to data for a specific Kubernetes pod (``k8s.pod.name``), you must first filter on a specific Kubernetes cluster (``k8s.cluster.name``). APM cannot guarantee an accurate Related Content Kubernetes pod destination in Infrastructure Monitoring without both values because Kubernetes pod names are not required to be unique across clusters. For example, consider a scenario in which Related Content needs to return data for a Kubernetes pod named :strong:`Pod-B`. As shown the following diagram, a Kubernetes implementation can have multiple pods with the same name. For Related Content to return the data for the correct :strong:`Pod-B`, you must also provide the name of the Kubernetes cluster the pod resides in. In this case, that name would be either :strong:`Cluster-West` or :strong:`Cluster-East`. This combination of filtering on cluster and pod names creates the unique combination that Related Content needs to link to the correct pod data in Infrastructure Monitoring. @@ -134,19 +146,15 @@ When to use Log Field Aliasing Use Log Field Aliasing to remap fields in Observability Cloud when you cannot or do not want to create a copy processor because any of the following are true: -- You use Log Observer Connect to get logs data and do not have access to Log Observer Pipeline Management - -- You do not want to use indexing capacity by creating additional log processing rules - -- You do not want to transform your data at index time - +- You use Log Observer Connect to get logs data and do not have access to Log Observer Pipeline Management. +- You do not want to use indexing capacity by creating additional log processing rules. +- You do not want to transform your data at index time. - You want the new alias to affect every log message, even those that came in from a time before you created the alias. - Kubernetes log fields -------------------------------------------------------------------------- -Do not change the following fields, which the Splunk Distribution of OpenTelemetry Collector injects into your Kubernetes logs: +The Splunk Distribution of the OpenTelemetry Collector injects the following fields into your Kubernetes logs: - ``k8s.cluster.name`` - ``k8s.node.name`` @@ -155,6 +163,8 @@ Do not change the following fields, which the Splunk Distribution of OpenTelemet - ``k8s.namespace.name`` - ``kubernetes.workload.name`` +Do not modify them if you want to use Related Content. + Learn more about the Collector for Kubernetes at :ref:`otel-install-k8s`. diff --git a/metrics-and-metadata/relatedcontent.rst b/metrics-and-metadata/relatedcontent.rst index f2c170b18..585ef7dc0 100644 --- a/metrics-and-metadata/relatedcontent.rst +++ b/metrics-and-metadata/relatedcontent.rst @@ -24,14 +24,14 @@ In the preceding example, the user navigates through the following sequence: In the Related Content bar, the user sees a tile showing logs related to the EC2 instance, so they click it. -3. Splunk Observability Cloud takes them to Log Observer where they can drill down into the related logs to find the root cause of the problem. +3. Splunk Observability Cloud takes the user to Log Observer where they can drill down into the related logs to find the root cause of the problem. .. note:: Related Content is different from data links, a separate capability, which lets you dynamically transfer contextual information about the property you're viewing to the resource, helping you get to relevant information faster. To learn more about data links, see :ref:`apm-create-data-links`. Prerequisites ================================================================= -Related Content relies on specific metadata that allow APM, Infrastructure Monitoring, and Log Observer to pass filters around Observability Cloud. +Related Content relies on specific metadata that allow APM, Infrastructure Monitoring, and Log Observer to pass filters around Splunk Observability Cloud. The following sections list the metadata key names required to enable Related Content for each view in Splunk Observability Cloud. If your data does not have the field names listed here, Splunk Observability Cloud cannot correlate your related data. @@ -60,9 +60,9 @@ The following Infrastructure Monitoring metadata keys are required to enable Rel - ``k8s.namespace.name`` - ``kubernetes.workload.name`` -If you're using the Splunk Distribution of the OpenTelemetry Collector for Kubernetes, the required Infrastructure Monitoring metadata is provided. See more at :ref:`otel-install-k8s`. +If you're using the default configuration of the Splunk Distribution of the OpenTelemetry Collector for Kubernetes, the required Infrastructure Monitoring metadata is provided. See more at :ref:`otel-install-k8s`. -If you're using other configurations to collect infrastructure data, Related Content won't work out of the box. +If you're using other distributions of the OpenTelemetry Collector or non-default configurations of the Splunk Distribution to collect infrastructure data, Related Content won't work out of the box. Log Observer ----------------------------------------------------------------- @@ -80,7 +80,7 @@ The following metadata keys are required to enable Related Content for Log Obser Enable Related Content ================================================================= -See :ref:`get-started-enablerelatedcontent` to learn how you can make any necessary updates to metadata key names to enable Related Content in Observability Cloud. +See :ref:`get-started-enablerelatedcontent` to learn how you can make any necessary updates to metadata key names to enable Related Content in Splunk Observability Cloud. Where can I see Related Content? ================================================================= From 3f2229021e64d88843b5f793894df5d4ba396b50 Mon Sep 17 00:00:00 2001 From: Anna Urbiztondo Date: Mon, 5 Feb 2024 07:50:50 +0100 Subject: [PATCH 2/6] Final edits --- metrics-and-metadata/data-tools-landing.rst | 1 - metrics-and-metadata/enablerelatedcontent.rst | 170 --------------- metrics-and-metadata/relatedcontent.rst | 204 ++++++++++++++---- 3 files changed, 168 insertions(+), 207 deletions(-) delete mode 100644 metrics-and-metadata/enablerelatedcontent.rst diff --git a/metrics-and-metadata/data-tools-landing.rst b/metrics-and-metadata/data-tools-landing.rst index 8e810958f..e28a572dc 100644 --- a/metrics-and-metadata/data-tools-landing.rst +++ b/metrics-and-metadata/data-tools-landing.rst @@ -13,7 +13,6 @@ Data tools in Splunk Observability Cloud Metric finder and metadata catalogue Related Content - Enable Related Content Global data links Splunk Observability Cloud provides a wide array of features and tools to help you manage, understand, and leverage your data: diff --git a/metrics-and-metadata/enablerelatedcontent.rst b/metrics-and-metadata/enablerelatedcontent.rst deleted file mode 100644 index 8fae8c03f..000000000 --- a/metrics-and-metadata/enablerelatedcontent.rst +++ /dev/null @@ -1,170 +0,0 @@ -.. _get-started-enablerelatedcontent: - -***************************************************************** -Enable Related Content in Splunk Observability Cloud -***************************************************************** - -Splunk Observability Cloud uses OpenTelemetry to correlate telemetry types. To enable this ability, your telemetry field names or metadata key names must exactly match the metadata key names used by OpenTelemetry and Splunk Observability Cloud. - -Related Content and the Splunk Distribution of the OpenTelemetry Collector compatibility -============================================================================================ - -When you deploy the Splunk Distribution of OpenTelemetry Collector to send your telemetry data to Splunk Observability Cloud, your metadata key names are automatically mapped correctly. - -.. caution:: If you don't use the Splunk Distribution of OpenTelemetry Collector, or you use a non-default configuration, your telemetry data might have metadata key names that are not consistent with those used by Splunk Observability Cloud and OpenTelemetry, and Related Content might not work. In that case, you must change your metadata key names. - -Example ------------------------------------------------------------------ - -For example, say Splunk Observability Cloud receives the following telemetry data: - -- Splunk APM receives a trace with the metadata key ``trace_id:2b78e7c951497655`` - -- Splunk Log Observer receives a log with the metadata key ``trace.id:2b78e7c951497655`` - -Although these refer to the same trace ID value, the log and the trace cannot be correlated in Splunk Observability Cloud because the field names, ``trace_id`` and ``trace.id`` do not match. - -To solve this, rename your log metadata key ``trace.id`` to ``trace_id`` using the field copy processor in Logs Pipeline Management. Alternatively, you can re-instrument your log collection to make metadata key names align. - -When the field names in APM and Log Observer match, the trace and the log with the same trace ID value can be correlated in Splunk Observability Cloud. Then, when you are viewing the trace in APM, you can select directly into the log with the same trace ID value and view the correlated log in Log Observer. - -How to change your metadata key names -================================================================= - -Change metric and traces names ------------------------------------------------------------------ - -Use the Splunk Distribution of OpenTelemetry Collector to ensure that your metrics and traces have the metadata key names required to use Observability Cloud's Related Content feature. If you did not use the Collector and your metrics or traces do not include the required metadata key names, you can instrument your applications and serverless functions to include them. See the following pages to learn how: - -- :ref:`get-started-application` - -- :ref:`instrument-serverless-functions` - -- :ref:`rum-gdi` - -Change logs names ------------------------------------------------------------------ - -If the required key names use different names in your log fields, remap them using one of the methods listed in :ref:`remap-log-fields`. - -The remainder of this page provides details on the required metadata fields for each view in Observability Cloud. - -Splunk APM -================================================================= - -To ensure full functionality of Related Content, do not change any of the metadata key names or span tags provided by the Splunk Distribution of OpenTelemetry Collector. To learn more about span tags in Splunk APM, see :ref:`apm-traces-spans`. - -The Splunk Distribution of OpenTelemetry Collector provides the following APM span tags that enable Related Content: - -- ``service.name`` -- ``deployment.environment`` - -To learn more about deployment environments in Splunk APM, see :ref:`apm-environments`. - -Leverage Related Content for APM and pod-specific Kubernetes data ------------------------------------------------------------------------------ - -For a Related Content tile in APM to link to data for a specific Kubernetes pod (``k8s.pod.name``), you must first filter on a specific Kubernetes cluster (``k8s.cluster.name``). APM cannot guarantee an accurate Related Content Kubernetes pod destination in Infrastructure Monitoring without both values because Kubernetes pod names are not required to be unique across clusters. - -For example, consider a scenario in which Related Content needs to return data for a Kubernetes pod named :strong:`Pod-B`. As shown the following diagram, a Kubernetes implementation can have multiple pods with the same name. For Related Content to return the data for the correct :strong:`Pod-B`, you must also provide the name of the Kubernetes cluster the pod resides in. In this case, that name would be either :strong:`Cluster-West` or :strong:`Cluster-East`. This combination of filtering on cluster and pod names creates the unique combination that Related Content needs to link to the correct pod data in Infrastructure Monitoring. - -.. source in figma: https://www.figma.com/file/sOEa3q92WJxB4uWb3Poftg/related-content-apm-k8s-constraint?node-id=0%3A1 - -.. image:: /_images/get-started/k8s-clusters-pods.png - :width: 80% - :alt: This diagram shows two uniquely named Kubernetes clusters, each containing pods that share names across clusters. - -.. _enablerelatedcontent-imm: - -Splunk Infrastructure Monitoring -================================================================= - -To ensure full functionality of Related Content, do not change any of the metadata key names provided by the Splunk Distribution of OpenTelemetry Collector. - -The Splunk Distribution of OpenTelemetry Collector provides the following Infrastructure Monitoring metadata keys that enable Related Content: - -- ``host.name`` -- ``k8s.cluster.name`` -- ``k8s.node.name`` -- ``k8s.pod.name`` -- ``container.id`` -- ``k8s.namespace.name`` -- ``kubernetes.workload.name`` - -.. _relatedcontent-log-observer: - -Splunk Log Observer -================================================================= - -To ensure full functionality of both Log Observer and Related Content, confirm that your log events fields are correctly mapped. Correct log field mappings enable built-in log filtering, embed logs in APM and -Infrastructure Monitoring functionality, and enable fast searches as well as the Related Content bar. - -The following key names are required to enable Related Content for Log Observer: - -- ``service.name`` -- ``deployment.environment`` -- ``host.name`` -- ``trace_id`` -- ``span_id`` - -If the key names in the preceding list use different names in your log fields, remap them to the key names listed here. For example, if you do not see values for :strong:`host.name` in the Log Observer UI, check to see whether your logs use a different field name, such as :strong:`host_name`. If your logs do not contain the default field names exactly as they appear in the preceding list, remap your logs using one of the methods in the following section. - -.. include:: /_includes/log-observer-transition.rst - -.. _remap-log-fields: - -Methods of remapping log fields --------------------------------------------------------------------------- - -The following table describes the four methods for remapping log fields: - -.. list-table:: - :header-rows: 1 - :widths: 50 50 - - * - :strong:`Remapping Method` - - :strong:`Instructions` - - * - Splunk Observability Cloud Logs Pipeline Management - - Create and apply a field copy processor. See the - :strong:`Field copy processors` section in - :ref:`logs-processors` to learn how. - Note: Only customers with a Splunk Log Observer entitlement in Splunk Observability Cloud can use this method. If you are using Log Observer Connect, use one of other methods in this table. - - * - Log Field Aliasing - - Create and activate a field alias. See :ref:`logs-alias` to learn how. Learn when to use Log Field Aliasing in the next section. - - * - Client-side - - Configure your app to remap the necessary fields. - - * - Collector-side - - Use a Fluentd or FluentBit configuration. See - :ref:`Configure Fluentd to send logs ` to learn how. - -When to use Log Field Aliasing -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -Use Log Field Aliasing to remap fields in Observability Cloud when you cannot or do not want to create a copy processor because any of the following are true: - -- You use Log Observer Connect to get logs data and do not have access to Log Observer Pipeline Management. -- You do not want to use indexing capacity by creating additional log processing rules. -- You do not want to transform your data at index time. -- You want the new alias to affect every log message, even those that came in from a time before you created the alias. - -Kubernetes log fields --------------------------------------------------------------------------- - -The Splunk Distribution of the OpenTelemetry Collector injects the following fields into your Kubernetes logs: - -- ``k8s.cluster.name`` -- ``k8s.node.name`` -- ``k8s.pod.name`` -- ``container.id`` -- ``k8s.namespace.name`` -- ``kubernetes.workload.name`` - -Do not modify them if you want to use Related Content. - -Learn more about the Collector for Kubernetes at :ref:`otel-install-k8s`. - - diff --git a/metrics-and-metadata/relatedcontent.rst b/metrics-and-metadata/relatedcontent.rst index 585ef7dc0..a4d7af403 100644 --- a/metrics-and-metadata/relatedcontent.rst +++ b/metrics-and-metadata/relatedcontent.rst @@ -1,4 +1,5 @@ .. _get-started-relatedcontent: +.. _get-started-enablerelatedcontent: ***************************************************************** Related Content in Splunk Observability Cloud @@ -7,12 +8,16 @@ Related Content in Splunk Observability Cloud .. meta:: :description: Ensure metadata keys are correct to enable full Related Content functionality. -The Related Content feature automatically correlates data between different views within Splunk Observability Cloud by presenting related data at the bottom of the screen. +The Related Content feature automatically correlates and presents data between different views within Splunk Observability Cloud. + +Use Related Content +========================================================================================================== Select tiles in the Related Content bar to seamlessly navigate from one view to another in Splunk Observability Cloud. The following animation shows a user navigating from APM to Infrastructure Monitoring to Log Observer. .. image:: /_images/get-started/Related1.gif :alt: Using Related Content in Splunk Observability Cloud. + :width: 90% In the preceding example, the user navigates through the following sequence: @@ -26,28 +31,106 @@ In the preceding example, the user navigates through the following sequence: 3. Splunk Observability Cloud takes the user to Log Observer where they can drill down into the related logs to find the root cause of the problem. -.. note:: Related Content is different from data links, a separate capability, which lets you dynamically transfer contextual information about the property you're viewing to the resource, helping you get to relevant information faster. To learn more about data links, see :ref:`apm-create-data-links`. +.. note:: Related Content is different from data links, a separate capability, which lets you dynamically transfer contextual information about the property you're viewing to the resource, helping you get to relevant information faster. To learn more about data links, see :ref:`apm-create-data-links`. + +Where can I see Related Content? +----------------------------------------------------------------- + +The following table describes when and where in Splunk Observability Cloud you can see Related Content: + +.. list-table:: + :header-rows: 1 + :widths: 50, 50 + + * - :strong:`Starting Point` + - :strong:`Possible Destinations` + + * - APM services + - Related Kubernetes clusters filtered by service, AWS EC2s, GCP GCEs, Azure VMs, all log lines for the service + + * - Database service + - Related database host or instance + + * - Database instance + - Related Database Query Performance, related APM services + + * - Host or Cloud compute instance (AWS EC2, GCP GCE, Azure VM) + - Related APM services, log lines for the specific instance + + * - Kubernetes cluster, node, pod, container + - Related log lines for the node + + * - Kubernetes pod or container + - Related APM service in that pod or container, log lines for that pod or container + + * - Specific log line + - Related APM service, trace, Kubernetes node/pod/container, Host or compute instance (AWS EC2, GCP GCE, Azure VM) + + * - Specific trace ID + - Related log line + +Related Content and the Splunk Distribution of the OpenTelemetry Collector metadata compatibility +========================================================================================================== + +Splunk Observability Cloud uses OpenTelemetry to correlate telemetry types. To enable this ability, your telemetry field names or metadata key names must exactly match the metadata key names used by both OpenTelemetry and Splunk Observability Cloud. + +When you deploy the Splunk Distribution of the OpenTelemetry Collector to send your telemetry data to Splunk Observability Cloud, your metadata key names are automatically mapped correctly. + +.. caution:: If you don't use the Splunk Distribution of OpenTelemetry Collector, or you use a non-default configuration, your telemetry data might have metadata key names that are not consistent with those used by Splunk Observability Cloud and OpenTelemetry, and Related Content might not work. In that case, you must change your metadata key names. + +Metadata compatibility example +----------------------------------------------------------------- + +For example, say Splunk Observability Cloud receives the following telemetry data: -Prerequisites +- Splunk APM receives a trace with the metadata key ``trace_id:2b78e7c951497655`` + +- Splunk Log Observer receives a log with the metadata key ``trace.id:2b78e7c951497655`` + +Although these refer to the same trace ID value, the log and the trace cannot be correlated in Splunk Observability Cloud because the field names, ``trace_id`` and ``trace.id`` do not match. + +To solve this, rename your log metadata key ``trace.id`` to ``trace_id`` using the field copy processor in Logs Pipeline Management. Alternatively, you can re-instrument your log collection to make metadata key names align. + +When the field names in APM and Log Observer match, the trace and the log with the same trace ID value can be correlated in Splunk Observability Cloud. Then, when you are viewing the trace in APM, you can select directly into the log with the same trace ID value and view the correlated log in Log Observer. + +Prerequisites and required metadata fields ================================================================= Related Content relies on specific metadata that allow APM, Infrastructure Monitoring, and Log Observer to pass filters around Splunk Observability Cloud. The following sections list the metadata key names required to enable Related Content for each view in Splunk Observability Cloud. If your data does not have the field names listed here, Splunk Observability Cloud cannot correlate your related data. -APM +Splunk APM ----------------------------------------------------------------- The following APM span tags are required to enable Related Content: - ``service.name`` -- ``deployment.environment`` +- ``deployment.environment`` + +The default configuration of the Splunk Distribution of the OpenTelemetry Collector already provides these span tags. To ensure full functionality of Related Content, do not change any of the metadata key names or span tags provided by the Splunk OTel Collector. -The Splunk Distribution of OpenTelemetry Collector already provides the span tags. +Learn more: -To learn more about deployment environments in Splunk APM, see :ref:`apm-environments`. +* About span tags in Splunk APM, see :ref:`apm-traces-spans`. +* About deployment environments in Splunk APM, see :ref:`apm-environments`. -Infrastructure Monitoring +Leverage Related Content for APM and pod-specific Kubernetes data +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +For a Related Content tile in APM to link to data for a specific Kubernetes pod (``k8s.pod.name``), you must first filter on a specific Kubernetes cluster (``k8s.cluster.name``). APM cannot guarantee an accurate Related Content Kubernetes pod destination in Infrastructure Monitoring without both values because Kubernetes pod names are not required to be unique across clusters. + +For example, consider a scenario in which Related Content needs to return data for a Kubernetes pod named :strong:`Pod-B`. As shown the following diagram, a Kubernetes implementation can have multiple pods with the same name. For Related Content to return the data for the correct :strong:`Pod-B`, you must also provide the name of the Kubernetes cluster the pod resides in. In this case, that name would be either :strong:`Cluster-West` or :strong:`Cluster-East`. This combination of filtering on cluster and pod names creates the unique combination that Related Content needs to link to the correct pod data in Infrastructure Monitoring. + +.. source in figma: https://www.figma.com/file/sOEa3q92WJxB4uWb3Poftg/related-content-apm-k8s-constraint?node-id=0%3A1 + +.. image:: /_images/get-started/k8s-clusters-pods.png + :width: 80% + :alt: This diagram shows two uniquely named Kubernetes clusters, each containing pods that share names across clusters. + +.. _enablerelatedcontent-imm: + +Splunk Infrastructure Monitoring ----------------------------------------------------------------- The following Infrastructure Monitoring metadata keys are required to enable Related Content: @@ -64,10 +147,14 @@ If you're using the default configuration of the Splunk Distribution of the Open If you're using other distributions of the OpenTelemetry Collector or non-default configurations of the Splunk Distribution to collect infrastructure data, Related Content won't work out of the box. -Log Observer +.. _relatedcontent-log-observer: + +Splunk Log Observer ----------------------------------------------------------------- -The following metadata keys are required to enable Related Content for Log Observer: +.. include:: /_includes/log-observer-transition.rst + +The following key names are required to enable Related Content for Log Observer: - ``service.name`` - ``deployment.environment`` @@ -75,45 +162,90 @@ The following metadata keys are required to enable Related Content for Log Obser - ``trace_id`` - ``span_id`` -.. include:: /_includes/log-observer-transition.rst +To ensure full functionality of both Log Observer and Related Content, verify that your log events fields are correctly mapped. Correct log field mappings enable built-in log filtering, embed logs in APM and Infrastructure Monitoring functionality, and enable fast searches as well as the Related Content bar. -Enable Related Content -================================================================= +If the key names in the preceding list use different names in your log fields, remap them to the key names listed here. For example, if you don't see values for :strong:`host.name` in the Log Observer UI, check to see whether your logs use a different field name, such as :strong:`host_name`. If your logs do not contain the default field names exactly as they appear in the preceding list, remap your logs using one of the methods in the following section. -See :ref:`get-started-enablerelatedcontent` to learn how you can make any necessary updates to metadata key names to enable Related Content in Splunk Observability Cloud. +.. _remap-log-fields: -Where can I see Related Content? -================================================================= +Remap log fields +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ -The following table describes when and where in Splunk Observability Cloud you can see Related Content: +The following table describes the four methods for remapping log fields: .. list-table:: :header-rows: 1 - :widths: 50, 50 + :widths: 50 50 + + * - :strong:`Remapping Method` + - :strong:`Instructions` + + * - Splunk Observability Cloud Logs Pipeline Management + - Create and apply a field copy processor. See the + :strong:`Field copy processors` section in + :ref:`logs-processors` to learn how. + Note: Only customers with a Splunk Log Observer entitlement in Splunk Observability Cloud can use this method. If you are using Log Observer Connect, use one of other methods in this table. + + * - Log Field Aliasing + - Create and activate a field alias. See :ref:`logs-alias` to learn how. Learn when to use Log Field Aliasing in the next section. + + * - Client-side + - Configure your app to remap the necessary fields. + + * - Collector-side + - Use a Fluentd or FluentBit configuration. See + :ref:`Configure Fluentd to send logs ` to learn how. + +When to use Log Field Aliasing +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Use Log Field Aliasing to remap fields in Observability Cloud when you cannot or do not want to create a copy processor because any of the following are true: + +- You use Log Observer Connect to get logs data and do not have access to Log Observer Pipeline Management. +- You do not want to use indexing capacity by creating additional log processing rules. +- You do not want to transform your data at index time. +- You want the new alias to affect every log message, even those that came in from a time before you created the alias. + +Kubernetes log fields +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +The Splunk Distribution of the OpenTelemetry Collector injects the following fields into your Kubernetes logs. Do not modify them if you want to use Related Content. + +- ``k8s.cluster.name`` +- ``k8s.node.name`` +- ``k8s.pod.name`` +- ``container.id`` +- ``k8s.namespace.name`` +- ``kubernetes.workload.name`` + +Learn more about the Collector for Kubernetes at :ref:`collector-kubernetes-intro`. + +How to change your metadata key names +================================================================= + +Change metric and traces names +----------------------------------------------------------------- + +Use the Splunk Distribution of the OpenTelemetry Collector to ensure that your metrics and traces have the metadata key names required to use Observability Cloud's Related Content feature. If you did not use the Collector and your metrics or traces do not include the required metadata key names, you can instrument your applications and serverless functions to include them. See the following pages to learn how: + +- :ref:`get-started-application` +- :ref:`instrument-serverless-functions` +- :ref:`rum-gdi` + +Change log names +----------------------------------------------------------------- + +If the required key names use different names in your log fields, remap them using one of the methods listed in :ref:`remap-log-fields`. + + + + - * - :strong:`Starting Point` - - :strong:`Possible Destinations` - * - APM services - - Related Kubernetes clusters filtered by service, AWS EC2s, GCP GCEs, Azure VMs, all log lines for the service - * - Database service - - Related database host or instance - * - Database instance - - Related Database Query Performance, related APM services - * - Host or Cloud compute instance (AWS EC2, GCP GCE, Azure VM) - - Related APM services, log lines for the specific instance - * - Kubernetes cluster, node, pod, container - - Related log lines for the node - * - Kubernetes pod or container - - Related APM service in that pod or container, log lines for that pod or container - * - Specific log line - - Related APM service, trace, Kubernetes node/pod/container, Host or compute instance (AWS EC2, GCP GCE, Azure VM) - * - Specific trace ID - - Related log line From 3570b99da06af469a28572818cf971017af85eac Mon Sep 17 00:00:00 2001 From: Anna Urbiztondo Date: Mon, 5 Feb 2024 11:37:44 +0100 Subject: [PATCH 3/6] Sfx exporter --- .../components/signalfx-exporter.rst | 4 +++- metrics-and-metadata/relatedcontent.rst | 24 +++++++++++++------ 2 files changed, 20 insertions(+), 8 deletions(-) diff --git a/gdi/opentelemetry/components/signalfx-exporter.rst b/gdi/opentelemetry/components/signalfx-exporter.rst index a7a540747..744ece271 100644 --- a/gdi/opentelemetry/components/signalfx-exporter.rst +++ b/gdi/opentelemetry/components/signalfx-exporter.rst @@ -11,7 +11,7 @@ SignalFx exporter The SignalFx exporter is a native OTel component that allows the OpenTelemetry Collector to send metrics and events to SignalFx endpoints. The supported pipeline types are ``traces``, ``metrics``, and ``logs``. See :ref:`otel-data-processing` for more information. -.. note:: While the SignalFx Smart Agent has reached End of Support, OTel native components such as the Smart Agent receiver, the SignalFx receiver, and the SignalFx exporter are available and supported. For information on the receivers, see :ref:`smartagent-receiver`: and :ref:`signalfx-receiver`. +While the SignalFx Smart Agent has reached End of Support, OTel native components such as the Smart Agent receiver, the SignalFx receiver, and the SignalFx exporter are available and supported. For information on the receivers, see :ref:`smartagent-receiver`: and :ref:`signalfx-receiver`. Get started ====================== @@ -288,6 +288,8 @@ Translation rules currently allow the following actions: .. include:: /_includes/gdi/default-translation-metrics.rst +.. _signalfx-exporter-settings: + Settings ====================== diff --git a/metrics-and-metadata/relatedcontent.rst b/metrics-and-metadata/relatedcontent.rst index a4d7af403..466e381c0 100644 --- a/metrics-and-metadata/relatedcontent.rst +++ b/metrics-and-metadata/relatedcontent.rst @@ -69,6 +69,8 @@ The following table describes when and where in Splunk Observability Cloud you c * - Specific trace ID - Related log line +.. _relatedcontent-collector: + Related Content and the Splunk Distribution of the OpenTelemetry Collector metadata compatibility ========================================================================================================== @@ -87,13 +89,24 @@ For example, say Splunk Observability Cloud receives the following telemetry dat - Splunk Log Observer receives a log with the metadata key ``trace.id:2b78e7c951497655`` -Although these refer to the same trace ID value, the log and the trace cannot be correlated in Splunk Observability Cloud because the field names, ``trace_id`` and ``trace.id`` do not match. +Although these refer to the same trace ID value, the log and the trace cannot be correlated in Splunk Observability Cloud because the field names, ``trace_id`` and ``trace.id``, do not match. To solve this, rename your log metadata key ``trace.id`` to ``trace_id`` using the field copy processor in Logs Pipeline Management. Alternatively, you can re-instrument your log collection to make metadata key names align. When the field names in APM and Log Observer match, the trace and the log with the same trace ID value can be correlated in Splunk Observability Cloud. Then, when you are viewing the trace in APM, you can select directly into the log with the same trace ID value and view the correlated log in Log Observer. -Prerequisites and required metadata fields +.. _relatedcontent-required-components: + +Required Collector components +================================================================= + +If you're using the Splunk OTel Collector and want to ensure Related Content behaves correctly, verify that the SignalFx exporter is included in your configuration. This exporter aggregates the metrics from the ``hostmetrics`` receiver and must be enabled for the ``metrics`` and ``traces`` pipelines. + +The Collector uses the correlation flag of the SignalFx exporter to make relevant API calls to correlate your spans with the infrastructure metrics. This flag is enabled by default. To adjust the correlation option further, see the SignalFx exporter's options at :ref:`signalfx-exporter-settings`. + +.. _relatedcontent-required-fields: + +Required metadata fields ================================================================= Related Content relies on specific metadata that allow APM, Infrastructure Monitoring, and Log Observer to pass filters around Splunk Observability Cloud. @@ -181,9 +194,7 @@ The following table describes the four methods for remapping log fields: - :strong:`Instructions` * - Splunk Observability Cloud Logs Pipeline Management - - Create and apply a field copy processor. See the - :strong:`Field copy processors` section in - :ref:`logs-processors` to learn how. + - Create and apply a field copy processor. See the :strong:`Field copy processors` section in :ref:`logs-processors` to learn how. Note: Only customers with a Splunk Log Observer entitlement in Splunk Observability Cloud can use this method. If you are using Log Observer Connect, use one of other methods in this table. * - Log Field Aliasing @@ -193,8 +204,7 @@ The following table describes the four methods for remapping log fields: - Configure your app to remap the necessary fields. * - Collector-side - - Use a Fluentd or FluentBit configuration. See - :ref:`Configure Fluentd to send logs ` to learn how. + - Use a Fluentd or FluentBit configuration. See :ref:`Configure Fluentd to send logs ` to learn how. When to use Log Field Aliasing ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ From a163c81c078d92d511e54dcff8760d5638038e83 Mon Sep 17 00:00:00 2001 From: Anna U <104845867+aurbiztondo-splunk@users.noreply.github.com> Date: Mon, 12 Feb 2024 08:50:29 +0100 Subject: [PATCH 4/6] Update metrics-and-metadata/relatedcontent.rst Co-authored-by: Aunsh Chaudhari --- metrics-and-metadata/relatedcontent.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/metrics-and-metadata/relatedcontent.rst b/metrics-and-metadata/relatedcontent.rst index 466e381c0..3b8d3da62 100644 --- a/metrics-and-metadata/relatedcontent.rst +++ b/metrics-and-metadata/relatedcontent.rst @@ -76,7 +76,7 @@ Related Content and the Splunk Distribution of the OpenTelemetry Collector metad Splunk Observability Cloud uses OpenTelemetry to correlate telemetry types. To enable this ability, your telemetry field names or metadata key names must exactly match the metadata key names used by both OpenTelemetry and Splunk Observability Cloud. -When you deploy the Splunk Distribution of the OpenTelemetry Collector to send your telemetry data to Splunk Observability Cloud, your metadata key names are automatically mapped correctly. +When you deploy the Splunk Distribution of the OpenTelemetry Collector with it's default configuration to send your telemetry data to Splunk Observability Cloud, your metadata key names are automatically mapped correctly. .. caution:: If you don't use the Splunk Distribution of OpenTelemetry Collector, or you use a non-default configuration, your telemetry data might have metadata key names that are not consistent with those used by Splunk Observability Cloud and OpenTelemetry, and Related Content might not work. In that case, you must change your metadata key names. From 03fa950897e4675fac9b2ccf425b2c245ce2d3ae Mon Sep 17 00:00:00 2001 From: Anna U <104845867+aurbiztondo-splunk@users.noreply.github.com> Date: Mon, 12 Feb 2024 08:50:40 +0100 Subject: [PATCH 5/6] Update metrics-and-metadata/relatedcontent.rst Co-authored-by: Aunsh Chaudhari --- metrics-and-metadata/relatedcontent.rst | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/metrics-and-metadata/relatedcontent.rst b/metrics-and-metadata/relatedcontent.rst index 3b8d3da62..f230b4474 100644 --- a/metrics-and-metadata/relatedcontent.rst +++ b/metrics-and-metadata/relatedcontent.rst @@ -100,7 +100,7 @@ When the field names in APM and Log Observer match, the trace and the log with t Required Collector components ================================================================= -If you're using the Splunk OTel Collector and want to ensure Related Content behaves correctly, verify that the SignalFx exporter is included in your configuration. This exporter aggregates the metrics from the ``hostmetrics`` receiver and must be enabled for the ``metrics`` and ``traces`` pipelines. +If you're using the Splunk Distribution of OpenTelemetry Collector or another distribution or the [upstream Collector](https://docs.splunk.com/observability/en/gdi/other-ingestion-methods/upstream-collector.html#using-upstream-otel) and want to ensure Related Content in Splunk Observability Cloud behaves correctly, verify that the SignalFx exporter is included in your configuration. This exporter aggregates the metrics from the ``hostmetrics`` receiver and must be enabled for the ``metrics`` and ``traces`` pipelines. The Collector uses the correlation flag of the SignalFx exporter to make relevant API calls to correlate your spans with the infrastructure metrics. This flag is enabled by default. To adjust the correlation option further, see the SignalFx exporter's options at :ref:`signalfx-exporter-settings`. From c61c76e72157dce54df61dcdc06754c1809719c5 Mon Sep 17 00:00:00 2001 From: Anna Urbiztondo Date: Mon, 12 Feb 2024 08:57:20 +0100 Subject: [PATCH 6/6] Feedback --- metrics-and-metadata/relatedcontent.rst | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/metrics-and-metadata/relatedcontent.rst b/metrics-and-metadata/relatedcontent.rst index f230b4474..585e1a6da 100644 --- a/metrics-and-metadata/relatedcontent.rst +++ b/metrics-and-metadata/relatedcontent.rst @@ -76,7 +76,7 @@ Related Content and the Splunk Distribution of the OpenTelemetry Collector metad Splunk Observability Cloud uses OpenTelemetry to correlate telemetry types. To enable this ability, your telemetry field names or metadata key names must exactly match the metadata key names used by both OpenTelemetry and Splunk Observability Cloud. -When you deploy the Splunk Distribution of the OpenTelemetry Collector with it's default configuration to send your telemetry data to Splunk Observability Cloud, your metadata key names are automatically mapped correctly. +When you deploy the Splunk Distribution of the OpenTelemetry Collector with its default configuration to send your telemetry data to Splunk Observability Cloud, your metadata key names are automatically mapped correctly. .. caution:: If you don't use the Splunk Distribution of OpenTelemetry Collector, or you use a non-default configuration, your telemetry data might have metadata key names that are not consistent with those used by Splunk Observability Cloud and OpenTelemetry, and Related Content might not work. In that case, you must change your metadata key names. @@ -100,7 +100,7 @@ When the field names in APM and Log Observer match, the trace and the log with t Required Collector components ================================================================= -If you're using the Splunk Distribution of OpenTelemetry Collector or another distribution or the [upstream Collector](https://docs.splunk.com/observability/en/gdi/other-ingestion-methods/upstream-collector.html#using-upstream-otel) and want to ensure Related Content in Splunk Observability Cloud behaves correctly, verify that the SignalFx exporter is included in your configuration. This exporter aggregates the metrics from the ``hostmetrics`` receiver and must be enabled for the ``metrics`` and ``traces`` pipelines. +If you're using the Splunk Distribution of OpenTelemetry Collector, another distribution of the Collector, or the :ref:`upstream Collector ` and want to ensure Related Content in Splunk Observability Cloud behaves correctly, verify that the SignalFx exporter is included in your configuration. This exporter aggregates the metrics from the ``hostmetrics`` receiver and must be enabled for the ``metrics`` and ``traces`` pipelines. The Collector uses the correlation flag of the SignalFx exporter to make relevant API calls to correlate your spans with the infrastructure metrics. This flag is enabled by default. To adjust the correlation option further, see the SignalFx exporter's options at :ref:`signalfx-exporter-settings`.