Update Software Catalog setup landing page and add correlation requirements (#31967)

* flesh out set up landing page and update discover entities with correlation requirements

* refinements

* Apply suggestions from code review

Co-authored-by: Brett Blue <[email protected]>

* rearrange ordering

* remove line about telemetry correlation and add a link to renaming rules

---------

Co-authored-by: Brett Blue <[email protected]>

Author: domalessi

content/en/internal_developer_portal/software_catalog/set_up/_index.md

@@ -57,33 +57,71 @@ further_reading:
 Software Catalog entities are defined through [Entity Definitions][1], which are Kubernetes-style YAML configuration files. 
 
 To populate Software Catalog, you can:
-- Set up Datadog Application Performance Monitoring (APM), Universal Service Monitoring (USM), Real User Monitoring (RUM), infrastructure metrics, or logs, which will automatically feed entity data into Software Catalog.
+- Set up Datadog Application Performance Monitoring (APM), Universal Service Monitoring (USM), Real User Monitoring (RUM), infrastructure metrics, or logs, which automatically feed entity data into Software Catalog.
 - Create entity definitions manually or through automation. 
--  Import existing entity definitions from third parties. 
-
-By default, these services are not associated with Datadog telemetry, but you can link telemetry data from Datadog or external sources manually using entity definition YAML files.
+- Import existing entity definitions from third parties. 
 
 ## Automatically discover entities from Datadog
 
+By default, Software Catalog is automatically populated with entries discovered from APM, USM, and RUM. You can also manually import entries from other Datadog telemetries, like logs. 
+
 {{< whatsnext desc=" " >}}
     {{< nextlink href="/internal_developer_portal/software_catalog/set_up/discover_entities#automatic-discovery-with-apm-usm-and-rum" >}}Discover from APM, USM, and RUM{{< /nextlink >}}
     {{< nextlink href="/internal_developer_portal/software_catalog/set_up/discover_entities#import-entities-from-infrastructure-and-logs" >}}Import from Infrastructure and Logs{{< /nextlink >}}
 {{< /whatsnext >}}
 
+### APM
+
+When you instrument your application code with Datadog APM SDKs or OpenTelemetry, your applications emit traces and generate unsampled trace metrics. These traces and metrics power the entity discovery and dependency mapping capabilities in IDP. Your instrumentation choices (for example, your Datadog Agent version, your SDK version, and whether you use custom instrumentation or service overrides) affect the quality and accuracy of your dependency maps. See [Discover from APM, USM, and RUM][5] for details.
+
+### USM
+
+USM detects Golden Signal metrics (for example, requests, errors, and durations) and maps out application dependencies based on eBPF. It does not require instrumentation of your application code.
+
+### RUM 
+
+RUM provides frontend user experience data, including page performance, errors, session events, and views. If you have RUM applications, they appear in Software Catalog as **Frontend Apps** in the component selector. 
+
+### Other Datadog telemetries
+
+You can also import entities that are identified from Datadog telemetries like logs, host metrics, container metrics, network metrics, and process metrics. 
+
+When you use [**Import Entities**][10] and choose a data source, Datadog queries that source and searches for valid `DD_SERVICE` tags. Entities are marked with the `kind:service` attribute.
+
+**Note**: You should only do this if the `DD_SERVICE` tags are well maintained and do not contain irrelevant or incorrect tag values.
+
 ## Create entities
 
+[Entity definitions][1], defined in entity YAML files, are the canonical source of truth in Software Catalog. You can: 
+- Create entity definitions manually through Datadog.
+- Store definitions in a version control system like Git, and set up [Source Code Integration][6] to sync definitions with IDP. Changes made to your files are reflected in Datadog within minutes.
+
 {{< whatsnext desc=" " >}}
     {{< nextlink href="/internal_developer_portal/software_catalog/set_up/create_entities#through-the-datadog-ui" >}}Create through the Datadog UI{{< /nextlink >}}
     {{< nextlink href="/internal_developer_portal/software_catalog/set_up/create_entities#through-automation" >}}Create through code automation{{< /nextlink >}}
 {{< /whatsnext >}}
 
+**Note**: To automatically correlate an entity to its telemetry, the `name` field in your definition must exactly match the primary identifier from the telemetry data. For most services, this is the `service` tag as defined in Datadog's Unified Service Tagging. See examples for [`kind:datastore`][7], [`kind:queue`][8], and other [`entity types`][9].
+
 ## Import entities
 
+If you maintain software inventories in Backstage or ServiceNow CMDB, you can sync these inventories into Datadog's Software Catalog.
+
 {{< whatsnext desc=" " >}}
     {{< nextlink href="/internal_developer_portal/software_catalog/set_up/import_entities#import-from-backstage" >}}Import from Backstage{{< /nextlink >}}
     {{< nextlink href="/internal_developer_portal/software_catalog/set_up/import_entities#import-from-servicenow" >}}Import from ServiceNow{{< /nextlink >}}
 {{< /whatsnext >}}
 
+### Backstage 
+
+You can bring your Backstage entities into Datadog's IDP in one of two ways:
+1. Install [Datadog's Backstage plugin][11]. 
+1. Import entity descriptor files from Backstage to IDP using the Datadog API, Terraform, or Datadog's GitHub integration. 
+
+### ServiceNow
+
+Sync your ServiceNow CMDB inventories with Datadog's Software Catalog by setting up a regular query against your ServiceNow CI tables.
+
 ## Verify configuration completeness 
 
 Following monitoring best practices such as tracing, logging, and code profiling helps you ensure that you have all the data you need during incident triage. Software Catalog provides automatic checks for these recommended setups. 
@@ -125,6 +163,13 @@ The permission is enabled by default in the **Datadog Admin Role** and **Datadog
 [2]: https://app.datadoghq.com/software
 [3]: /account_management/rbac
 [4]: /account_management/rbac/permissions
+[5]: /internal_developer_portal/software_catalog/set_up/discover_entities#automatic-discovery-with-apm-usm-and-rum
+[6]: /integrations/guide/source-code-integration/
+[7]: /internal_developer_portal/software_catalog/entity_model/entity_types?tab=datastore#datastore-peer-tags
+[8]: /internal_developer_portal/software_catalog/entity_model/entity_types?tab=queue#datastore-peer-tags
+[9]: /internal_developer_portal/software_catalog/entity_model?tab=v30
+[10]: https://app.datadoghq.com/software/settings/get-started
+[11]: https://www.npmjs.com/package/@datadog/backstage-plugin-datadog-entity-sync-backend
 
 
 

content/en/internal_developer_portal/software_catalog/set_up/discover_entities.md

@@ -25,7 +25,7 @@ Datadog Software Catalog is pre-populated with entries detected through [APM][1]
 
 All automatically detected entities appear in the Component Selector within Software Catalog.
 
-APM and USM automatically detect the following entities types: `service`,`datastore`, `queue`, `external providers`, `inferred services`, and `endpoints`. APM SDKs identify dependencies of instrumented services and classify them as databases, queues, or third-party APIs—even if those dependencies are not directly instrumented. Custom instrumentation may affect how entities are auto-discovered and how the `service tag` is assigned. To learn more, see [APM Inferred Services][3].
+APM and USM automatically detect the following entities types: `service`,`datastore`, `queue`, `external providers`, `inferred services`, and `endpoints`. APM SDKs identify dependencies of instrumented services and classify them as databases, queues, or third-party APIs—even if those dependencies are not directly instrumented. Custom instrumentation may affect how entities are discovered and how the `service tag` is assigned. To learn more, see [APM Inferred Services][3].
 
 RUM is responsible for discovering `frontend apps` entities.
 
@@ -37,10 +37,21 @@ For information about discovering endpoints, see [Discovering Endpoints from APM
 
 To adjust how inferred entities appear in Software Catalog, you can:
 
-- Assign a display name to identify an entity by a custom name.
+- [Create a renaming rule][18] to identify an entity by a custom name.
 - [Filter out entities][3] by type (database, queue, third-party).
 - [Remove service overrides][4] such as `service:my-service-http-client` from your catalog or map.
 
+### Correlation requirements for APM
+
+To ensure proper auto-correlation between APM and Software Catalog, follow these best practices:
+
+- Ensure your APM SDK and Datadog Agent versions support [inferred entities][15].
+- Avoid arbitrary [service overrides][16] and use [inferred entities][15] instead. 
+  - Arbitrary service overrides can make correlation unreliable and clutter your catalog. They also make Software Catalog hard to use for those unfamiliar with the override names.
+- [Rename inferred entities][14], like datastores and queues, to match your team's naming conventions.
+- Use [Unified Service Tagging][17] to ensure service names, environments, and versions match across APM, infrastructure, and catalog definitions.
+  - In Datadog APM, a service is an identifier used to group traces and aggregate metrics, not necessarily a logical service. Define what constitutes a "service" for your organization (for example, a Kubernetes namespace, deployment, or individual pod) and apply this definition consistently across all telemetry sources. 
+
 
 ## Import entities from Infrastructure and Logs
 
@@ -72,3 +83,8 @@ To remove imported services from the default **Explore** view, click **Clear Pre
 [11]: https://app.datadoghq.com/software/settings/get-started
 [12]: /software_catalog/apis/
 [13]: /integrations/github/
+[14]: /tracing/services/renaming_rules/#step-3-name-your-rule-and-review
+[15]: /tracing/services/inferred_services/?tab=agentv7600#set-up-inferred-services
+[16]: /tracing/guide/service_overrides/#how-service-overrides-are-set
+[17]: /getting_started/tagging/unified_service_tagging/
+[18]: /tracing/services/renaming_rules/