From ca4a5a55ceb23019a9baff541850c00fe944dfc2 Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Tue, 6 Jan 2026 18:40:10 -0800 Subject: [PATCH 01/21] docs: Add docs for tracking policy activity --- README.md | 2 +- .../observability/elastic/policy-activity.mdx | 70 +++++++++++++++++++ sidebars-calico-enterprise.js | 1 + 3 files changed, 72 insertions(+), 1 deletion(-) create mode 100644 calico-enterprise/observability/elastic/policy-activity.mdx diff --git a/README.md b/README.md index 93ae51f80e..a67959dad6 100644 --- a/README.md +++ b/README.md @@ -75,7 +75,7 @@ You can find links to these preview builds as comments from Netlify. * Deploy preview. A full build with our production configuration. * Deploy preview for _calico-docs-preview-next_. This builds the site based on the current state of our unversioned development directories: * `calico/` - * `calico-enteprise/` + * `calico-enterprise/` * `calico-cloud/` If you're making changes to an upcoming version of any of the products, review your changes in _calico-docs-preview-next_. diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx new file mode 100644 index 0000000000..d6fad4863a --- /dev/null +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -0,0 +1,70 @@ +--- +description: Learn how to use Policy Activity Logs to gain visibility into policy and rule activity. +--- + +# Policy Activity Logs + +:::note + +This feature is tech preview. Tech preview features may be subject to significant changes before they become GA. + +::: + +## Big picture + +Policy Activity Logs provide granular visibility into network policy behavior within your cluster. + +## Value + +Track exactly which rules are evaluated and triggered, offering policy-centric context that complements standard Flow Logs. + +## How it Works + +- **Collection**: The Calico Enterprise dataplane automatically monitors policy rule evaluations across your cluster. +- **Storage**: These events are securely forwarded and indexed in your Elasticsearch cluster for analysis. +- **Analysis**: You can query the logs to audit network policy and rules and validate policy enforcement. + +## Configuration + +Policy Activity Logs are enabled by default. + +### Disabling the Feature + +To disable Policy Activity Logs, set the following environment variables: + +- **Felix**: `FELIX_PolicyActivityLogsFileEnabled: false` +- **Fluentd**: `DISABLE_ES_POLICY_ACTIVITY_LOG: true` + +## Accessing the Data + +You can access Policy Activity Logs by querying Elasticsearch directly or visualizing them in Kibana (if configured). + +### Log Structure + +Each log entry contains the following key fields: + +| Field | Description | Example | +| ----------------- | -------------------------------------------------------------------------------------------- | -------------------------------- | +| `policy.kind` | The type of the policy (e.g., `NetworkPolicy`, `GlobalNetworkPolicy`). | `NetworkPolicy` | +| `policy.namespace`| The namespace where the policy is defined. | `my-test` | +| `policy.name` | The unique name of the policy. | `my-tier.my-networkpolicy` | +| `rule` | The identifier for the specific rule that was triggered, formatted as `{generation}-{direction}-{rule_index}`. | `2-ingress-4` | +| `cluster` | The name of the cluster where the activity occurred. | `cluster` | +| `last_evaluated` | The timestamp when the rule was last evaluated. | `2025-12-01T23:09:28.384714204Z` | + +### Sample Query in Elasticsearch + +To find policies unused for 90 days: + +``` +GET tigera_secure_ee_policy_activity*/_search +{ + "query": { + "range": { + "last_evaluated": { + "lte": "now-90d" + } + } + } +} +``` diff --git a/sidebars-calico-enterprise.js b/sidebars-calico-enterprise.js index f4b0683cb9..1f4a792134 100644 --- a/sidebars-calico-enterprise.js +++ b/sidebars-calico-enterprise.js @@ -392,6 +392,7 @@ module.exports = { link: { type: 'doc', id: 'observability/elastic/l7/index' }, items: ['observability/elastic/l7/configure', 'observability/elastic/l7/datatypes'], }, + 'observability/elastic/policy-activity', 'observability/elastic/troubleshoot', ], }, From 430896228be2b20d3be62f3b965a918caae96612 Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Wed, 7 Jan 2026 11:09:49 -0800 Subject: [PATCH 02/21] chore: nitpicking --- .../observability/elastic/policy-activity.mdx | 12 ++++++------ 1 file changed, 6 insertions(+), 6 deletions(-) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index d6fad4863a..d67ecc56a2 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -18,28 +18,28 @@ Policy Activity Logs provide granular visibility into network policy behavior wi Track exactly which rules are evaluated and triggered, offering policy-centric context that complements standard Flow Logs. -## How it Works +## How it works - **Collection**: The Calico Enterprise dataplane automatically monitors policy rule evaluations across your cluster. - **Storage**: These events are securely forwarded and indexed in your Elasticsearch cluster for analysis. -- **Analysis**: You can query the logs to audit network policy and rules and validate policy enforcement. +- **Analysis**: You can query the logs to audit network policy and rules and validate policy enforcement history. ## Configuration Policy Activity Logs are enabled by default. -### Disabling the Feature +### Disabling the feature To disable Policy Activity Logs, set the following environment variables: - **Felix**: `FELIX_PolicyActivityLogsFileEnabled: false` - **Fluentd**: `DISABLE_ES_POLICY_ACTIVITY_LOG: true` -## Accessing the Data +## Accessing the data You can access Policy Activity Logs by querying Elasticsearch directly or visualizing them in Kibana (if configured). -### Log Structure +### Log structure Each log entry contains the following key fields: @@ -52,7 +52,7 @@ Each log entry contains the following key fields: | `cluster` | The name of the cluster where the activity occurred. | `cluster` | | `last_evaluated` | The timestamp when the rule was last evaluated. | `2025-12-01T23:09:28.384714204Z` | -### Sample Query in Elasticsearch +### Sample query in Elasticsearch To find policies unused for 90 days: From a0b9f258ca46fcc6caa7758da737e5392fe1b975 Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Wed, 7 Jan 2026 11:32:05 -0800 Subject: [PATCH 03/21] chore: nitpicking --- calico-enterprise/observability/elastic/policy-activity.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index d67ecc56a2..12d705dd56 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -20,7 +20,7 @@ Track exactly which rules are evaluated and triggered, offering policy-centric c ## How it works -- **Collection**: The Calico Enterprise dataplane automatically monitors policy rule evaluations across your cluster. +- **Collection**: The Calico Enterprise automatically monitors policy and rule evaluations across your cluster. - **Storage**: These events are securely forwarded and indexed in your Elasticsearch cluster for analysis. - **Analysis**: You can query the logs to audit network policy and rules and validate policy enforcement history. From a993a09328651da48befba9fa83527b6e5011e28 Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Wed, 7 Jan 2026 12:53:22 -0800 Subject: [PATCH 04/21] chore: remove configuration part since it is impossible for customer --- .../observability/elastic/policy-activity.mdx | 11 ----------- 1 file changed, 11 deletions(-) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index 12d705dd56..019fe00349 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -24,17 +24,6 @@ Track exactly which rules are evaluated and triggered, offering policy-centric c - **Storage**: These events are securely forwarded and indexed in your Elasticsearch cluster for analysis. - **Analysis**: You can query the logs to audit network policy and rules and validate policy enforcement history. -## Configuration - -Policy Activity Logs are enabled by default. - -### Disabling the feature - -To disable Policy Activity Logs, set the following environment variables: - -- **Felix**: `FELIX_PolicyActivityLogsFileEnabled: false` -- **Fluentd**: `DISABLE_ES_POLICY_ACTIVITY_LOG: true` - ## Accessing the data You can access Policy Activity Logs by querying Elasticsearch directly or visualizing them in Kibana (if configured). From a4a1e6a48f92163fef2054d465aa3002958deb6c Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Wed, 7 Jan 2026 13:37:20 -0800 Subject: [PATCH 05/21] docs: add more query examples --- .../observability/elastic/policy-activity.mdx | 43 +++++++++++++++++-- 1 file changed, 39 insertions(+), 4 deletions(-) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index 019fe00349..85cc165b20 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -20,7 +20,7 @@ Track exactly which rules are evaluated and triggered, offering policy-centric c ## How it works -- **Collection**: The Calico Enterprise automatically monitors policy and rule evaluations across your cluster. +- **Collection**: Calico Enterprise automatically monitors policy and rule evaluations across your cluster. - **Storage**: These events are securely forwarded and indexed in your Elasticsearch cluster for analysis. - **Analysis**: You can query the logs to audit network policy and rules and validate policy enforcement history. @@ -43,17 +43,52 @@ Each log entry contains the following key fields: ### Sample query in Elasticsearch -To find policies unused for 90 days: +To find policies used for 90 days: ``` -GET tigera_secure_ee_policy_activity*/_search +POST /tigera_secure_ee_policy_activity*/_search { + "size": 0, "query": { "range": { "last_evaluated": { - "lte": "now-90d" + "gte": "now-90d" + } + } + }, + "aggs": { + "policy_summary": { + "composite": { + "sources": [ + { "kind": { "terms": { "field": "policy.kind" } } }, + { "namespace": { "terms": { "field": "policy.namespace" } } }, + { "name": { "terms": { "field": "policy.name" } } } + ] + }, + "aggs": { + "latest_activity": { + "max": { + "field": "last_evaluated" + } + } } } } } ``` + +To find rules used for 90 days: + +``` +POST /tigera_secure_ee_policy_activity*/_search +{ + "size": 0, + "query": { + "range": { + "last_evaluated": { + "gte": "now-90d" + } + } + } +} +``` \ No newline at end of file From 58b98c6f22f47b851ce3c529f691ce32dead7eba Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Wed, 7 Jan 2026 14:30:58 -0800 Subject: [PATCH 06/21] docs: add more detailed query samples --- .../observability/elastic/policy-activity.mdx | 46 ++++++++++++++++--- 1 file changed, 39 insertions(+), 7 deletions(-) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index 85cc165b20..8ddb17fcda 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -41,11 +41,35 @@ Each log entry contains the following key fields: | `cluster` | The name of the cluster where the activity occurred. | `cluster` | | `last_evaluated` | The timestamp when the rule was last evaluated. | `2025-12-01T23:09:28.384714204Z` | -### Sample query in Elasticsearch +Supported values for `policy.kind` include: + +| Kind | Description | +| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | +| `NetworkPolicy` | A standard namespaced Calico NetworkPolicy. | +| `GlobalNetworkPolicy` | A global Calico NetworkPolicy that applies to resources across all namespaces. | +| `KubernetesNetworkPolicy` | A standard, namespaced Kubernetes NetworkPolicy. | +| `KubernetesAdminNetworkPolicy` | A Kubernetes AdminNetworkPolicy (ANP), allowing cluster administrators to secure the cluster with high-priority rules. | +| `KubernetesBaselineAdminNetworkPolicy` | A Kubernetes BaselineAdminNetworkPolicy (BANP), establishing a default security baseline for the entire cluster. | +| `StagedNetworkPolicy` | A Calico NetworkPolicy in "Staged" mode (logs activity without enforcement). | +| `StagedGlobalNetworkPolicy` | A Calico GlobalNetworkPolicy in "Staged" mode (logs activity without enforcement). | +| `StagedKubernetesNetworkPolicy` | A Kubernetes NetworkPolicy that is being previewed in "Staged" mode. | -To find policies used for 90 days: +:::note -``` +The generation number represents the specific version of the policy at the time of evaluation. +It increments whenever the policy is updated, allowing you to correlate activity logs with the exact policy configuration that was active when the traffic occurred. + +::: + +### Identifying unused policies and rules + +Policy Activity Logs record only policies and rules that have been evaluated by actual network traffic. To identify unused policies or rules in your cluster, follow these steps: + +#### Find recently active policies or rules + +Use the following Elasticsearch query to list all policies that have been evaluated in the last 90 days: + +```json POST /tigera_secure_ee_policy_activity*/_search { "size": 0, @@ -57,8 +81,9 @@ POST /tigera_secure_ee_policy_activity*/_search } }, "aggs": { - "policy_summary": { + "active_policies": { "composite": { + "size": 1000, "sources": [ { "kind": { "terms": { "field": "policy.kind" } } }, { "namespace": { "terms": { "field": "policy.namespace" } } }, @@ -77,9 +102,9 @@ POST /tigera_secure_ee_policy_activity*/_search } ``` -To find rules used for 90 days: +To see which rules have been triggered in the last 90 days, use this query: -``` +```json POST /tigera_secure_ee_policy_activity*/_search { "size": 0, @@ -91,4 +116,11 @@ POST /tigera_secure_ee_policy_activity*/_search } } } -``` \ No newline at end of file +``` + +#### Compare with your all network policies and rules +- Export your current policy and rule inventory using `kubectl get ...` commands. +- Compare the exported cluster with the list of active policies or rules returned by the queries above. +- Any policies or rules present in your cluster but missing from the activity logs have not been evaluated by traffic in the last 90 days and may be considered unused. + +> **Tip:** Regularly reviewing unused policies and rules can help you maintain a clean and secure cluster by removing obsolete or unnecessary resources. From ffdfe3edbf75ef9990f69051ec05a78be2bf6a39 Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Wed, 7 Jan 2026 14:40:06 -0800 Subject: [PATCH 07/21] chore: polish the sample query --- .../observability/elastic/policy-activity.mdx | 29 ++++++++++--------- 1 file changed, 15 insertions(+), 14 deletions(-) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index 8ddb17fcda..f3391b081b 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -41,35 +41,36 @@ Each log entry contains the following key fields: | `cluster` | The name of the cluster where the activity occurred. | `cluster` | | `last_evaluated` | The timestamp when the rule was last evaluated. | `2025-12-01T23:09:28.384714204Z` | + +:::note + +The generation number in rule identifier represents the specific version of the policy at the time of evaluation. +It increments whenever the policy is updated, allowing you to correlate activity logs with the exact policy configuration that was active when the traffic occurred. + +::: + Supported values for `policy.kind` include: | Kind | Description | | -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | | `NetworkPolicy` | A standard namespaced Calico NetworkPolicy. | | `GlobalNetworkPolicy` | A global Calico NetworkPolicy that applies to resources across all namespaces. | -| `KubernetesNetworkPolicy` | A standard, namespaced Kubernetes NetworkPolicy. | +| `KubernetesNetworkPolicy` | A standard Kubernetes NetworkPolicy. | | `KubernetesAdminNetworkPolicy` | A Kubernetes AdminNetworkPolicy (ANP), allowing cluster administrators to secure the cluster with high-priority rules. | | `KubernetesBaselineAdminNetworkPolicy` | A Kubernetes BaselineAdminNetworkPolicy (BANP), establishing a default security baseline for the entire cluster. | | `StagedNetworkPolicy` | A Calico NetworkPolicy in "Staged" mode (logs activity without enforcement). | | `StagedGlobalNetworkPolicy` | A Calico GlobalNetworkPolicy in "Staged" mode (logs activity without enforcement). | | `StagedKubernetesNetworkPolicy` | A Kubernetes NetworkPolicy that is being previewed in "Staged" mode. | -:::note - -The generation number represents the specific version of the policy at the time of evaluation. -It increments whenever the policy is updated, allowing you to correlate activity logs with the exact policy configuration that was active when the traffic occurred. - -::: - -### Identifying unused policies and rules +### Examples to identify unused policies and rules Policy Activity Logs record only policies and rules that have been evaluated by actual network traffic. To identify unused policies or rules in your cluster, follow these steps: -#### Find recently active policies or rules +#### Find recently active policies or rules within a period Use the following Elasticsearch query to list all policies that have been evaluated in the last 90 days: -```json +``` POST /tigera_secure_ee_policy_activity*/_search { "size": 0, @@ -104,7 +105,7 @@ POST /tigera_secure_ee_policy_activity*/_search To see which rules have been triggered in the last 90 days, use this query: -```json +``` POST /tigera_secure_ee_policy_activity*/_search { "size": 0, @@ -120,7 +121,7 @@ POST /tigera_secure_ee_policy_activity*/_search #### Compare with your all network policies and rules - Export your current policy and rule inventory using `kubectl get ...` commands. -- Compare the exported cluster with the list of active policies or rules returned by the queries above. -- Any policies or rules present in your cluster but missing from the activity logs have not been evaluated by traffic in the last 90 days and may be considered unused. +- Compare the exported policies/rules with the list of active policies/rules returned by the queries above. +- Any policies/rules present in your cluster but missing from the activity logs have not been evaluated by traffic in the last 90 days and may be considered unused. > **Tip:** Regularly reviewing unused policies and rules can help you maintain a clean and secure cluster by removing obsolete or unnecessary resources. From e4b1817e311103a67f1ea9d077d89d16f3eb9dda Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Thu, 8 Jan 2026 09:13:34 -0800 Subject: [PATCH 08/21] Update calico-enterprise/observability/elastic/policy-activity.mdx Co-authored-by: pasanw --- calico-enterprise/observability/elastic/policy-activity.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index f3391b081b..39ee9037af 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -6,7 +6,7 @@ description: Learn how to use Policy Activity Logs to gain visibility into polic :::note -This feature is tech preview. Tech preview features may be subject to significant changes before they become GA. +This feature is in tech preview. Tech preview features may be subject to significant changes before they become GA. ::: From 6b0e53af5427928250f87ab534e43d5618a28e59 Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Thu, 8 Jan 2026 09:14:16 -0800 Subject: [PATCH 09/21] Update calico-enterprise/observability/elastic/policy-activity.mdx Co-authored-by: pasanw --- calico-enterprise/observability/elastic/policy-activity.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index 39ee9037af..68a06c2dec 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -45,7 +45,7 @@ Each log entry contains the following key fields: :::note The generation number in rule identifier represents the specific version of the policy at the time of evaluation. -It increments whenever the policy is updated, allowing you to correlate activity logs with the exact policy configuration that was active when the traffic occurred. +It increments whenever the spec of the policy is updated, allowing you to correlate activity logs with the exact policy configuration that was active when the traffic occurred. ::: From cedf0c4ee90b7378883d1ffde10d4ea97e761f66 Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Thu, 8 Jan 2026 09:14:47 -0800 Subject: [PATCH 10/21] Update calico-enterprise/observability/elastic/policy-activity.mdx Co-authored-by: pasanw --- calico-enterprise/observability/elastic/policy-activity.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index 68a06c2dec..f159bfb4d4 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -34,7 +34,7 @@ Each log entry contains the following key fields: | Field | Description | Example | | ----------------- | -------------------------------------------------------------------------------------------- | -------------------------------- | -| `policy.kind` | The type of the policy (e.g., `NetworkPolicy`, `GlobalNetworkPolicy`). | `NetworkPolicy` | +| `policy.kind` | The type of the policy. Possible values: `NetworkPolicy`, `GlobalNetworkPolicy`, `KubernetesNetworkPolicy`, `KubernetesAdminNetworkPolicy`, `KubernetesBaselineAdminNetworkPolicy`, `StagedNetworkPolicy`, `StagedGlobalNetworkPolicy`, `StagedKubernetesNetworkPolicy` | `NetworkPolicy` | | `policy.namespace`| The namespace where the policy is defined. | `my-test` | | `policy.name` | The unique name of the policy. | `my-tier.my-networkpolicy` | | `rule` | The identifier for the specific rule that was triggered, formatted as `{generation}-{direction}-{rule_index}`. | `2-ingress-4` | From c1d4711657a8b70d9604b1788fe1b5b2b580c4ac Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Thu, 8 Jan 2026 09:14:56 -0800 Subject: [PATCH 11/21] Update calico-enterprise/observability/elastic/policy-activity.mdx Co-authored-by: pasanw --- calico-enterprise/observability/elastic/policy-activity.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index f159bfb4d4..6a9d341e47 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -66,7 +66,7 @@ Supported values for `policy.kind` include: Policy Activity Logs record only policies and rules that have been evaluated by actual network traffic. To identify unused policies or rules in your cluster, follow these steps: -#### Find recently active policies or rules within a period +#### Find policies active within a recent period Use the following Elasticsearch query to list all policies that have been evaluated in the last 90 days: From 359ce91f84350e7822a878b9d4c4f7ef4d7d2002 Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Thu, 8 Jan 2026 09:15:06 -0800 Subject: [PATCH 12/21] Update calico-enterprise/observability/elastic/policy-activity.mdx Co-authored-by: pasanw --- calico-enterprise/observability/elastic/policy-activity.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index 6a9d341e47..0dfe6037ab 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -103,6 +103,7 @@ POST /tigera_secure_ee_policy_activity*/_search } ``` +#### Find rules active within a recent period To see which rules have been triggered in the last 90 days, use this query: ``` From 4686146169d85b1c7a632ce1d78b92ce519efb2e Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Thu, 8 Jan 2026 09:32:29 -0800 Subject: [PATCH 13/21] Update calico-enterprise/observability/elastic/policy-activity.mdx Co-authored-by: Christopher Tauchen --- calico-enterprise/observability/elastic/policy-activity.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index 0dfe6037ab..20951f98ab 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -2,7 +2,7 @@ description: Learn how to use Policy Activity Logs to gain visibility into policy and rule activity. --- -# Policy Activity Logs +# Policy activity logs :::note From 006003dec4f4b8c59a1cf70bad6aa106219d8484 Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Thu, 8 Jan 2026 11:14:11 -0800 Subject: [PATCH 14/21] docs: rewrite according to suggestions --- .../observability/elastic/policy-activity.mdx | 60 ++++++------------- 1 file changed, 19 insertions(+), 41 deletions(-) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index 20951f98ab..72df246961 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -1,5 +1,5 @@ --- -description: Learn how to use Policy Activity Logs to gain visibility into policy and rule activity. +description: Learn how to use policy activity logs to gain visibility into policy and rule activity. --- # Policy activity logs @@ -10,31 +10,19 @@ This feature is in tech preview. Tech preview features may be subject to signifi ::: -## Big picture +## About policy activity logs -Policy Activity Logs provide granular visibility into network policy behavior within your cluster. +Policy activity logs provide granular visibility into network policy behavior within your cluster. By tracking exactly which rules are evaluated and triggered, these logs offer policy-centric context that complements standard Flow Logs. -## Value +Regularly reviewing policy activity, especially unused policies and rules, helps you maintain a clean and secure cluster by identifying and removing obsolete resources. -Track exactly which rules are evaluated and triggered, offering policy-centric context that complements standard Flow Logs. - -## How it works - -- **Collection**: Calico Enterprise automatically monitors policy and rule evaluations across your cluster. -- **Storage**: These events are securely forwarded and indexed in your Elasticsearch cluster for analysis. -- **Analysis**: You can query the logs to audit network policy and rules and validate policy enforcement history. - -## Accessing the data - -You can access Policy Activity Logs by querying Elasticsearch directly or visualizing them in Kibana (if configured). - -### Log structure +## Log structure / Data types Each log entry contains the following key fields: | Field | Description | Example | | ----------------- | -------------------------------------------------------------------------------------------- | -------------------------------- | -| `policy.kind` | The type of the policy. Possible values: `NetworkPolicy`, `GlobalNetworkPolicy`, `KubernetesNetworkPolicy`, `KubernetesAdminNetworkPolicy`, `KubernetesBaselineAdminNetworkPolicy`, `StagedNetworkPolicy`, `StagedGlobalNetworkPolicy`, `StagedKubernetesNetworkPolicy` | `NetworkPolicy` | +| `policy.kind` | The type of the policy. Possible values: `NetworkPolicy`, `GlobalNetworkPolicy`, `KubernetesNetworkPolicy`, `KubernetesAdminNetworkPolicy`, `KubernetesBaselineAdminNetworkPolicy`, `StagedNetworkPolicy`, `StagedGlobalNetworkPolicy`, `StagedKubernetesNetworkPolicy` | `NetworkPolicy` | | `policy.namespace`| The namespace where the policy is defined. | `my-test` | | `policy.name` | The unique name of the policy. | `my-tier.my-networkpolicy` | | `rule` | The identifier for the specific rule that was triggered, formatted as `{generation}-{direction}-{rule_index}`. | `2-ingress-4` | @@ -44,29 +32,16 @@ Each log entry contains the following key fields: :::note -The generation number in rule identifier represents the specific version of the policy at the time of evaluation. +The generation number in the `rule` identifier (e.g., the `2` in `2-ingress-4`) reflects the specific version of the policy at the time of evaluation. It increments whenever the spec of the policy is updated, allowing you to correlate activity logs with the exact policy configuration that was active when the traffic occurred. ::: -Supported values for `policy.kind` include: +## Find unused policies -| Kind | Description | -| -------------------------------------- | ---------------------------------------------------------------------------------------------------------------------- | -| `NetworkPolicy` | A standard namespaced Calico NetworkPolicy. | -| `GlobalNetworkPolicy` | A global Calico NetworkPolicy that applies to resources across all namespaces. | -| `KubernetesNetworkPolicy` | A standard Kubernetes NetworkPolicy. | -| `KubernetesAdminNetworkPolicy` | A Kubernetes AdminNetworkPolicy (ANP), allowing cluster administrators to secure the cluster with high-priority rules. | -| `KubernetesBaselineAdminNetworkPolicy` | A Kubernetes BaselineAdminNetworkPolicy (BANP), establishing a default security baseline for the entire cluster. | -| `StagedNetworkPolicy` | A Calico NetworkPolicy in "Staged" mode (logs activity without enforcement). | -| `StagedGlobalNetworkPolicy` | A Calico GlobalNetworkPolicy in "Staged" mode (logs activity without enforcement). | -| `StagedKubernetesNetworkPolicy` | A Kubernetes NetworkPolicy that is being previewed in "Staged" mode. | +Policy activity logs record only policies and rules that have been evaluated by actual network traffic. To identify unused policies or rules in your cluster, follow these steps: -### Examples to identify unused policies and rules - -Policy Activity Logs record only policies and rules that have been evaluated by actual network traffic. To identify unused policies or rules in your cluster, follow these steps: - -#### Find policies active within a recent period +### Step 1: Retrieve active policies/rules Use the following Elasticsearch query to list all policies that have been evaluated in the last 90 days: @@ -103,7 +78,6 @@ POST /tigera_secure_ee_policy_activity*/_search } ``` -#### Find rules active within a recent period To see which rules have been triggered in the last 90 days, use this query: ``` @@ -120,9 +94,13 @@ POST /tigera_secure_ee_policy_activity*/_search } ``` -#### Compare with your all network policies and rules -- Export your current policy and rule inventory using `kubectl get ...` commands. -- Compare the exported policies/rules with the list of active policies/rules returned by the queries above. -- Any policies/rules present in your cluster but missing from the activity logs have not been evaluated by traffic in the last 90 days and may be considered unused. +### Step 2: Retrieve all network policies and rules + +Export your current policy and rule inventory using `kubectl get ...` commands. + +### Step 3: Compare lists -> **Tip:** Regularly reviewing unused policies and rules can help you maintain a clean and secure cluster by removing obsolete or unnecessary resources. +Compare the output from the steps above to identify unused resources: +* Match: Check if each policy (or specific rule ID) in your Cluster exists in the Active lists from Elasticsearch. +* Identify Unused: Any policy or rule present in the inventory but missing from the active list has not been evaluated by traffic in the last 90 days. +* Verify: Before deleting any resource, ensure it is not a newly deployed policy that simply hasn't received traffic yet. From 6a88102d405f18359cb968fd7f2758f07befb324 Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Thu, 8 Jan 2026 11:21:22 -0800 Subject: [PATCH 15/21] docs: add limitations --- .../observability/elastic/policy-activity.mdx | 11 ++++++++--- 1 file changed, 8 insertions(+), 3 deletions(-) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index 72df246961..8348c5ed10 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -101,6 +101,11 @@ Export your current policy and rule inventory using `kubectl get ...` commands. ### Step 3: Compare lists Compare the output from the steps above to identify unused resources: -* Match: Check if each policy (or specific rule ID) in your Cluster exists in the Active lists from Elasticsearch. -* Identify Unused: Any policy or rule present in the inventory but missing from the active list has not been evaluated by traffic in the last 90 days. -* Verify: Before deleting any resource, ensure it is not a newly deployed policy that simply hasn't received traffic yet. +- Match: Check if each policy (or specific rule ID) in your Cluster exists in the Active lists from Elasticsearch. +- Identify Unused: Any policy or rule present in the inventory but missing from the active list has not been evaluated by traffic in the last 90 days. +- Verify: Before deleting any resource, ensure it is not a newly deployed policy that simply hasn't received traffic yet. + +## Limitations + +- **Logs require traffic**: Policy activity logs are only generated when a rule is evaluated by traffic. Resources that have never been hit will not appear in these logs. You cannot identify them by querying for "old" logs; you must identify them by their absence from the active log data (as performed in the comparison steps above). +- **Long-lived connections**: A policy evaluation is logged only when a connection is established. If a connection remains open for a long duration (e.g., longer than your 90-day query window), the associated policy may not generate new logs, potentially making it appear "unused" despite actively carrying traffic. From d17e2d9a9d62ae0d66cc9758f9a9d5c9a84c0f1c Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Thu, 8 Jan 2026 11:26:06 -0800 Subject: [PATCH 16/21] chore: format note section --- calico-enterprise/observability/elastic/policy-activity.mdx | 1 + 1 file changed, 1 insertion(+) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index 8348c5ed10..e9afd29668 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -33,6 +33,7 @@ Each log entry contains the following key fields: :::note The generation number in the `rule` identifier (e.g., the `2` in `2-ingress-4`) reflects the specific version of the policy at the time of evaluation. + It increments whenever the spec of the policy is updated, allowing you to correlate activity logs with the exact policy configuration that was active when the traffic occurred. ::: From 289817adbdaa56720a2283fffb807b2997df8122 Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Thu, 8 Jan 2026 11:31:03 -0800 Subject: [PATCH 17/21] chore: nit --- calico-enterprise/observability/elastic/policy-activity.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index e9afd29668..fb2430b9b7 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -102,7 +102,7 @@ Export your current policy and rule inventory using `kubectl get ...` commands. ### Step 3: Compare lists Compare the output from the steps above to identify unused resources: -- Match: Check if each policy (or specific rule ID) in your Cluster exists in the Active lists from Elasticsearch. +- Match: Check if each policy (or specific rule ID) in your Cluster exists in the active lists from Elasticsearch. - Identify Unused: Any policy or rule present in the inventory but missing from the active list has not been evaluated by traffic in the last 90 days. - Verify: Before deleting any resource, ensure it is not a newly deployed policy that simply hasn't received traffic yet. From a24b0d8824ceb6740edb8430fbecd6e2c15a60f3 Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Thu, 8 Jan 2026 13:14:10 -0800 Subject: [PATCH 18/21] chore: describe generation --- calico-enterprise/observability/elastic/policy-activity.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index fb2430b9b7..23764f3da2 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -32,7 +32,7 @@ Each log entry contains the following key fields: :::note -The generation number in the `rule` identifier (e.g., the `2` in `2-ingress-4`) reflects the specific version of the policy at the time of evaluation. +The generation number in the `rule` identifier (e.g., the `2` in `2-ingress-4`) reflects the `metadata.generation` of the policy at the time of evaluation. It increments whenever the spec of the policy is updated, allowing you to correlate activity logs with the exact policy configuration that was active when the traffic occurred. From e50743a379beade2151534a51d4912a02bd9f815 Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Thu, 8 Jan 2026 13:14:34 -0800 Subject: [PATCH 19/21] Update calico-enterprise/observability/elastic/policy-activity.mdx Co-authored-by: pasanw --- calico-enterprise/observability/elastic/policy-activity.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index 23764f3da2..bf173e9a0a 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -16,7 +16,7 @@ Policy activity logs provide granular visibility into network policy behavior wi Regularly reviewing policy activity, especially unused policies and rules, helps you maintain a clean and secure cluster by identifying and removing obsolete resources. -## Log structure / Data types +## Log structure Each log entry contains the following key fields: From 46f9e4243ac900ad7a5778f7bddbed607e8a8896 Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Fri, 9 Jan 2026 08:56:13 -0800 Subject: [PATCH 20/21] fix: set max size of the returned log --- calico-enterprise/observability/elastic/policy-activity.mdx | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index bf173e9a0a..adb19f0c1b 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -84,7 +84,7 @@ To see which rules have been triggered in the last 90 days, use this query: ``` POST /tigera_secure_ee_policy_activity*/_search { - "size": 0, + "size": 10000, "query": { "range": { "last_evaluated": { From 868a5626ca331a4556a53a51f68f358837bae612 Mon Sep 17 00:00:00 2001 From: Tian Feng Date: Fri, 9 Jan 2026 09:16:48 -0800 Subject: [PATCH 21/21] docs: add notes for query large size of data --- .../observability/elastic/policy-activity.mdx | 7 +++++++ 1 file changed, 7 insertions(+) diff --git a/calico-enterprise/observability/elastic/policy-activity.mdx b/calico-enterprise/observability/elastic/policy-activity.mdx index adb19f0c1b..93f6aefef1 100644 --- a/calico-enterprise/observability/elastic/policy-activity.mdx +++ b/calico-enterprise/observability/elastic/policy-activity.mdx @@ -95,6 +95,13 @@ POST /tigera_secure_ee_policy_activity*/_search } ``` +::: note + +If you need to retrieve more than 10,000 logs, do not simply increase the size. Instead, use pagination (especially `scroll API`) to scroll through results efficiently. + +[Learn more](https://www.elastic.co/docs/reference/elasticsearch/rest-apis/paginate-search-results) +::: + ### Step 2: Retrieve all network policies and rules Export your current policy and rule inventory using `kubectl get ...` commands.