update docs and new docs

Author: DebashisBorgohainO2

docs/api/.pages

@@ -4,4 +4,5 @@ nav:
     - Search: search
     - Functions: function
     - Users: user
+    - Cluster: cluster
 

docs/api/cluster/.pages

@@ -0,0 +1,2 @@
+nav:
+    - Cluster Info API: cluster-info.md

docs/api/cluster/cluster-info.md

@@ -0,0 +1,26 @@
+Retrieves operational information about the OpenObserve cluster, currently reporting the number of pending compaction jobs per node. The response is organized by region and node, supporting multi-region environments.
+
+### Request
+```
+GET /api/{org_id}/cluster_info
+```
+**Path Parameter**: {org_id} is the unique identifier of the organization. 
+
+### Response
+Returns a JSON object containing region-wise pending compaction job counts for each node. 
+
+### Example Response
+```json
+{
+  "regions": {
+    "openobserve": {
+      "zo1": {
+        "pending_jobs": 0
+      }
+    }
+  }
+}
+```
+!!! Note
+    The response can be extended to include additional cluster-level metrics.
+    The endpoint supports region-based filtering.

docs/api/function/.pages

@@ -4,3 +4,4 @@ nav:
     - Delete: delete.md
     - List: list.md
     - Examples: examples.md
+

docs/api/search/.pages

@@ -2,3 +2,5 @@ nav:
     - Search: search.md
     - Around: around.md
     - Value: value.md
+    - Profile: profile.md
+    

docs/api/search/profile.md

@@ -0,0 +1,19 @@
+## Search Inspector
+The Search Inspector in OpenObserve helps you identify what happens during a search operation. 
+
+## How to Use It
+Make a request to the following endpoint to get profiling information for your search operations:
+```
+GET /api/{org_id}/search/profile
+```
+This returns structured metadata about recent search executions for your organization.
+
+## Why Use the Search Inspector
+- To find out why a search is slow and which part is causing the delay
+- To see how a search is processed step by step, including WAL, storage, and execution phases
+- To trace how the query moved across nodes, using OpenTelemetry for distributed tracking
+- To view all search metadata in one place, including role, timing, and component context
+- To debug reliably using logs that show durations in human-readable format and clear size metrics
+
+## No Setup Required
+You do not need to configure anything. The Search Inspector works automatically in the background and shows real-time details when you query the endpoint.

docs/environment-variables.md

@@ -128,6 +128,8 @@ OpenObserve is configured through the use of below environment variables.
 | ZO_CALCULATE_STATS_INTERVAL          | 600                        | No           | In seconds. How often stream stats (total size) is calculated  |
 | ZO_ENRICHMENT_TABLE_LIMIT            |                            | No           |  |
 | ZO_SWAGGER_ENABLED                   | true                       | No           | Generate SWAGGER API documentation by default. (since v0.10.8)  |
+| ZO_INGEST_ALLOWED_UPTO                   | 5                       | No           | Discards events older than the specified number of hours. By default, OpenObserve accepts data only if it is not older than 5 hours from the current ingestion time.|
+| ZO_INGEST_ALLOWED_IN_FUTURE                   | 24                       | No           | Discards events dated beyond the specified number of future hours. By default, OpenObserve accepts data only if it is not timestamped more than 24 hours into the future.|
 
 > For local mode, OpenObserve use sqlite as the metadata store.
 >
@@ -203,7 +205,10 @@ OpenObserve is configured through the use of below environment variables.
 | -------------------------------------- | ------------------------- | --------- | ------------------------------------------------------------------------- |
 | ZO_DATA_CACHE_DIR                      | ./data/openobserve/cache/ | No        | local query cache storage directory, applicable only for cluster mode.    |
 | ZO_MEMORY_CACHE_ENABLED                | true                      | No        | enable in-memory caching for files, default is true, the latest files are cached for accelerated queries.           |
-| ZO_MEMORY_CACHE_CACHE_LATEST_FILES     | false                     | No        | by default we just cache files required by data being queried, enable this option to cache all the latest generated files. Caching all latest generated files can accelerate the queries on latest data, the time range for latest cached files depends on the max cache size.   |
+| ZO_CACHE_LATEST_FILES_ENABLED                | false                      | No        | Enables or disables latest file caching.|
+| ZO_CACHE_LATEST_FILES_PARQUET                | true                      | No        | Enables caching of latest parquet files.|
+| ZO_CACHE_LATEST_FILES_INDEX                | true                      | No        | Enables caching of index files.|
+| ZO_CACHE_LATEST_FILES_DELETE_MERGE_FILES                | false                      | No        | Controls whether merged files should be deleted from cache.|
 | ZO_MEMORY_CACHE_MAX_SIZE               | -                         | No        | default 50% of the total memory used for in-memory cache, one can set it to desired amount unit: MB  |
 | ZO_MEMORY_CACHE_SKIP_SIZE              | -                         | No        | default 80% of the total memory cache size, A query will skip memory cache if it need more than this value. one can set it to desired amount unit: MB  |
 | ZO_MEMORY_CACHE_RELEASE_SIZE           | -                         | No        | default drop 1% entries from in-memory cache as cache is full, one can set it to desired amount unit: MB |