reorganizing and a few cleanups

docs/administration/harper-studio/manage-replication.md

@@ -4,7 +4,7 @@ title: Manage Replication
 
 # Manage Replication
 
-Harper instance clustering and replication can be configured directly through the Harper Studio. It is recommended to read through the [clustering documentation](../../developers/clustering/) first to gain a strong understanding of Harper clustering behavior.
+Harper instance clustering and replication can be configured directly through the Harper Studio. It is recommended to read through the [clustering documentation](../../reference/clustering/) first to gain a strong understanding of Harper clustering behavior.
 
 All clustering configuration is handled through the **replication** page of the Harper Studio, accessed with the following instructions:
 
@@ -22,7 +22,7 @@ Note, the **replication** page will only be available to super users.
 
 ## Initial Configuration
 
-Harper instances do not have clustering configured by default. The Harper Studio will walk you through the initial configuration. Upon entering the **replication** screen for the first time you will need to complete the following configuration. Configurations are set in the **enable clustering** panel on the left while actions are described in the middle of the screen. It is worth reviewing the [Creating a Cluster User](../../developers/clustering/creating-a-cluster-user) document before proceeding.
+Harper instances do not have clustering configured by default. The Harper Studio will walk you through the initial configuration. Upon entering the **replication** screen for the first time you will need to complete the following configuration. Configurations are set in the **enable clustering** panel on the left while actions are described in the middle of the screen. It is worth reviewing the [Creating a Cluster User](../../reference/clustering/creating-a-cluster-user) document before proceeding.
 
 1. Enter Cluster User username. (Defaults to `cluster_user`).
 1. Enter Cluster Password.

docs/administration/logging/transaction-logging.md

@@ -10,7 +10,7 @@ Harper offers two options for logging transactions executed against a table. The
 
 The first option is `read_transaction_log`. The transaction log is built upon clustering streams. Clustering streams are per-table message stores that enable data to be propagated across a cluster. Harper leverages streams for use with the transaction log. When clustering is enabled all transactions that occur against a table are pushed to its stream, and thus make up the transaction log.
 
-If you would like to use the transaction log, but have not set up clustering yet, please see ["How to Cluster"](../../developers/clustering/).
+If you would like to use the transaction log, but have not set up clustering yet, please see ["How to Cluster"](../../reference/clustering/).
 
 ## Transaction Log Operations
 

docs/developers/operations-api/clustering-nats.md

@@ -158,7 +158,7 @@ _Operation is restricted to super_user roles only_
 
 ## Add Node
 
-Registers an additional Harper instance with associated subscriptions. Learn more about [Harper clustering here](../clustering/).
+Registers an additional Harper instance with associated subscriptions. Learn more about [Harper clustering here](../../reference/clustering).
 
 _Operation is restricted to super_user roles only_
 
@@ -201,7 +201,7 @@ _Operation is restricted to super_user roles only_
 
 ## Update Node
 
-Modifies an existing Harper instance registration and associated subscriptions. This operation behaves as a PATCH/upsert, meaning it will insert or update the specified replication configurations while leaving other table replication configuration untouched. Learn more about [Harper clustering here](../clustering/).
+Modifies an existing Harper instance registration and associated subscriptions. This operation behaves as a PATCH/upsert, meaning it will insert or update the specified replication configurations while leaving other table replication configuration untouched. Learn more about [Harper clustering here](../../reference/clustering).
 
 _Operation is restricted to super_user roles only_
 
@@ -285,7 +285,7 @@ _Operation is restricted to super_user roles only_
 
 ## Cluster Status
 
-Returns an array of status objects from a cluster. A status object will contain the clustering node name, whether or not clustering is enabled, and a list of possible connections. Learn more about [Harper clustering here](../clustering/).
+Returns an array of status objects from a cluster. A status object will contain the clustering node name, whether or not clustering is enabled, and a list of possible connections. Learn more about [Harper clustering here](../../reference/clustering).
 
 _Operation is restricted to super_user roles only_
 
@@ -332,7 +332,7 @@ _Operation is restricted to super_user roles only_
 
 ## Cluster Network
 
-Returns an object array of enmeshed nodes. Each node object will contain the name of the node, the amount of time (in milliseconds) it took for it to respond, the names of the nodes it is enmeshed with and the routes set in its config file. Learn more about [Harper clustering here](../clustering/).
+Returns an object array of enmeshed nodes. Each node object will contain the name of the node, the amount of time (in milliseconds) it took for it to respond, the names of the nodes it is enmeshed with and the routes set in its config file. Learn more about [Harper clustering here](../../reference/clustering).
 
 _Operation is restricted to super_user roles only_
 
@@ -379,7 +379,7 @@ _Operation is restricted to super_user roles only_
 
 ## Remove Node
 
-Removes a Harper instance and associated subscriptions from the cluster. Learn more about [Harper clustering here](../clustering/).
+Removes a Harper instance and associated subscriptions from the cluster. Learn more about [Harper clustering here](../../reference/clustering).
 
 _Operation is restricted to super_user roles only_
 
@@ -408,7 +408,7 @@ _Operation is restricted to super_user roles only_
 ## Configure Cluster
 
 Bulk create/remove subscriptions for any number of remote nodes. Resets and replaces any existing clustering setup.
-Learn more about [Harper clustering here](../clustering/).
+Learn more about [Harper clustering here](../../reference/clustering).
 
 _Operation is restricted to super_user roles only_
 

docs/developers/security/certificate-management.md

@@ -4,7 +4,7 @@ title: Certificate Management
 
 # Certificate Management
 
-This document is information on managing certificates for Harper external facing APIs. For information on certificate management for clustering see [clustering certificate management](../clustering/certificate-management).
+This document is information on managing certificates for Harper external facing APIs. For information on certificate management for clustering see [clustering certificate management](../../reference/clustering/certificate-management).
 
 ## Development
 

docs/reference/sql-guide/json-search.md

@@ -0,0 +1,177 @@
+---
+title: SQL JSON Search
+---
+
+:::warning
+Harper encourages developers to utilize other querying tools over SQL for performance purposes. Harper SQL is intended for data investigation purposes and uses cases where performance is not a priority. SQL optimizations are on our roadmap for the future.
+:::
+
+# SQL JSON Search
+
+Harper automatically indexes all top level attributes in a row / object written to a table. However, any attributes which hold JSON data do not have their nested attributes indexed. In order to make searching and/or transforming these JSON documents easy, Harper offers a special SQL function called SEARCH_JSON. The SEARCH_JSON function works in SELECT & WHERE clauses allowing queries to perform powerful filtering on any element of your JSON by implementing the [JSONata library](https://docs.jsonata.org/overview.html) into our SQL engine.
+
+## Syntax
+
+`SEARCH_JSON(expression, attribute)`
+
+Executes the supplied string _expression_ against data of the defined top level _attribute_ for each row. The expression both filters and defines output from the JSON document.
+
+### Example 1
+
+#### Search a string array
+
+Here are two records in the database:
+
+```json
+[
+	{
+		"id": 1,
+		"name": ["Harper", "Penny"]
+	},
+	{
+		"id": 2,
+		"name": ["Penny"]
+	}
+]
+```
+
+Here is a simple query that gets any record with "Harper" found in the name.
+
+```
+SELECT *
+FROM dev.dog
+WHERE search_json('"Harper" in *', name)
+```
+
+### Example 2
+
+The purpose of this query is to give us every movie where at least two of our favorite actors from Marvel films have acted together. The results will return the movie title, the overview, release date and an object array of the actor’s name and their character name in the movie.
+
+Both function calls evaluate the credits.cast attribute, this attribute is an object array of every cast member in a movie.
+
+```
+SELECT m.title,
+    m.overview,
+    m.release_date,
+    SEARCH_JSON($[name in ["Robert Downey Jr.", "Chris Evans", "Scarlett Johansson", "Mark Ruffalo", "Chris Hemsworth", "Jeremy Renner", "Clark Gregg", "Samuel L. Jackson", "Gwyneth Paltrow", "Don Cheadle"]].{"actor": name, "character": character}, c.`cast`) AS characters
+FROM movies.credits c
+    INNER JOIN movies.movie m
+    ON c.movie_id = m.id
+WHERE SEARCH_JSON($count($[name in ["Robert Downey Jr.", "Chris Evans", "Scarlett Johansson", "Mark Ruffalo", "Chris Hemsworth", "Jeremy Renner", "Clark Gregg", "Samuel L. Jackson", "Gwyneth Paltrow", "Don Cheadle"]]), c.`cast`) >= 2
+```
+
+A sample of this data from the movie The Avengers looks like
+
+```json
+[
+	{
+		"cast_id": 46,
+		"character": "Tony Stark / Iron Man",
+		"credit_id": "52fe4495c3a368484e02b251",
+		"gender": "male",
+		"id": 3223,
+		"name": "Robert Downey Jr.",
+		"order": 0
+	},
+	{
+		"cast_id": 2,
+		"character": "Steve Rogers / Captain America",
+		"credit_id": "52fe4495c3a368484e02b19b",
+		"gender": "male",
+		"id": 16828,
+		"name": "Chris Evans",
+		"order": 1
+	},
+	{
+		"cast_id": 307,
+		"character": "Bruce Banner / The Hulk",
+		"credit_id": "5e85e8083344c60015411cfa",
+		"gender": "male",
+		"id": 103,
+		"name": "Mark Ruffalo",
+		"order": 2
+	}
+]
+```
+
+Let’s break down the SEARCH_JSON function call in the SELECT:
+
+```
+SEARCH_JSON(
+    $[name in [
+        "Robert Downey Jr.",
+        "Chris Evans",
+        "Scarlett Johansson",
+        "Mark Ruffalo",
+        "Chris Hemsworth",
+        "Jeremy Renner",
+        "Clark Gregg",
+        "Samuel L. Jackson",
+        "Gwyneth Paltrow",
+        "Don Cheadle"
+    ]].{
+        "actor": name,
+        "character": character
+    },
+    c.`cast`
+)
+```
+
+The first argument passed to SEARCH_JSON is the expression to execute against the second argument which is the cast attribute on the credits table. This expression will execute for every row. Looking into the expression it starts with "$[…]" this tells the expression to iterate all elements of the cast array.
+
+Then the expression tells the function to only return entries where the name attribute matches any of the actors defined in the array:
+
+```
+name in ["Robert Downey Jr.", "Chris Evans", "Scarlett Johansson", "Mark Ruffalo", "Chris Hemsworth", "Jeremy Renner", "Clark Gregg", "Samuel L. Jackson", "Gwyneth Paltrow", "Don Cheadle"]
+```
+
+So far, we’ve iterated the array and filtered out rows, but we also want the results formatted in a specific way, so we’ve chained an expression on our filter with: `{"actor": name, "character": character}`. This tells the function to create a specific object for each matching entry.
+
+**Sample Result**
+
+```json
+[
+	{
+		"actor": "Robert Downey Jr.",
+		"character": "Tony Stark / Iron Man"
+	},
+	{
+		"actor": "Chris Evans",
+		"character": "Steve Rogers / Captain America"
+	},
+	{
+		"actor": "Mark Ruffalo",
+		"character": "Bruce Banner / The Hulk"
+	}
+]
+```
+
+Just having the SEARCH_JSON function in our SELECT is powerful, but given our criteria it would still return every other movie that doesn’t have our matching actors, in order to filter out the movies we do not want we also use SEARCH_JSON in the WHERE clause.
+
+This function call in the WHERE clause is similar, but we don’t need to perform the same transformation as occurred in the SELECT:
+
+```
+SEARCH_JSON(
+    $count(
+    $[name in [
+            "Robert Downey Jr.",
+            "Chris Evans",
+            "Scarlett Johansson",
+            "Mark Ruffalo",
+            "Chris Hemsworth",
+            "Jeremy Renner",
+            "Clark Gregg",
+            "Samuel L. Jackson",
+            "Gwyneth Paltrow",
+            "Don Cheadle"
+        ]]
+    ),
+    c.`cast`
+) >= 2
+```
+
+As seen above we execute the same name filter against the cast array, the primary difference is we are wrapping the filtered results in $count(…). As it looks this returns a count of the results back which we then use against our SQL comparator of >= 2.
+
+To see further SEARCH_JSON examples in action view our Postman Collection that provides a [sample database & data with query examples](../../../developers/operations-api/advanced-json-sql-examples).
+
+To learn more about how to build expressions check out the JSONata documentation: [https://docs.jsonata.org/overview](https://docs.jsonata.org/overview)

redirects.ts

@@ -149,8 +149,12 @@ function generateDocsRedirects(basePath: string): RedirectRule[] {
 			to: withBase('/developers/applications/debugging'),
 		},
 
-		// SQL Guide
-		{ from: withBase('/sql-guide'), to: withBase('/developers/sql-guide/') },
+		// SQL Guide Root Page
+		{ from: withBase('/sql-guide'), to: withBase('/reference/sql-guide/') },
+		{ from: withBase('/developers/sql-guide'), to: withBase('/reference/sql-guide/') },
+
+		// Clustering Root Page
+		{ from: withBase('/clustering'), to: withBase('/reference/clustering/') },
 
 		// CLI
 		{ from: withBase('/harperdb-cli'), to: withBase('/deployments/harper-cli') },
@@ -285,6 +289,21 @@ export function createRedirects(existingPath: string, basePath: string = ''): st
 		}
 	}
 
+	if (existingPath.startsWith(`${basePath}/reference/sql-guide`)) {
+		const subpath = existingPath.replace(`${basePath}/reference/sql-guide`, '');
+		if (subpath) {
+			redirects.push(`${basePath}/sql-guide${subpath}`);
+			redirects.push(`${basePath}/developers/sql-guide${subpath}`);
+		}
+	}
+
+	if (existingPath.startsWith(`${basePath}/reference/clustering`)) {
+		const subpath = existingPath.replace(`${basePath}/reference/clustering`, '');
+		if (subpath) {
+			redirects.push(`${basePath}/developers/clustering${subpath}`);
+		}
+	}
+
 	// Old Technical Details -> Reference paths
 	if (existingPath.startsWith(`${basePath}/reference/`)) {
 		const subpath = existingPath.replace(`${basePath}/reference/`, '');
@@ -296,7 +315,6 @@ export function createRedirects(existingPath: string, basePath: string = ''): st
 	// Don't create wildcard redirects for these as they're all explicitly defined:
 	// - /developers/security/* (all subpaths are explicit)
 	// - /deployments/harper-cli (explicit)
-	// - /developers/sql-guide/* (has explicit redirect)
 	// - /developers/operations-api/* (has explicit redirects)
 
 	return redirects.length > 0 ? redirects : undefined;

versioned_docs/version-4.1/security/certificate-management.md

@@ -4,7 +4,7 @@ title: Certificate Management
 
 # Certificate Management
 
-This document is information on managing certificates for the Operations API and the Custom Functions API. For information on certificate managment for clustering see [clustering certificate management](../clustering/certificate-management).
+This document is information on managing certificates for the Operations API and the Custom Functions API. For information on certificate managment for clustering see [clustering certificate management](../../reference/clustering/certificate-management).
 
 ## Development
 

versioned_docs/version-4.2/administration/logging/transaction-logging.md

@@ -10,7 +10,7 @@ HarperDB offers two options for logging transactions executed against a table. T
 
 The first option is `read_transaction_log`. The transaction log is built upon clustering streams. Clustering streams are per-table message stores that enable data to be propagated across a cluster. HarperDB leverages streams for use with the transaction log. When clustering is enabled all transactions that occur against a table are pushed to its stream, and thus make up the transaction log.
 
-If you would like to use the transaction log, but have not set up clustering yet, please see ["How to Cluster"](../../developers/clustering/).
+If you would like to use the transaction log, but have not set up clustering yet, please see ["How to Cluster"](../../reference/clustering).
 
 ## Transaction Log Operations
 

versioned_docs/version-4.2/developers/operations-api/index.md

@@ -24,8 +24,8 @@ The operations API reference is available below and categorized by topic:
 - [NoSQL Operations](operations-api/nosql-operations)
 - [Bulk Operations](operations-api/bulk-operations)
 - [Users and Roles](operations-api/users-and-roles)
-- [Clustering](clustering)
-- [Components](components)
+- [Clustering](operations-api/clustering)
+- [Components](operations-api/components)
 - [Registration](operations-api/registration)
 - [Jobs](operations-api/jobs)
 - [Logs](operations-api/logs)

versioned_docs/version-4.2/developers/security/certificate-management.md

@@ -4,7 +4,7 @@ title: Certificate Management
 
 # Certificate Management
 
-This document is information on managing certificates for HarperDB external facing APIs. For information on certificate management for clustering see [clustering certificate management](../clustering/certificate-management).
+This document is information on managing certificates for HarperDB external facing APIs. For information on certificate management for clustering see [clustering certificate management](../../reference/clustering/certificate-management).
 
 ## Development
 

versioned_docs/version-4.2/index.md

@@ -85,7 +85,7 @@ Wondering what's new with HarperDB 4.2? Take a look at our latest [Release Notes
 <div style={{display: 'grid', gridTemplateColumns: 'repeat(auto-fit, minmax(250px, 1fr))', gap: '1rem', margin: '2rem 0'}}>
   <div style={{border: '1px solid #e0e0e0', borderRadius: '8px', padding: '1.5rem', transition: 'box-shadow 0.2s'}}>
     <h3 style={{marginTop: '0'}}>
-      <a href="./4.2/developers/clustering/" style={{textDecoration: 'none', color: 'inherit'}}>
+      <a href="./4.2/reference/clustering/" style={{textDecoration: 'none', color: 'inherit'}}>
         Clustering &#x26; Replication
       </a>
     </h3>

versioned_docs/version-4.2/developers/sql-guide/json-search.md → versioned_docs/version-4.2/reference/sql-guide/json-search.md

@@ -168,6 +168,6 @@ SEARCH_JSON(
 
 As seen above we execute the same name filter against the cast array, the primary difference is we are wrapping the filtered results in $count(…). As it looks this returns a count of the results back which we then use against our SQL comparator of >= 2.
 
-To see further SEARCH_JSON examples in action view our Postman Collection that provides a [sample schema & data with query examples](../operations-api/advanced-json-sql-examples).
+To see further SEARCH_JSON examples in action view our Postman Collection that provides a [sample schema & data with query examples](../../developers/operations-api/advanced-json-sql-examples).
 
 To learn more about how to build expressions check out the JSONata documentation: [https://docs.jsonata.org/overview](https://docs.jsonata.org/overview)