metadata

Author: retroryan

METADATA_PLAN.md

@@ -3,7 +3,7 @@
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": "# Metadata Sync: Materialized Delta Tables (Approach 3)\n\nThis notebook materializes Neo4j node labels and relationship types as **managed Delta tables**\nin Unity Catalog. When data is written as a Delta table, UC automatically registers the full\nschema metadata — column names, types, nullability, row counts, and statistics — making it\nbrowsable in **Catalog Explorer** and queryable via `INFORMATION_SCHEMA`.\n\n**What this proves:** Neo4j graph schema can be synchronized into Unity Catalog with zero\ncustom API calls. The Spark Connector infers the schema, and `saveAsTable()` does the rest.\n\n### Steps\n\n1. Load configuration from Databricks Secrets\n2. Verify Neo4j connectivity\n3. Discover all node labels and their properties\n4. Create target UC schemas (catalog must already exist — see README)\n5. Materialize a single label as a Delta table (test)\n6. Verify metadata in `INFORMATION_SCHEMA`\n7. Materialize all discovered labels\n8. Materialize relationship types\n9. Final verification and summary\n\n### Prerequisites\n\n- **Target catalog `neo4j_metadata` must already exist** — see `METADATA_SYNC_README.md`\n- `neo4j-uc-creds` secret scope configured via `setup.sh`\n- Neo4j Spark Connector installed on cluster (`org.neo4j:neo4j-connector-apache-spark_2.12:5.4.0_for_spark_3`)\n- Neo4j Python driver installed (`neo4j`)\n- **Single user** access mode cluster (required by Spark Connector)",
+   "source": "# Metadata Sync: Materialized Delta Tables\n\nThis notebook materializes Neo4j node labels and relationship types as **managed Delta tables**\nin Unity Catalog. When data is written as a Delta table, UC automatically registers the full\nschema metadata — column names, types, nullability, row counts, and statistics — making it\nbrowsable in **Catalog Explorer** and queryable via `INFORMATION_SCHEMA`.\n\n**What this proves:** Neo4j graph schema can be synchronized into Unity Catalog with zero\ncustom API calls. The Spark Connector infers the schema, and `saveAsTable()` does the rest.\n\n### Steps\n\n1. Load configuration from Databricks Secrets\n2. Verify Neo4j connectivity\n3. Discover all node labels and their properties\n4. Create target UC schemas (catalog must already exist — see README)\n5. Materialize a single label as a Delta table (test)\n6. Verify metadata in `INFORMATION_SCHEMA`\n7. Materialize all discovered labels\n8. Materialize relationship types\n9. Final verification and summary\n\n### Prerequisites\n\n- **Target catalog `neo4j_metadata` must already exist** — see `METADATA_SYNC_README.md`\n- `neo4j-uc-creds` secret scope configured via `setup.sh`\n- Neo4j Spark Connector installed on cluster (`org.neo4j:neo4j-connector-apache-spark_2.12:5.4.0_for_spark_3`)\n- Neo4j Python driver installed (`neo4j`)\n- **Single user** access mode cluster (required by Spark Connector)",
    "id": "intro"
   },
   {