Merge pull request #12 from dimitri-yatsenko/main

Refresh sections on relationships and diagrams.

Author: dimitri-yatsenko

book/00-introduction/49-connect.ipynb

@@ -28,19 +28,34 @@
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": "# Connect with DataJoint\n\nDataJoint is the primary way to connect to the database in this book. The DataJoint client library reads the database credentials from the environment variables `DJ_HOST`, `DJ_USER`, and `DJ_PASS`. \n\nSimply importing the DataJoint library is sufficient—it will connect to the database automatically when needed. Here we call `dj.conn()` only to verify the connection, but this step is not required in normal use."
+   "source": [
+    "# Connect with DataJoint\n",
+    "\n",
+    "DataJoint is the primary way to connect to the database in this book. The DataJoint client library reads the database credentials from the environment variables `DJ_HOST`, `DJ_USER`, and `DJ_PASS`. \n",
+    "\n",
+    "Simply importing the DataJoint library is sufficient—it will connect to the database automatically when needed. Here we call `dj.conn()` only to verify the connection, but this step is not required in normal use."
+   ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": "import datajoint as dj\ndj.conn()  # test the connection (optional)"
+   "source": [
+    "import datajoint as dj\n",
+    "dj.conn()  # test the connection (optional)"
+   ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": "# Connect with SQL Magic\n\nSQL \"Jupyter magic\" allows executing SQL statements directly in Jupyter notebooks, implemented by the [`jupysql`](https://ploomber.io/blog/jupysql/) library. This is useful for quick interactive SQL queries and for learning SQL syntax. We will use SQL magic in this book for demonstrating SQL concepts, but it is not used as part of Python application code.\n\nThe following cell sets up the SQL magic connection to the database."
+   "source": [
+    "# Connect with SQL Magic\n",
+    "\n",
+    "SQL \"Jupyter magic\" allows executing SQL statements directly in Jupyter notebooks, implemented by the [`jupysql`](https://ploomber.io/blog/jupysql/) library. This is useful for quick interactive SQL queries and for learning SQL syntax. We will use SQL magic in this book for demonstrating SQL concepts, but it is not used as part of Python application code.\n",
+    "\n",
+    "The following cell sets up the SQL magic connection to the database."
+   ]
   },
   {
    "cell_type": "code",
@@ -51,43 +66,74 @@
     }
    },
    "outputs": [],
-   "source": "%load_ext sql\n%sql mysql+pymysql://dev:devpass@db"
+   "source": [
+    "%load_ext sql\n",
+    "%sql mysql+pymysql://dev:devpass@db"
+   ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": "You can issue SQL commands from a Jupyter cell by starting it with `%%sql`.\nChange the cell type to `SQL` for appropriate syntax highlighting."
+   "source": [
+    "You can issue SQL commands from a Jupyter cell by starting it with `%%sql`.\n",
+    "Change the cell type to `SQL` for appropriate syntax highlighting."
+   ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": "%%sql\n-- show all users\nSELECT User FROM mysql.user"
+   "source": [
+    "%%sql\n",
+    "-- show all users\n",
+    "SELECT User FROM mysql.user"
+   ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": "# Connect with a Python MySQL Client\n\nTo issue SQL queries directly from Python code (outside of Jupyter magic), you can use a conventional SQL client library such as `pymysql`. This approach gives you full programmatic control over database interactions and is useful when you need to execute raw SQL within Python scripts."
+   "source": [
+    "# Connect with a Python MySQL Client\n",
+    "\n",
+    "To issue SQL queries directly from Python code (outside of Jupyter magic), you can use a conventional SQL client library such as `pymysql`. This approach gives you full programmatic control over database interactions and is useful when you need to execute raw SQL within Python scripts."
+   ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": "import os\nimport pymysql\n\n# create a database connection\nconn = pymysql.connect(\n    host=os.environ['DJ_HOST'], \n    user=os.environ['DJ_USER'], \n    password=os.environ['DJ_PASS']\n)"
+   "source": [
+    "import os\n",
+    "import pymysql\n",
+    "\n",
+    "# create a database connection\n",
+    "conn = pymysql.connect(\n",
+    "    host=os.environ['DJ_HOST'], \n",
+    "    user=os.environ['DJ_USER'], \n",
+    "    password=os.environ['DJ_PASS']\n",
+    ")"
+   ]
   },
   {
    "cell_type": "code",
    "execution_count": null,
    "metadata": {},
    "outputs": [],
-   "source": "# create a query cursor and issue an SQL query\ncur = conn.cursor()\ncur.execute('SELECT User FROM mysql.user')\ncur.fetchall()"
+   "source": [
+    "# create a query cursor and issue an SQL query\n",
+    "cur = conn.cursor()\n",
+    "cur.execute('SELECT User FROM mysql.user')\n",
+    "cur.fetchall()"
+   ]
   },
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": "We are all set for executing all the database queries in this book!"
+   "source": [
+    "We are all set for executing all the database queries in this book!"
+   ]
   }
  ],
  "metadata": {

book/20-concepts/04-integrity.md

@@ -1,8 +1,5 @@
 ---
 title: Data Integrity
-date: 2025-10-31
-authors:
-  - name: Dimitri Yatsenko
 ---
 
 # Why Data Integrity Matters
@@ -95,7 +92,7 @@ Entity integrity ensures a **one-to-one correspondence** between real-world enti
 **Example:** Each mouse in the lab has exactly one unique ID, and that ID refers to exactly one mouse—never two different mice sharing the same ID, and never one mouse having multiple IDs.
 
 **Covered in:**
-- [Primary Keys](../30-design/020-primary-key.md) — Entity integrity and the 1:1 correspondence guarantee (elaborated in detail)
+- [Primary Keys](../30-design/018-primary-key.md) — Entity integrity and the 1:1 correspondence guarantee (elaborated in detail)
 - [UUID](../85-special-topics/025-uuid.ipynb) — Universally unique identifiers
 
 ---
@@ -111,7 +108,7 @@ Referential integrity maintains logical associations across tables:
 **Example:** A recording session cannot reference a non-existent mouse.
 
 **Covered in:**
-- [Foreign Keys](../30-design/030-foreign-keys.ipynb) — Cross-table relationships
+- [Foreign Keys](../30-design/030-foreign-keys.md) — Cross-table relationships
 - [Relationships](../30-design/050-relationships.ipynb) — Dependency patterns
 
 ---
@@ -161,7 +158,7 @@ Workflow integrity maintains valid operation sequences through:
 **Example:** An analysis pipeline cannot compute results before acquiring raw data. If `NeuronAnalysis` depends on `SpikeData`, which depends on `RecordingSession`, the database enforces that recordings are created before spike detection, which occurs before analysis—maintaining the integrity of the entire scientific workflow.
 
 **Covered in:**
-- [Foreign Keys](../30-design/030-foreign-keys.ipynb) — How foreign keys encode workflow dependencies
+- [Foreign Keys](../30-design/030-foreign-keys.md) — How foreign keys encode workflow dependencies
 - [Populate](../40-operations/050-populate.ipynb) — Automatic workflow execution and dependency resolution
 
 ---
@@ -212,8 +209,8 @@ Now that you understand *why* integrity matters, the next chapter introduces how
 The [Design](../30-design/010-schema.ipynb) section then shows *how* to implement each constraint type:
 
 1. **[Tables](../30-design/015-table.ipynb)** — Basic structure with domain integrity
-2. **[Primary Keys](../30-design/020-primary-key.md)** — Entity integrity through unique identification
-3. **[Foreign Keys](../30-design/030-foreign-keys.ipynb)** — Referential integrity across tables
+2. **[Primary Keys](../30-design/018-primary-key.md)** — Entity integrity through unique identification
+3. **[Foreign Keys](../30-design/030-foreign-keys.md)** — Referential integrity across tables
 
 Each chapter builds on these foundational integrity concepts.
 ```

book/30-design/010-schema.ipynb

@@ -3,7 +3,7 @@
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": "---\ntitle: Schemas\nauthors:\n  - name: Dimitri Yatsenko\n---\n\n# What is a schema?\n\nThe term schema has two related meanings in the context of databases:\n\n## 1. Schema as a Data Blueprint\nA **schema** is a formal specification of the structure of data and the rules governing its integrity.\nIt serves as a blueprint that defines how data is organized, stored, and accessed within a database.\nThis ensures that the database reflects the rules and requirements of the underlying business or research project it supports.\n\nIn structured data models, such as the relational model, a schema provides a robust framework for defining:\n* The structure of tables (relations) and their attributes (columns).\n* Rules and constraints that ensure data consistency, accuracy, and reliability.\n* Relationships between tables, such as primary keys (unique identifiers for records) and foreign keys (references to related records in other tables).\n\n### Aims of Good Schema Design\n* **Data Integrity**: Ensures consistency and prevents anomalies.\n* **Query Efficiency**: Facilitates fast and accurate data retrieval, supports complex queries, and optimizes database performance.\n* **Scalability**: Allows the database to grow and adapt as data volumes increase.\n\n### Key Elements of Schema Design\n* **Tables and Attributes**: Each table is defined with specific attributes (columns), each assigned a data type.\n* **Primary Keys**: Uniquely identify each record in a table.\n* **Foreign Keys**: Establish relationships between entities in tables.\n* **Indexes**: Support efficient queries.\n\nThrough careful schema design, database architects create systems that are both efficient and flexible, meeting the current and future needs of an organization. The schema acts as a living document that guides the structure, operations, and integrity of the database.\n\n## 2. Schema as a Database Module\n\nIn complex database designs, the term \"schema\" is also used to describe a distinct module of a larger database with its own namespace that groups related tables together. \nThis modular approach:\n* Separates tables into logical groups for better organization.\n* Avoids naming conflicts in large databases with multiple schemas."
+   "source": "---\ntitle: Schemas\n---\n\n# What is a schema?\n\nThe term schema has two related meanings in the context of databases:\n\n## 1. Schema as a Data Blueprint\nA **schema** is a formal specification of the structure of data and the rules governing its integrity.\nIt serves as a blueprint that defines how data is organized, stored, and accessed within a database.\nThis ensures that the database reflects the rules and requirements of the underlying business or research project it supports.\n\nIn structured data models, such as the relational model, a schema provides a robust framework for defining:\n* The structure of tables (relations) and their attributes (columns).\n* Rules and constraints that ensure data consistency, accuracy, and reliability.\n* Relationships between tables, such as primary keys (unique identifiers for records) and foreign keys (references to related records in other tables).\n\n### Aims of Good Schema Design\n* **Data Integrity**: Ensures consistency and prevents anomalies.\n* **Query Efficiency**: Facilitates fast and accurate data retrieval, supports complex queries, and optimizes database performance.\n* **Scalability**: Allows the database to grow and adapt as data volumes increase.\n\n### Key Elements of Schema Design\n* **Tables and Attributes**: Each table is defined with specific attributes (columns), each assigned a data type.\n* **Primary Keys**: Uniquely identify each record in a table.\n* **Foreign Keys**: Establish relationships between entities in tables.\n* **Indexes**: Support efficient queries.\n\nThrough careful schema design, database architects create systems that are both efficient and flexible, meeting the current and future needs of an organization. The schema acts as a living document that guides the structure, operations, and integrity of the database.\n\n## 2. Schema as a Database Module\n\nIn complex database designs, the term \"schema\" is also used to describe a distinct module of a larger database with its own namespace that groups related tables together. \nThis modular approach:\n* Separates tables into logical groups for better organization.\n* Avoids naming conflicts in large databases with multiple schemas."
   },
   {
    "cell_type": "markdown",
@@ -40,7 +40,7 @@
   {
    "cell_type": "markdown",
    "metadata": {},
-   "source": "# Using the `schema` Object\n\nThe schema object groups related tables together and helps prevent naming conflicts.\n\nBy convention, the object created by `dj.Schema` is named `schema`. Typically, only one schema object is used in any given Python namespace, usually at the level of a Python module.\n\nThe schema object serves multiple purposes:\n* **Creating Tables**: Used as a *class decorator* (`@schema`) to declare tables within the schema. \nFor details, see the next section, [Create Tables](015-table.ipynb)\n* **Visualizing the Schema**: Generates diagrams to illustrate relationships between tables.\n* **Exporting Data**: Facilitates exporting data for external use or backup.\n\nWith this foundation, you are ready to begin declaring tables and building your data pipeline."
+   "source": "# Using the `schema` Object\n\nThe schema object groups related tables together and helps prevent naming conflicts.\n\nBy convention, the object created by `dj.Schema` is named `schema`. Typically, only one schema object is used in any given Python namespace, usually at the level of a Python module.\n\nThe schema object serves multiple purposes:\n* **Creating Tables**: Used as a *class decorator* (`@schema`) to declare tables within the schema. \nFor details, see the next section, [Tables](015-table.ipynb)\n* **Visualizing the Schema**: Generates diagrams to illustrate relationships between tables.\n* **Exporting Data**: Facilitates exporting data for external use or backup.\n\nWith this foundation, you are ready to begin declaring tables and building your data pipeline."
   },
   {
    "cell_type": "markdown",