|
1 | 1 | # DataJoint Documentation |
2 | 2 |
|
3 | | -Official documentation for [DataJoint](https://github.com/datajoint/datajoint-python) 2.0, |
4 | | -an open-source framework for building scientific data pipelines. |
| 3 | +Official documentation for [DataJoint](https://github.com/datajoint/datajoint-python) 2.0 — an open-source framework for building scientific data pipelines. |
5 | 4 |
|
6 | | -**Live site:** https://docs.datajoint.com |
| 5 | +**📖 Live site:** https://docs.datajoint.com |
7 | 6 |
|
8 | | -> **📘 Upgrading from legacy DataJoint (pre-2.0)?** |
9 | | -> See the **[Migration Guide](https://docs.datajoint.com/how-to/migrate-from-0x/)** for a step-by-step upgrade path. |
| 7 | +> **Upgrading from pre-2.0?** See the [Migration Guide](https://docs.datajoint.com/how-to/migrate-to-v20/) |
10 | 8 |
|
11 | | -## What is DataJoint? |
| 9 | +## About DataJoint |
12 | 10 |
|
13 | | -DataJoint is a Python framework for building scientific data pipelines using relational |
14 | | -databases. It implements the **Relational Workflow Model**—a paradigm that extends |
15 | | -relational databases with native support for computational workflows. |
16 | | - |
17 | | -Key features: |
18 | | - |
19 | | -- **Declarative schema design** — Define tables and relationships in Python |
20 | | -- **Automatic dependency tracking** — Foreign keys encode workflow dependencies |
21 | | -- **Built-in computation** — Imported and Computed tables run automatically |
22 | | -- **Data integrity** — Referential integrity and transaction support |
23 | | -- **Reproducibility** — Immutable data with full provenance |
24 | | - |
25 | | -## Quick Start |
26 | | - |
27 | | -### Installation |
28 | | - |
29 | | -```bash |
30 | | -pip install datajoint |
31 | | -``` |
32 | | - |
33 | | -For schema diagrams, install Graphviz (the system library, not just Python bindings): |
34 | | - |
35 | | -```bash |
36 | | -# macOS |
37 | | -brew install graphviz |
38 | | - |
39 | | -# Ubuntu/Debian |
40 | | -sudo apt-get install graphviz libgraphviz-dev |
41 | | - |
42 | | -# conda (any platform) |
43 | | -conda install -c conda-forge graphviz pygraphviz |
44 | | -``` |
45 | | - |
46 | | -If using pip (after installing system Graphviz): |
47 | | -```bash |
48 | | -pip install pygraphviz |
49 | | -``` |
50 | | - |
51 | | -### Configuration |
52 | | - |
53 | | -DataJoint uses configuration files to manage database credentials securely. Create these |
54 | | -files in your project directory: |
55 | | - |
56 | | -**datajoint.json** (non-sensitive settings, commit to version control): |
57 | | -```json |
58 | | -{ |
59 | | - "database": { |
60 | | - "host": "localhost", |
61 | | - "port": 3306 |
62 | | - } |
63 | | -} |
64 | | -``` |
65 | | - |
66 | | -**.secrets/database.user** and **.secrets/database.password** (sensitive, add to .gitignore): |
67 | | -```bash |
68 | | -mkdir -p .secrets |
69 | | -echo "your_username" > .secrets/database.user |
70 | | -echo "your_password" > .secrets/database.password |
71 | | -chmod 600 .secrets/* |
72 | | -echo ".secrets/" >> .gitignore |
73 | | -``` |
74 | | - |
75 | | -DataJoint automatically discovers these files by searching up from the current directory. |
76 | | -This keeps credentials out of your code and version control. |
77 | | - |
78 | | -### Define a Schema |
79 | | - |
80 | | -```python |
81 | | -import datajoint as dj |
82 | | - |
83 | | -schema = dj.Schema('my_pipeline') |
84 | | - |
85 | | -@schema |
86 | | -class Subject(dj.Manual): |
87 | | - definition = """ |
88 | | - subject_id : int32 |
89 | | - --- |
90 | | - name : varchar(100) |
91 | | - date_of_birth : date |
92 | | - """ |
93 | | - |
94 | | -@schema |
95 | | -class Session(dj.Manual): |
96 | | - definition = """ |
97 | | - -> Subject |
98 | | - session_idx : int32 |
99 | | - --- |
100 | | - session_date : date |
101 | | - duration : float32 # minutes |
102 | | - notes = '' : varchar(1000) |
103 | | - """ |
104 | | - |
105 | | -@schema |
106 | | -class ProcessedData(dj.Computed): |
107 | | - definition = """ |
108 | | - -> Session |
109 | | - --- |
110 | | - result : float64 |
111 | | - """ |
112 | | - |
113 | | - def make(self, key): |
114 | | - # Compute result from session data |
115 | | - duration = (Session & key).fetch1('duration') |
116 | | - self.insert1({**key, 'result': duration * 2}) |
117 | | -``` |
118 | | - |
119 | | -Note: Use DataJoint core types (`int32`, `float32`, `float64`, `varchar`) for portability |
120 | | -across database backends. |
121 | | - |
122 | | -### View Schema Diagram |
123 | | - |
124 | | -```python |
125 | | -dj.Diagram(schema) |
126 | | -``` |
127 | | - |
128 | | -### Run Computations |
129 | | - |
130 | | -```python |
131 | | -ProcessedData.populate() |
132 | | -``` |
| 11 | +DataJoint is a Python framework for scientific data pipelines built on the **Relational Workflow Model**. For installation, tutorials, and complete documentation, visit **[docs.datajoint.com](https://docs.datajoint.com)**. |
133 | 12 |
|
134 | 13 | ## Documentation Structure |
135 | 14 |
|
136 | | -This documentation follows the [Diátaxis](https://diataxis.fr/) framework: |
137 | | - |
138 | | -| Section | Purpose | Link | |
139 | | -|---------|---------|------| |
140 | | -| **Tutorials** | Learn by building real pipelines | [src/tutorials/](src/tutorials/) | |
141 | | -| **How-To Guides** | Practical guides for common tasks | [src/how-to/](src/how-to/) | |
142 | | -| **Explanation** | Understand the principles behind DataJoint | [src/explanation/](src/explanation/) | |
143 | | -| **Reference** | Specifications and API documentation | [src/reference/](src/reference/) | |
| 15 | +This repository contains the source for the DataJoint documentation, organized using the [Diátaxis](https://diataxis.fr/) framework: |
144 | 16 |
|
145 | | -Key pages: |
146 | | -- **[Migration Guide](src/how-to/migrate-from-0x.md)** — Upgrade from legacy DataJoint (pre-2.0) |
147 | | -- **[What's New in 2.0](src/explanation/whats-new-2.md)** — Major changes and improvements |
| 17 | +- **[Tutorials](https://docs.datajoint.com/tutorials/)** — Learn by building real pipelines |
| 18 | +- **[How-To Guides](https://docs.datajoint.com/how-to/)** — Practical task-oriented guides |
| 19 | +- **[Explanation](https://docs.datajoint.com/explanation/)** — Understanding concepts and design |
| 20 | +- **[Reference](https://docs.datajoint.com/reference/)** — Specifications and API documentation |
148 | 21 |
|
149 | | -## Local Development with Docker (Recommended) |
| 22 | +## Local Development |
150 | 23 |
|
151 | | -The Docker environment includes MySQL, MinIO (S3-compatible storage), Graphviz, and all |
152 | | -dependencies needed to build documentation and execute tutorial notebooks. |
153 | | - |
154 | | -### Start the Environment |
| 24 | +### Docker (Recommended) |
155 | 25 |
|
156 | 26 | ```bash |
157 | | -# Clone the documentation repository |
| 27 | +# Clone repositories |
158 | 28 | git clone https://github.com/datajoint/datajoint-docs.git |
159 | 29 | cd datajoint-docs |
160 | | - |
161 | | -# Clone datajoint-python pre-release branch (required for API docs) |
162 | 30 | cd .. |
163 | 31 | git clone -b pre/v2.0 https://github.com/datajoint/datajoint-python.git |
164 | 32 | cd datajoint-docs |
165 | 33 |
|
166 | | -# Start all services (MySQL, MinIO, docs server) |
| 34 | +# Start live preview at http://localhost:8000 |
167 | 35 | MODE="LIVE" docker compose up --build |
168 | | -``` |
169 | | - |
170 | | -Navigate to http://127.0.0.1:8000/ |
171 | | - |
172 | | -### Services |
173 | | - |
174 | | -| Service | Port | Description | |
175 | | -|---------|------|-------------| |
176 | | -| `docs` | 8000 | MkDocs live server | |
177 | | -| `mysql` | 3306 | MySQL 8.0 database | |
178 | | -| `minio` | 9002 | MinIO S3 API | |
179 | | -| `minio` | 9003 | MinIO console | |
180 | | - |
181 | | -### Execute Tutorial Notebooks |
182 | | - |
183 | | -Tutorial notebooks can be executed inside the Docker environment where the database |
184 | | -is available: |
185 | 36 |
|
186 | | -```bash |
187 | | -# Execute a single notebook |
188 | | -docker compose exec docs jupyter nbconvert \ |
189 | | - --to notebook --execute --inplace \ |
190 | | - /main/src/tutorials/01-getting-started.ipynb |
191 | | - |
192 | | -# Execute all tutorials |
193 | | -docker compose exec docs bash -c ' |
194 | | - for nb in /main/src/tutorials/*.ipynb; do |
195 | | - jupyter nbconvert --to notebook --execute --inplace "$nb" |
196 | | - done |
197 | | -' |
| 37 | +# Build static site (optional) |
| 38 | +# MODE="BUILD" docker compose up --build |
198 | 39 | ``` |
199 | 40 |
|
200 | | -### Build Static Site |
| 41 | +The Docker environment includes MySQL, MinIO, and all dependencies. |
201 | 42 |
|
202 | | -```bash |
203 | | -# Build static HTML (output in site/) |
204 | | -MODE="BUILD" docker compose up --build |
205 | | -``` |
| 43 | +### Native Python |
206 | 44 |
|
207 | | -### Reset Database |
| 45 | +**Prerequisites:** Python 3.10+, MySQL 8.0+ |
208 | 46 |
|
209 | 47 | ```bash |
210 | | -# Stop services and remove data volumes |
211 | | -docker compose down -v |
212 | | -``` |
213 | | - |
214 | | -## Local Development without Docker |
215 | | - |
216 | | -### Prerequisites |
217 | | - |
218 | | -- Python 3.10+ |
219 | | -- MySQL 8.0+ (running locally) |
220 | | -- Graphviz (for schema diagrams) |
221 | | - |
222 | | -### Setup |
223 | | - |
224 | | -```bash |
225 | | -# Clone the repository |
| 48 | +# Setup |
226 | 49 | git clone https://github.com/datajoint/datajoint-docs.git |
227 | 50 | cd datajoint-docs |
228 | | - |
229 | | -# Create virtual environment |
230 | | -python -m venv .venv |
231 | | -source .venv/bin/activate # or .venv\Scripts\activate on Windows |
232 | | - |
233 | | -# Install dependencies |
| 51 | +python -m venv .venv && source .venv/bin/activate # Windows: .venv\Scripts\activate |
234 | 52 | pip install -r pip_requirements.txt |
235 | | -``` |
236 | | - |
237 | | -Note: For schema diagrams, ensure Graphviz system libraries are installed (see Quick Start). |
238 | 53 |
|
239 | | -### Configure Database Connection |
240 | | - |
241 | | -The repository includes a `datajoint.json` with default settings. Create the secrets |
242 | | -directory with your credentials: |
243 | | - |
244 | | -```bash |
| 54 | +# Configure credentials |
245 | 55 | mkdir -p .secrets |
246 | 56 | echo "your_username" > .secrets/database.user |
247 | 57 | echo "your_password" > .secrets/database.password |
248 | 58 | chmod 600 .secrets/* |
249 | | -``` |
250 | | - |
251 | | -### Preview Documentation |
252 | 59 |
|
253 | | -```bash |
| 60 | +# Start live preview at http://localhost:8000 |
254 | 61 | mkdocs serve |
255 | 62 | ``` |
256 | 63 |
|
257 | | -Navigate to http://127.0.0.1:8000/ |
258 | | - |
259 | 64 | ## Contributing |
260 | 65 |
|
261 | | -Contributions are welcome! See our [contribution guidelines](src/about/contributing.md). |
262 | | - |
263 | | -### Quick Fixes |
264 | | - |
265 | | -1. Fork the repository |
266 | | -2. Edit the relevant markdown file in `src/` |
267 | | -3. Submit a pull request |
268 | | - |
269 | | -### Larger Changes |
| 66 | +Contributions welcome! See [contribution guidelines](https://docs.datajoint.com/about/contributing/). |
270 | 67 |
|
271 | | -1. Open an issue to discuss the change |
272 | | -2. Fork and create a feature branch |
273 | | -3. Make changes with `mkdocs serve` for preview |
274 | | -4. Submit a pull request |
275 | | - |
276 | | -### Executing Notebooks for CI |
277 | | - |
278 | | -When modifying tutorial notebooks, re-execute them to update outputs: |
| 68 | +**Quick fixes:** Fork, edit markdown in `src/`, submit PR. |
279 | 69 |
|
| 70 | +**Tutorial notebooks:** Re-execute after changes: |
280 | 71 | ```bash |
281 | | -docker compose exec docs jupyter nbconvert \ |
282 | | - --to notebook --execute --inplace \ |
283 | | - --ExecutePreprocessor.timeout=300 \ |
| 72 | +docker compose exec docs jupyter nbconvert --to notebook --execute --inplace \ |
284 | 73 | /main/src/tutorials/YOUR_NOTEBOOK.ipynb |
285 | 74 | ``` |
286 | 75 |
|
287 | | -## Related Repositories |
| 76 | +## Related |
288 | 77 |
|
289 | | -- **[datajoint-python](https://github.com/datajoint/datajoint-python)** — Core DataJoint library |
290 | | -- **[DataJoint Elements](https://datajoint.com/docs/elements/)** — Neuroscience pipeline elements |
291 | | -- **[DataJoint Works](https://datajoint.com)** — Company and commercial support |
| 78 | +- [datajoint-python](https://github.com/datajoint/datajoint-python) — Core library |
| 79 | +- [DataJoint Elements](https://docs.datajoint.com/elements/) — Neuroscience pipeline modules |
| 80 | +- [datajoint.com](https://datajoint.com) — Commercial support |
292 | 81 |
|
293 | 82 | ## Citation |
294 | 83 |
|
295 | | -If you use DataJoint in your research, please cite: |
296 | | - |
297 | | -> Yatsenko D, Walker EY, Tolias AS. DataJoint: A Simpler Relational Data Model. |
298 | | -> arXiv:2303.00102. 2023. doi: [10.48550/arXiv.2303.00102](https://doi.org/10.48550/arXiv.2303.00102) |
| 84 | +> Yatsenko D, Walker EY, Tolias AS. DataJoint: A Simpler Relational Data Model. arXiv:2303.00102. 2023. doi: [10.48550/arXiv.2303.00102](https://doi.org/10.48550/arXiv.2303.00102) |
299 | 85 |
|
300 | | -Earlier publication: |
301 | | - |
302 | | -> Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, |
303 | | -> Cotton RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data |
304 | | -> using MATLAB or Python. bioRxiv. 2015. doi: [10.1101/031658](https://doi.org/10.1101/031658) |
| 86 | +Full citation information: [docs.datajoint.com/about/citation/](https://docs.datajoint.com/about/citation/) |
305 | 87 |
|
306 | 88 | ## License |
307 | 89 |
|
308 | | -Documentation: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
309 | | - |
310 | | -DataJoint software: [Apache 2.0](https://github.com/datajoint/datajoint-python/blob/master/LICENSE) |
| 90 | +- Documentation: [CC BY 4.0](https://creativecommons.org/licenses/by/4.0/) |
| 91 | +- DataJoint software: [Apache 2.0](https://github.com/datajoint/datajoint-python/blob/master/LICENSE) (LGPLv2.1 prior to v2.0) |
0 commit comments