You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
DataJoint for Python is a framework for scientific workflow management based on relational principles. DataJoint is built on the foundation of the relational data model and prescribes a consistent method for organizing, populating, computing, and querying data.
11
11
12
12
DataJoint was initially developed in 2009 by Dimitri Yatsenko in Andreas Tolias' Lab at Baylor College of Medicine for the distributed processing and management of large volumes of data streaming from regular experiments. Starting in 2011, DataJoint has been available as an open-source project adopted by other labs and improved through contributions from several developers.
13
-
Presently, the primary developer of DataJoint open-source software is the company DataJoint (https://datajoint.com). Related resources are listed at https://datajoint.org.
13
+
Presently, the primary developer of DataJoint open-source software is the company DataJoint (https://datajoint.com).
- DataJoint ([RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543)) - DataJoint for Python (version `<Enter version number>`)
40
32
41
33
- Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, Cotton RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data using MATLAB or Python. bioRxiv. 2015 Jan 1:031658. doi: https://doi.org/10.1101/031658
42
-
43
-
## Python Native Blobs
44
-
45
-
<details>
46
-
<summary>Click to expand details</summary>
47
-
48
-
DataJoint 0.12 adds full support for all native python data types in blobs: tuples, lists, sets, dicts, strings, bytes, `None`, and all their recursive combinations.
49
-
The new blobs are a superset of the old functionality and are fully backward compatible.
50
-
In previous versions, only MATLAB-style numerical arrays were fully supported.
51
-
Some Python datatypes such as dicts were coerced into numpy recarrays and then fetched as such.
52
-
53
-
However, since some Python types were coerced into MATLAB types, old blobs and new blobs may now be fetched as different types of objects even if they were inserted the same way.
54
-
For example, new `dict` objects will be returned as `dict` while the same types of objects inserted with `datajoint 0.11` will be recarrays.
55
-
56
-
Since this is a big change, we chose to temporarily disable this feature by default in DataJoint for Python 0.12.x, allowing users to adjust their code if necessary.
57
-
From 13.x, the flag will default to True (on), and will ultimately be removed when corresponding decode support for the new format is added to datajoint-matlab (see: datajoint-matlab #222, datajoint-python #765).
58
-
59
-
The flag is configured by setting the `enable_python_native_blobs` flag in `dj.config`.
60
-
61
-
```python
62
-
import datajoint as dj
63
-
dj.config["enable_python_native_blobs"] =True
64
-
```
65
-
66
-
You can safely enable this setting if both of the following are true:
67
-
68
-
- The only kinds of blobs your pipeline have inserted previously were numerical arrays.
69
-
- You do not need to share blob data between Python and MATLAB.
70
-
71
-
Otherwise, read the following explanation.
72
-
73
-
DataJoint v0.12 expands DataJoint's blob serialization mechanism with
74
-
improved support for complex native python datatypes, such as dictionaries
75
-
and lists of strings.
76
-
77
-
Prior to DataJoint v0.12, certain python native datatypes such as
78
-
dictionaries were 'squashed' into numpy structured arrays when saved into
79
-
blob attributes. This facilitated easier data sharing between MATLAB
80
-
and Python for certain record types. However, this created a discrepancy
81
-
between insert and fetch datatypes which could cause problems in other
82
-
portions of users pipelines.
83
-
84
-
DataJoint v0.12, removes the squashing behavior, instead encoding native python datatypes in blobs directly.
85
-
However, this change creates a compatibility problem for pipelines
86
-
which previously relied on the type squashing behavior since records
87
-
saved via the old squashing format will continue to fetch
88
-
as structured arrays, whereas new record inserted in DataJoint 0.12 with
89
-
`enable_python_native_blobs` would result in records returned as the
90
-
appropriate native python type (dict, etc).
91
-
Furthermore, DataJoint for MATLAB does not yet support unpacking native Python datatypes.
92
-
93
-
With `dj.config["enable_python_native_blobs"]` set to `False`,
94
-
any attempt to insert any datatype other than a numpy array will result in an exception.
95
-
This is meant to get users to read this message in order to allow proper testing
96
-
and migration of pre-0.12 pipelines to 0.12 in a safe manner.
97
-
98
-
The exact process to update a specific pipeline will vary depending on
99
-
the situation, but generally the following strategies may apply:
100
-
101
-
- Altering code to directly store numpy structured arrays or plain
102
-
multidimensional arrays. This strategy is likely best one for those
103
-
tables requiring compatibility with MATLAB.
104
-
- Adjust code to deal with both structured array and native fetched data
105
-
for those tables that are populated with `dict`s in blobs in pre-0.12 version.
106
-
In this case, insert logic is not adjusted, but downstream consumers
107
-
are adjusted to handle records saved under the old and new schemes.
108
-
- Migrate data into a fresh schema, fetching the old data, converting blobs to
109
-
a uniform data type and re-inserting.
110
-
- Drop/Recompute imported/computed tables to ensure they are in the new
111
-
format.
112
-
113
-
As always, be sure that your data is safely backed up before modifying any
114
-
important DataJoint schema or records.
115
-
116
-
</details>
117
-
118
-
## API docs
119
-
120
-
The API documentation can be built with mkdocs using the docker compose file in
The following will ensure the codebase has been formatted with [black](https://black.readthedocs.io/en/stable/).
57
+
58
+
```
59
+
black datajoint --check -v
60
+
```
61
+
62
+
The following will ensure the test suite has been formatted with [black](https://black.readthedocs.io/en/stable/).
63
+
64
+
```
65
+
black tests --check -v
66
+
```
67
+
68
+
### Jupyter
69
+
70
+
Jupyter notebooks are supported in this environment. This means that when you `import datajoint`, it will use the current state of the source.
71
+
72
+
Be sure to see the reference documenation if you are new to [running Jupyter notebooks w/ VSCode](https://code.visualstudio.com/docs/datascience/jupyter-notebooks#_create-or-open-a-jupyter-notebook).
73
+
74
+
### Debugger
75
+
76
+
[VSCode Debugger](https://code.visualstudio.com/docs/editor/debugging) is a powerful tool that can really accelerate fixes.
77
+
78
+
Try it as follows:
79
+
80
+
- Create a python script of your choice
81
+
-`import datajoint` (This will use the current state of the source)
82
+
- Add breakpoints by adding red dots next to line numbers
83
+
- Select the `Run and Debug` tab
84
+
- Start by clicking the button `Run and Debug`
85
+
86
+
### MySQL CLI
87
+
88
+
It is often useful in development to connect to DataJoint's relational database backend directly using the MySQL CLI.
89
+
90
+
Connect as follows to the database running within your developer environment:
91
+
92
+
```
93
+
mysql -hfakeservices.datajoint.io -uroot -psimple
94
+
```
95
+
96
+
### Documentation
97
+
98
+
Our documentation is built using [MkDocs Material](https://squidfunk.github.io/mkdocs-material/). The easiest way to improve the documentation is by using the `docs/docker-compose.yaml` environment. The source can be modified in `docs/src` using markdown.
99
+
100
+
The docs environment can be run using 3 modes:
101
+
102
+
-**LIVE**: (*recommended*) This serves the docs locally. It supports live reloading on saves to `docs/src` files but does not support the docs version dropdown. Useful to see changes live.
When the docs are served locally, use the VSCode `PORTS` tab (next to `TERMINAL`) to manage access to the forwarded ports. Docs are served on port `8080`.
0 commit comments