Skip to content

Commit 7ce32f6

Browse files
Adjust readme headings.
1 parent 903a2ef commit 7ce32f6

File tree

1 file changed

+59
-51
lines changed

1 file changed

+59
-51
lines changed

README.md

Lines changed: 59 additions & 51 deletions
Original file line numberDiff line numberDiff line change
@@ -6,37 +6,42 @@
66
[![Slack](https://img.shields.io/badge/slack-chat-green.svg)](https://datajoint.slack.com/)
77

88
# Welcome to DataJoint for Python!
9+
910
DataJoint for Python is a framework for scientific workflow management based on relational principles. DataJoint is built on the foundation of the relational data model and prescribes a consistent method for organizing, populating, computing, and querying data.
1011

1112
DataJoint was initially developed in 2009 by Dimitri Yatsenko in Andreas Tolias' Lab at Baylor College of Medicine for the distributed processing and management of large volumes of data streaming from regular experiments. Starting in 2011, DataJoint has been available as an open-source project adopted by other labs and improved through contributions from several developers.
1213
Presently, the primary developer of DataJoint open-source software is the company DataJoint (https://datajoint.com). Related resources are listed at https://datajoint.org.
1314

1415
## Installation
16+
1517
```
1618
pip3 install datajoint
1719
```
1820

1921
If you already have an older version of DataJoint installed using `pip`, upgrade with
22+
2023
```bash
2124
pip3 install --upgrade datajoint
2225
```
2326

2427
## Documentation and Tutorials
2528

26-
* https://datajoint.org -- start page
27-
* https://docs.datajoint.org -- up-to-date documentation
28-
* https://tutorials.datajoint.io -- step-by-step tutorials
29-
* https://elements.datajoint.org -- catalog of example pipelines
30-
* https://codebook.datajoint.io -- interactive online tutorials
29+
- https://datajoint.org -- start page
30+
- https://docs.datajoint.org -- up-to-date documentation
31+
- https://tutorials.datajoint.io -- step-by-step tutorials
32+
- https://elements.datajoint.org -- catalog of example pipelines
33+
- https://codebook.datajoint.io -- interactive online tutorials
3134

3235
## Citation
33-
+ If your work uses DataJoint for Python, please cite the following Research Resource Identifier (RRID) and manuscript.
3436

35-
+ DataJoint ([RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543)) - DataJoint for Python (version `<Enter version number>`)
37+
- If your work uses DataJoint for Python, please cite the following Research Resource Identifier (RRID) and manuscript.
3638

37-
+ Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, Cotton RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data using MATLAB or Python. bioRxiv. 2015 Jan 1:031658. doi: https://doi.org/10.1101/031658
39+
- DataJoint ([RRID:SCR_014543](https://scicrunch.org/resolver/SCR_014543)) - DataJoint for Python (version `<Enter version number>`)
40+
41+
- Yatsenko D, Reimer J, Ecker AS, Walker EY, Sinz F, Berens P, Hoenselaar A, Cotton RJ, Siapas AS, Tolias AS. DataJoint: managing big scientific data using MATLAB or Python. bioRxiv. 2015 Jan 1:031658. doi: https://doi.org/10.1101/031658
3842

3943
## Python Native Blobs
44+
4045
<details>
4146
<summary>Click to expand details</summary>
4247

@@ -45,13 +50,13 @@ The new blobs are a superset of the old functionality and are fully backward com
4550
In previous versions, only MATLAB-style numerical arrays were fully supported.
4651
Some Python datatypes such as dicts were coerced into numpy recarrays and then fetched as such.
4752

48-
However, since some Python types were coerced into MATLAB types, old blobs and new blobs may now be fetched as different types of objects even if they were inserted the same way.
53+
However, since some Python types were coerced into MATLAB types, old blobs and new blobs may now be fetched as different types of objects even if they were inserted the same way.
4954
For example, new `dict` objects will be returned as `dict` while the same types of objects inserted with `datajoint 0.11` will be recarrays.
5055

51-
Since this is a big change, we chose to temporarily disable this feature by default in DataJoint for Python 0.12.x, allowing users to adjust their code if necessary.
56+
Since this is a big change, we chose to temporarily disable this feature by default in DataJoint for Python 0.12.x, allowing users to adjust their code if necessary.
5257
From 13.x, the flag will default to True (on), and will ultimately be removed when corresponding decode support for the new format is added to datajoint-matlab (see: datajoint-matlab #222, datajoint-python #765).
5358

54-
The flag is configured by setting the `enable_python_native_blobs` flag in `dj.config`.
59+
The flag is configured by setting the `enable_python_native_blobs` flag in `dj.config`.
5560

5661
```python
5762
import datajoint as dj
@@ -60,8 +65,8 @@ dj.config["enable_python_native_blobs"] = True
6065

6166
You can safely enable this setting if both of the following are true:
6267

63-
* The only kinds of blobs your pipeline have inserted previously were numerical arrays.
64-
* You do not need to share blob data between Python and MATLAB.
68+
- The only kinds of blobs your pipeline have inserted previously were numerical arrays.
69+
- You do not need to share blob data between Python and MATLAB.
6570

6671
Otherwise, read the following explanation.
6772

@@ -76,91 +81,94 @@ and Python for certain record types. However, this created a discrepancy
7681
between insert and fetch datatypes which could cause problems in other
7782
portions of users pipelines.
7883

79-
DataJoint v0.12, removes the squashing behavior, instead encoding native python datatypes in blobs directly.
80-
However, this change creates a compatibility problem for pipelines
81-
which previously relied on the type squashing behavior since records
84+
DataJoint v0.12, removes the squashing behavior, instead encoding native python datatypes in blobs directly.
85+
However, this change creates a compatibility problem for pipelines
86+
which previously relied on the type squashing behavior since records
8287
saved via the old squashing format will continue to fetch
8388
as structured arrays, whereas new record inserted in DataJoint 0.12 with
8489
`enable_python_native_blobs` would result in records returned as the
8590
appropriate native python type (dict, etc).
8691
Furthermore, DataJoint for MATLAB does not yet support unpacking native Python datatypes.
8792

88-
With `dj.config["enable_python_native_blobs"]` set to `False`,
93+
With `dj.config["enable_python_native_blobs"]` set to `False`,
8994
any attempt to insert any datatype other than a numpy array will result in an exception.
9095
This is meant to get users to read this message in order to allow proper testing
9196
and migration of pre-0.12 pipelines to 0.12 in a safe manner.
9297

9398
The exact process to update a specific pipeline will vary depending on
9499
the situation, but generally the following strategies may apply:
95100

96-
* Altering code to directly store numpy structured arrays or plain
97-
multidimensional arrays. This strategy is likely best one for those
98-
tables requiring compatibility with MATLAB.
99-
* Adjust code to deal with both structured array and native fetched data
100-
for those tables that are populated with `dict`s in blobs in pre-0.12 version.
101-
In this case, insert logic is not adjusted, but downstream consumers
102-
are adjusted to handle records saved under the old and new schemes.
103-
* Migrate data into a fresh schema, fetching the old data, converting blobs to
104-
a uniform data type and re-inserting.
105-
* Drop/Recompute imported/computed tables to ensure they are in the new
106-
format.
101+
- Altering code to directly store numpy structured arrays or plain
102+
multidimensional arrays. This strategy is likely best one for those
103+
tables requiring compatibility with MATLAB.
104+
- Adjust code to deal with both structured array and native fetched data
105+
for those tables that are populated with `dict`s in blobs in pre-0.12 version.
106+
In this case, insert logic is not adjusted, but downstream consumers
107+
are adjusted to handle records saved under the old and new schemes.
108+
- Migrate data into a fresh schema, fetching the old data, converting blobs to
109+
a uniform data type and re-inserting.
110+
- Drop/Recompute imported/computed tables to ensure they are in the new
111+
format.
107112

108113
As always, be sure that your data is safely backed up before modifying any
109114
important DataJoint schema or records.
110115

111116
</details>
112117

113-
### API docs
118+
## API docs
114119

115120
The API documentation can be built with mkdocs using the docker compose file in
116121
`docs/` with the following command:
117122

118-
``` bash
123+
```bash
119124
MODE="LIVE" PACKAGE=datajoint UPSTREAM_REPO=https://github.com/datajoint/datajoint-python.git HOST_UID=$(id -u) docker compose -f docs/docker-compose.yaml up --build
120125
```
121126

122127
The site will then be available at `http://localhost/`. When finished, be sure to run
123128
the same command as above, but replace `up --build` with `down`.
124129

125130
## Running Tests Locally
131+
126132
<details>
127133
<summary>Click to expand details</summary>
128134

129-
* Create an `.env` with desired development environment values e.g.
130-
``` sh
135+
- Create an `.env` with desired development environment values e.g.
136+
137+
```sh
131138
PY_VER=3.9
132139
MYSQL_VER=5.7
133140
DISTRO=alpine
134141
MINIO_VER=RELEASE.2022-01-03T18-22-58Z
135142
HOST_UID=1000
136143
```
137-
* `cp local-docker-compose.yml docker-compose.yml`
138-
* `docker-compose up -d` (Note configured `JUPYTER_PASSWORD`)
139-
* Select a means of running Tests e.g. Docker Terminal, or Local Terminal (see bottom)
140-
* Add entry in `/etc/hosts` for `127.0.0.1 fakeservices.datajoint.io`
141-
* Run desired tests. Some examples are as follows:
142144

143-
| Use Case | Shell Code |
144-
| ---------------------------- | ------------------------------------------------------------------------------ |
145-
| Run all tests | `nosetests -vsw tests --with-coverage --cover-package=datajoint` |
146-
| Run one specific class test | `nosetests -vs --tests=tests.test_fetch:TestFetch.test_getattribute_for_fetch1` |
147-
| Run one specific basic test | `nosetests -vs --tests=tests.test_external_class:test_insert_and_fetch` |
145+
- `cp local-docker-compose.yml docker-compose.yml`
146+
- `docker-compose up -d` (Note configured `JUPYTER_PASSWORD`)
147+
- Select a means of running Tests e.g. Docker Terminal, or Local Terminal (see bottom)
148+
- Add entry in `/etc/hosts` for `127.0.0.1 fakeservices.datajoint.io`
149+
- Run desired tests. Some examples are as follows:
148150

151+
| Use Case | Shell Code |
152+
| --------------------------- | ------------------------------------------------------------------------------- |
153+
| Run all tests | `nosetests -vsw tests --with-coverage --cover-package=datajoint` |
154+
| Run one specific class test | `nosetests -vs --tests=tests.test_fetch:TestFetch.test_getattribute_for_fetch1` |
155+
| Run one specific basic test | `nosetests -vs --tests=tests.test_external_class:test_insert_and_fetch` |
149156

150157
### Launch Docker Terminal
151-
* Shell into `datajoint-python_app_1` i.e. `docker exec -it datajoint-python_app_1 sh`
152158

159+
- Shell into `datajoint-python_app_1` i.e. `docker exec -it datajoint-python_app_1 sh`
153160

154161
### Launch Local Terminal
155-
* See `datajoint-python_app` environment variables in `local-docker-compose.yml`
156-
* Launch local terminal
157-
* `export` environment variables in shell
158-
* Add entry in `/etc/hosts` for `127.0.0.1 fakeservices.datajoint.io`
159162

163+
- See `datajoint-python_app` environment variables in `local-docker-compose.yml`
164+
- Launch local terminal
165+
- `export` environment variables in shell
166+
- Add entry in `/etc/hosts` for `127.0.0.1 fakeservices.datajoint.io`
160167

161168
### Launch Jupyter Notebook for Interactive Use
162-
* Navigate to `localhost:8888`
163-
* Input Jupyter password
164-
* Launch a notebook i.e. `New > Python 3`
165169

166-
</details>
170+
- Navigate to `localhost:8888`
171+
- Input Jupyter password
172+
- Launch a notebook i.e. `New > Python 3`
173+
174+
</details>

0 commit comments

Comments
 (0)