|
45 | 45 | ] |
46 | 46 | }, |
47 | 47 | { |
| 48 | + "attachments": {}, |
48 | 49 | "cell_type": "markdown", |
49 | 50 | "metadata": {}, |
50 | 51 | "source": [ |
51 | 52 | "If you visit the [documentation for DataJoint](https://docs.datajoint.io/introduction/Data-pipelines.html), we define a data pipeline as follows:\n", |
52 | 53 | "> A data pipeline is a sequence of steps (more generally a directed acyclic graph) with integrated storage at each step. These steps may be thought of as nodes in a graph.\n", |
53 | 54 | "\n", |
54 | | - "While this is an accurate description, it may not be the most intuitive definition. Put succinctly, a data pipeline is a listing or a \"map\" of various \"things\" that you work with in a project, with line connecting things to each other to indicate their dependecies. The \"things\" in a data pipeline tends to be the *nouns* you find when describing a project. The \"things\" may include anything from mouse, experimenter, equipment, to experiment session, trial, two-photon scans, electric activities, to receptive fields, neuronal spikes, to figures for a publication! A data pipeline gives you a framework to:\n", |
| 55 | + "While this is an accurate description, it may not be the most intuitive definition. Put succinctly, a data pipeline is a listing or a \"map\" of various \"things\" that you work with in a project, with line connecting things to each other to indicate their dependencies. The \"things\" in a data pipeline tends to be the *nouns* you find when describing a project. The \"things\" may include anything from mouse, experimenter, equipment, to experiment session, trial, two-photon scans, electric activities, to receptive fields, neuronal spikes, to figures for a publication! A data pipeline gives you a framework to:\n", |
55 | 56 | "\n", |
56 | 57 | "1. define these \"things\" as tables in which you can store the information about them\n", |
57 | 58 | "2. define the relationships (in particular the dependencies) between the \"things\"\n", |
|
76 | 77 | ] |
77 | 78 | }, |
78 | 79 | { |
| 80 | + "attachments": {}, |
79 | 81 | "cell_type": "markdown", |
80 | 82 | "metadata": {}, |
81 | 83 | "source": [ |
|
86 | 88 | "> * In a session of electrophysiology\n", |
87 | 89 | ">> * you record electrical activity from a single neuron. You use recording equipment that produces separate data files for each neuron you recorded.\n", |
88 | 90 | ">> * Neuron's activities are recorded as raw traces. Neuron's spikes needs to be detected for further analysis to be performed.\n", |
89 | | - "> * In a sesssion of calcium imaging\n", |
| 91 | + "> * In a session of calcium imaging\n", |
90 | 92 | ">> * you scan a brain region containing a number of neurons. You use recording equipment that produces separate data files for each scan you performed.\n", |
91 | 93 | ">> * you would like to segment the frames and get the regions of interest (ROIs), and save a mask for each ROI\n", |
92 | 94 | ">> * finally you would like to extract the trace from each segmented ROI" |
|
107 | 109 | ] |
108 | 110 | }, |
109 | 111 | { |
| 112 | + "attachments": {}, |
110 | 113 | "cell_type": "markdown", |
111 | 114 | "metadata": {}, |
112 | 115 | "source": [ |
|
117 | 120 | "> * In a session of electrophysiology\n", |
118 | 121 | ">> * you record electrical activity from a **single neuron**. You use recording equipment that produces separate data files for each neuron you recorded.\n", |
119 | 122 | ">> * Neuron's activities are recorded as raw traces. **Neuron's spikes** needs to be detected for further analysis to be performed.\n", |
120 | | - "> * In a sesssion of calcium imaging\n", |
| 123 | + "> * In a session of calcium imaging\n", |
121 | 124 | ">> * you scan a brain region containing a number of neurons. You use recording equipment that produces separate data files for each **scan** you performed.\n", |
122 | 125 | ">> * you would like to segment the frames and get the **regions of interest (ROIs)**, and save a mask for each ROI\n", |
123 | 126 | ">> * finally you would like to extract the **trace** from each segmented ROI" |
|
225 | 228 | ] |
226 | 229 | }, |
227 | 230 | { |
| 231 | + "attachments": {}, |
228 | 232 | "cell_type": "markdown", |
229 | 233 | "metadata": {}, |
230 | 234 | "source": [ |
231 | | - "We create the schema using `dj.schema()` function, passing in the name of the schema. For this workshop, you are given the database privilege to create any schema **starting with your username followed by a `_` charcter**. So if your username is `john`, you can make any schema starting with `john_`, such as `john_tutorial`." |
232 | | - ] |
233 | | - }, |
234 | | - { |
235 | | - "cell_type": "markdown", |
236 | | - "metadata": {}, |
237 | | - "source": [ |
238 | | - "Let's create a schema called `pipeline`, prefixed by `username_`." |
| 235 | + "We create the schema using `dj.schema()` function, passing in the name of the schema. For this workshop, you are given the database privilege to use any schema name. Let's create a schema called `tutorial`." |
239 | 236 | ] |
240 | 237 | }, |
241 | 238 | { |
|
252 | 249 | } |
253 | 250 | ], |
254 | 251 | "source": [ |
255 | | - "schema = dj.schema('{YOUR_USERNAME}_tutorial')" |
| 252 | + "schema = dj.schema('tutorial')" |
256 | 253 | ] |
257 | 254 | }, |
258 | 255 | { |
|
1059 | 1056 | ] |
1060 | 1057 | }, |
1061 | 1058 | { |
| 1059 | + "attachments": {}, |
1062 | 1060 | "cell_type": "markdown", |
1063 | 1061 | "metadata": {}, |
1064 | 1062 | "source": [ |
1065 | 1063 | "Note that, to uniquely identify an experimental session (or simply a **session**), we need to know the mouse that the session was about. In other words, a session cannot existing without a corresponding mouse! \n", |
1066 | 1064 | "\n", |
1067 | | - "With **mouse** already represented as a table in our pipeline, we say that the session **depends on** the mouse! We could graphically represent this in an **entity relationship diagram (ERD)** by drawing the line between two tables, with the one below (**session**) dependeing on the one above (**mouse**)." |
| 1065 | + "With **mouse** already represented as a table in our pipeline, we say that the session **depends on** the mouse! We could graphically represent this in an **entity relationship diagram (ERD)** by drawing the line between two tables, with the one below (**session**) depending on the one above (**mouse**)." |
1068 | 1066 | ] |
1069 | 1067 | }, |
1070 | 1068 | { |
|
1109 | 1107 | "outputs": [ |
1110 | 1108 | { |
1111 | 1109 | "data": { |
1112 | | - "image/svg+xml": [ |
1113 | | - "<svg height=\"114pt\" viewBox=\"0.00 0.00 66.69 114.00\" width=\"67pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n", |
1114 | | - "<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 110)\">\n", |
1115 | | - "<title>%3</title>\n", |
1116 | | - "<polygon fill=\"#ffffff\" points=\"-4,4 -4,-110 62.6914,-110 62.6914,4 -4,4\" stroke=\"transparent\"/>\n", |
1117 | | - "<!-- Mouse -->\n", |
1118 | | - "<g class=\"node\" id=\"node1\">\n", |
1119 | | - "<title>Mouse</title>\n", |
1120 | | - "<g id=\"a_node1\"><a xlink:title=\"mouse_id \r", |
1121 | | - "------------------------------\r", |
1122 | | - "dob=null \r", |
1123 | | - "sex="unknown" \r", |
1124 | | - "\">\n", |
1125 | | - "<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" points=\"55.3633,-106 3.3281,-106 3.3281,-71 55.3633,-71 55.3633,-106\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\"/>\n", |
1126 | | - "<text fill=\"#006400\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"11.3369\" y=\"-86.7994\">Mouse</text>\n", |
1127 | | - "</a>\n", |
1128 | | - "</g>\n", |
1129 | | - "</g>\n", |
1130 | | - "<!-- Session -->\n", |
1131 | | - "<g class=\"node\" id=\"node2\">\n", |
1132 | | - "<title>Session</title>\n", |
1133 | | - "<g id=\"a_node2\"><a xlink:title=\"→ Mouse\r", |
1134 | | - "session_date \r", |
1135 | | - "------------------------------\r", |
1136 | | - "experiment_setup \r", |
1137 | | - "experimenter \r", |
1138 | | - "data_path="" \r", |
1139 | | - "\">\n", |
1140 | | - "<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" points=\"58.5379,-35 .1535,-35 .1535,0 58.5379,0 58.5379,-35\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\"/>\n", |
1141 | | - "<text fill=\"#006400\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"8\" y=\"-15.7994\">Session</text>\n", |
1142 | | - "</a>\n", |
1143 | | - "</g>\n", |
1144 | | - "</g>\n", |
1145 | | - "<!-- Mouse->Session -->\n", |
1146 | | - "<g class=\"edge\" id=\"edge1\">\n", |
1147 | | - "<title>Mouse->Session</title>\n", |
1148 | | - "<path d=\"M29.3457,-70.9494C29.3457,-60.1083 29.3457,-46.2366 29.3457,-35.3418\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\".75\"/>\n", |
1149 | | - "</g>\n", |
1150 | | - "</g>\n", |
1151 | | - "</svg>" |
1152 | | - ], |
| 1110 | + "image/svg+xml": "<svg height=\"114pt\" viewBox=\"0.00 0.00 66.69 114.00\" width=\"67pt\" xmlns=\"http://www.w3.org/2000/svg\" xmlns:xlink=\"http://www.w3.org/1999/xlink\">\n<g class=\"graph\" id=\"graph0\" transform=\"scale(1 1) rotate(0) translate(4 110)\">\n<title>%3</title>\n<polygon fill=\"#ffffff\" points=\"-4,4 -4,-110 62.6914,-110 62.6914,4 -4,4\" stroke=\"transparent\"/>\n<!-- Mouse -->\n<g class=\"node\" id=\"node1\">\n<title>Mouse</title>\n<g id=\"a_node1\"><a xlink:title=\"mouse_id \r\n------------------------------\r\ndob=null \r\nsex="unknown" \r\n\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" points=\"55.3633,-106 3.3281,-106 3.3281,-71 55.3633,-71 55.3633,-106\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\"/>\n<text fill=\"#006400\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"11.3369\" y=\"-86.7994\">Mouse</text>\n</a>\n</g>\n</g>\n<!-- Session -->\n<g class=\"node\" id=\"node2\">\n<title>Session</title>\n<g id=\"a_node2\"><a xlink:title=\"→ Mouse\r\nsession_date \r\n------------------------------\r\nexperiment_setup \r\nexperimenter \r\ndata_path="" \r\n\">\n<polygon fill=\"#00ff00\" fill-opacity=\"0.188235\" points=\"58.5379,-35 .1535,-35 .1535,0 58.5379,0 58.5379,-35\" stroke=\"#00ff00\" stroke-opacity=\"0.188235\"/>\n<text fill=\"#006400\" font-family=\"arial\" font-size=\"12.00\" text-anchor=\"start\" text-decoration=\"underline\" x=\"8\" y=\"-15.7994\">Session</text>\n</a>\n</g>\n</g>\n<!-- Mouse->Session -->\n<g class=\"edge\" id=\"edge1\">\n<title>Mouse->Session</title>\n<path d=\"M29.3457,-70.9494C29.3457,-60.1083 29.3457,-46.2366 29.3457,-35.3418\" fill=\"none\" stroke=\"#000000\" stroke-opacity=\"0.250980\" stroke-width=\".75\"/>\n</g>\n</g>\n</svg>", |
1153 | 1111 | "text/plain": [ |
1154 | 1112 | "<datajoint.diagram.Diagram at 0xa1d699588>" |
1155 | 1113 | ] |
|
1160 | 1118 | } |
1161 | 1119 | ], |
1162 | 1120 | "source": [ |
1163 | | - "dj.ERD(schema)" |
| 1121 | + "dj.Diagram(schema)" |
1164 | 1122 | ] |
1165 | 1123 | }, |
1166 | 1124 | { |
|
4052 | 4010 | ] |
4053 | 4011 | }, |
4054 | 4012 | { |
| 4013 | + "attachments": {}, |
4055 | 4014 | "cell_type": "markdown", |
4056 | 4015 | "metadata": {}, |
4057 | 4016 | "source": [ |
4058 | 4017 | "### Rename attribute with proj()\n", |
4059 | | - "Say we want to rename the exisiting attribute `dob` of the `Mouse` table to `date_of_birth`, this can be done using `.proj()`" |
| 4018 | + "Say we want to rename the existing attribute `dob` of the `Mouse` table to `date_of_birth`, this can be done using `.proj()`" |
4060 | 4019 | ] |
4061 | 4020 | }, |
4062 | 4021 | { |
|
4284 | 4243 | ] |
4285 | 4244 | }, |
4286 | 4245 | { |
| 4246 | + "attachments": {}, |
4287 | 4247 | "cell_type": "markdown", |
4288 | 4248 | "metadata": {}, |
4289 | 4249 | "source": [ |
4290 | | - "Note: as you can see, the projection results keep the primary attributes from the `Mouse * Session` joinning operation, while removing all other non-primary attributes. To Keep all other attributes, you can use the `...` syntax" |
| 4250 | + "Note: as you can see, the projection results keep the primary attributes from the `Mouse * Session` joining operation, while removing all other non-primary attributes. To Keep all other attributes, you can use the `...` syntax" |
4291 | 4251 | ] |
4292 | 4252 | }, |
4293 | 4253 | { |
|
0 commit comments