Skip to content

Commit a698bbf

Browse files
jcb91jcb91
authored
Merge pull request #1213 from benelot/execution_dependencies
Execution dependency extension
2 parents 6320b74 + 0adfb29 commit a698bbf

File tree

4 files changed

+194
-0
lines changed

4 files changed

+194
-0
lines changed

‎CHANGELOG.md

Lines changed: 3 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -50,6 +50,9 @@ Repo-level stuff:
5050

5151
New features and bugfixes:
5252

53+
- `execution_dependencies` __new nbextension added!__
54+
[#1213](https://github.com/ipython-contrib/jupyter_contrib_nbextensions/pull/1213)
55+
[@benelot](https://github.com/benelot)
5356
- `livemdpreview` __new nbextension added!__
5457
[#1155](https://github.com/ipython-contrib/jupyter_contrib_nbextensions/pull/1155)
5558
[@jcb91](https://github.com/jcb91)

‎src/jupyter_contrib_nbextensions/nbextensions/execution_dependencies/README.md

Lines changed: 43 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,43 @@
1+
execution_dependencies
2+
======================
3+
4+
Writing extensive notebooks can become very complicated since many cells act as stepping stones to produce intermediate results for later cells. Thus, it becomes tedious to
5+
keep track of the cells that have to be run in order to run a certain cell. This extension simplifies handling the execution dependencies by introducing tag annotations to
6+
identify each cell and indicate a dependency on others. This improves on the current state which requires remembering all dependencies by heart or annotating the cells in the comments.
7+
8+
If a cell with dependencies is run, the extension checks recursively for all dependencies of the cell, then executes them before executing the cell after all the dependencies have finished.
9+
Dependencies are definitely executed and not only once per kernel session.
10+
11+
The two annotations are added to the tags of a cell and are as follows:
12+
13+
* add a hashmark (#) and an identification tag to the tags to identify a cell (e.g. #initializer-cell). The #identifiers must be unique among all cells.
14+
* add an arrow (=>) and an identification tag to the tags to add a dependency on a certain cell (e.g. =>initializer-cell).
15+
16+
Based on these dependencies, the kernel will now execute the dependencies before the cell that depends on them. If the cell's dependencies have further dependencies, these will in turn
17+
be executed before them. In conclusion, the kernel looks through the tree of dependencies of the cell executed by the user and executes its dependencies in their appropriate order,
18+
then executes the cell.
19+
20+
A more extensive example is described below:
21+
22+
A cell A has the identifier #A.
23+
24+
| Cell A [tags: #A] |
25+
| ------------- |
26+
| Content Cell |
27+
| Content Cell |
28+
29+
30+
A cell B has the identifier #B and depends on A (=>A).
31+
32+
33+
| Cell B [tags: #B, =>A] |
34+
| ------------- |
35+
| Content Cell |
36+
| Content Cell |
37+
38+
If the user runs A, only A is executed, since it has no dependencies. On the other hand, if the user runs B, the kernel finds the dependency on A, and thus first runs A and then runs B.
39+
40+
Running a cell C that is dependent on B and on A as well, the kernel then first runs A and then runs B before running C, avoiding to run cell A twice.
41+
42+
43+
If you are missing anything, open up an issue at the repository prepending [execute_dependencies] to the title.

‎src/jupyter_contrib_nbextensions/nbextensions/execution_dependencies/execution_dependencies.js

Lines changed: 139 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,139 @@
1+
/**
2+
* execution_dependencies.js
3+
* Introduce tag annotations to identify each cell and indicate a dependency on others.
4+
* Upon running a cell, its dependencies are run first to prepare all dependencies.
5+
* Then the cell triggered by the user is run as soon as all its dependencies are met.
6+
*
7+
*
8+
* @version 0.1.0
9+
* @author Benjamin Ellenberger, https://github.com/benelot
10+
* @updated 2018-01-31
11+
*
12+
*
13+
*/
14+
define([
15+
'jquery',
16+
'base/js/dialog',
17+
'base/js/namespace',
18+
'notebook/js/codecell'
19+
], function (
20+
$,
21+
dialog,
22+
Jupyter,
23+
codecell
24+
) {
25+
"use strict";
26+
27+
var CodeCell = codecell.CodeCell;
28+
29+
return {
30+
load_ipython_extension: function () {
31+
console.log('[execution_dependencies] patching CodeCell.execute');
32+
var orig_execute = codecell.CodeCell.prototype.execute; // keep original cell execute function
33+
CodeCell.prototype.execute = function (stop_on_error) {
34+
var root_tags = this.metadata.tags || []; // get tags of the cell executed by the user (root cell)
35+
if(root_tags.some(tag => /=>.*/.test(tag))) { // if the root cell contains any dependencies, resolve dependency tree
36+
var root_cell = this;
37+
var root_cell_id = root_cell.cell_id;
38+
var cells_with_id = Jupyter.notebook.get_cells().filter(function (cell, idx, cells) { // ...get all cells which have at least one id (these are the only ones we could have in deps)
39+
var tags = cell.metadata.tags || [];
40+
return (cell === root_cell || tags.some(tag => /#.*/.test(tag)));
41+
});
42+
43+
console.log('[execution_dependencies] collecting ids and dependencies...');
44+
var cell_map = {}
45+
var dep_graph = {}
46+
cells_with_id.forEach(function (cell) { // ...get all identified cells (the ones that have at least one #tag)
47+
var tags = cell.metadata.tags || [];
48+
var cell_ids = tags.filter(tag => /#.*/.test(tag)).map(tag => tag.substring(1)); // ...get all identifiers of the current cell and drop the #
49+
if(cell === root_cell){
50+
if(cell_ids.length < 1) {
51+
cell_ids.push(root_cell.cell_id); // ...use internal root cell id for internal usage
52+
}
53+
else {
54+
root_cell_id = cell_ids[0]; // get any of the root cell ids
55+
}
56+
}
57+
58+
var dep_ids = tags.filter(tag => /=>.*/.test(tag)).map(tag => tag.substring(2)); // ...get all dependencies and drop the =>
59+
60+
cell_ids.forEach(function (id) {
61+
//console.log('ID:', id, 'deps: ', dep_ids.toString())
62+
cell_map[id] = cell;
63+
dep_graph[id] = dep_ids;
64+
65+
});
66+
});
67+
68+
if(dep_graph[root_cell_id].length > 0) {
69+
console.log('[execution_dependencies] collecting depdendency graph in-degrees...');
70+
var processing_queue = [root_cell_id];
71+
var processed_nodes = 0;
72+
var in_degree = {}; // ...collect in-degrees of nodes
73+
while(processing_queue.length > 0 && processed_nodes < Object.keys(dep_graph).length) {// ...stay processing deps while the queue contains nodes and the processed nodes are below total node quantity
74+
var id = processing_queue.shift(); // .....pop front of queue and front-push it to the processing order
75+
//console.log("ID: ", id);
76+
for(var i=0, dep_qty=dep_graph[id].length; i < dep_qty; i++) {
77+
var dep = dep_graph[id][i];
78+
// console.log(' dep: ', dep);
79+
in_degree[id] = in_degree[id] || 0;
80+
in_degree[dep] = in_degree[dep] === undefined ? 1 : ++in_degree[dep];
81+
processing_queue.unshift(dep);
82+
}
83+
processed_nodes++;
84+
}
85+
86+
console.log('[execution_dependencies] starting topological sort...');
87+
processing_queue = [root_cell_id]; // ...add root node with in-degree 0 to queue (this excludes all disconnected subgraphs)
88+
processed_nodes = 0; // ...number of processed nodes (to detect circular dependencies)
89+
var processing_order = [];
90+
while(processing_queue.length > 0 && processed_nodes < Object.keys(dep_graph).length) {// ...stay processing deps while the queue contains nodes and the processed nodes are below total node quantity
91+
var id = processing_queue.shift(); // .....pop front of queue and front-push it to the processing order
92+
processing_order.unshift(id);
93+
//console.log("ID: ", id);
94+
for(var i=0, dep_qty=dep_graph[id].length; i < dep_qty; i++) { // ......iterate over dependent nodes of current id and decrease their in-degree by 1
95+
var dep = dep_graph[id][i];
96+
// console.log(' dep: ', dep);
97+
in_degree[dep]--;
98+
if(in_degree[dep] == 0) { // ......queue dependency if in-degree is 0
99+
processing_queue.unshift(dep);
100+
}
101+
}
102+
processed_nodes++;
103+
}
104+
105+
console.log('[execution_dependencies] checking for circular dependencies...');
106+
if(processed_nodes > Object.keys(dep_graph).length) { // ...if more nodes where processed than the number of graph nodes, there is a circular dependency
107+
dialog.modal({
108+
title : 'Circular dependencies in the execute dependencies of this cell',
109+
body : 'There is a circular dependency in this cell\'s execute dependencies. The cell will be run without dependencies. If this does not work, fix the dependencies and rerun the cell.',
110+
buttons: {'OK': {'class' : 'btn-primary'}},
111+
notebook: Jupyter.notebook,
112+
keyboard_manager: Jupyter.keyboard_manager,
113+
});
114+
}
115+
else if(!Jupyter.notebook.trusted) { // ...if the notebook is not trusted, we do not execute dependencies, but only print them out to the user
116+
dialog.modal({
117+
title : 'Execute dependencies in untrusted notebook',
118+
body : 'This notebook is not trusted, so execute dependencies will not be automatically run. You can still run them manually, though. Run in order (the last one is the cell you wanted to execute): ' + processing_order,
119+
buttons: {'OK': {'class' : 'btn-primary'}},
120+
notebook: Jupyter.notebook,
121+
keyboard_manager: Jupyter.keyboard_manager,
122+
});
123+
}
124+
else{
125+
processing_order.pop()
126+
console.log('[execution_dependencies] executing dependency cells in order ', processing_order ,'...');
127+
var dependency_cells = processing_order.map(id =>cell_map[id]); // ...get dependent cells by their id
128+
//console.log("Execute cells..", dependency_cells)
129+
dependency_cells.forEach(cell => orig_execute.call(cell, stop_on_error)); // ...execute all dependent cells in sequence using the original execute method
130+
}
131+
}
132+
}
133+
console.log('[execution_dependencies] executing requested cell...');
134+
orig_execute.call(this, stop_on_error); // execute original cell execute function
135+
};
136+
console.log('[execution_dependencies] loaded');
137+
}
138+
};
139+
});

‎src/jupyter_contrib_nbextensions/nbextensions/execution_dependencies/execution_dependencies.yml

Lines changed: 9 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,9 @@
1+
Type: Jupyter Notebook Extension
2+
Compatibility: 4.x, 5.x
3+
Name: Execution Dependencies
4+
Main: execution_dependencies.js
5+
Link: README.md
6+
Description: |
7+
Introduce tag annotations to identify each cell and indicate a dependency on others.
8+
Upon running a cell, its dependencies are run first to prepare all dependencies.
9+
Then the cell triggered by the user is run as soon as all its dependencies are met.

0 commit comments

Comments
 (0)