Skip to content

Commit 890b047

Browse files
ankostisankostis
committed
DOC(TODOs) Augment & UnComment (publicize)
1 parent f87c787 commit 890b047

File tree

1 file changed

+173
-117
lines changed

1 file changed

+173
-117
lines changed

‎CHANGES.rst‎

Lines changed: 173 additions & 117 deletions
Original file line numberDiff line numberDiff line change
@@ -12,123 +12,179 @@ Graphtik Changelog
1212

1313
</details>
1414

15-
..
16-
TODOs
17-
%%%%%
18-
<hidden from RTD docs, as comment>
19-
20-
- [+] ENH: planning reports unsatisfied node modus, for plotting
21-
- [+] ENH: Plan accepts previous solutions
22-
- [+] fix: FnOp.compute() should (or should not?) accept Pipeline.compute() args.
23-
- [+] FEAT: ``compute(solution, recompute_from=["input_1"])``
24-
25-
- Configs:
26-
- [ ] FEAT: drop MANY contextvars with 1 cfg-dict in contextvars?
27-
- [ ] FEAT: drop contextvars, use merging-dicts on (cfg, pipe, op) instead?
28-
BUT then cannot change configs amidst run, unless next todo ...
29-
- [ ] FEAT: configs & op attributes accept 2-tuples: (flag, priority)
30-
default priorities: cfg, fnop
31-
(+cat: Compose)
32-
- [ ] FEAT: Config DEBUG flags:
33-
- [ ] skip - evictions(drop config)
34-
- [ ] keep SFX in outputs
35-
36-
- Compose:
37-
- [+] feat: add a real ``implicit`` modifier.
38-
- [?] REFACT: separate op-decorator from factory (to facilitate defining conveyor operations).
39-
- [?] `cwd()` modifier:
40-
Isn't already working with relative jsonps?
41-
- [ ] ++FEAT: Teach pipelines how to accept positional arguments with a tuple with their names
42-
- [ ] ++FEAT: +1 merge method for pipelines: nest=False: treat Pipelines as Operations
43-
(need pipeline feat for positional-args, above).
44-
- [ ] break/enh(modifier): augment modifier table with `implicits` for REPR.
45-
- [ ] BREAK(modif): "stuffy" names for `is_xxx(modifier)` funcs returning stuff.
46-
- [ ] +++FEAT: Autograph functions by type-annotations
47-
(PY39 :pep:`593` e.g. ``Annotated[int, CWD]``)
48-
49-
- Planning:
50-
- [ ] FEAT: break cycles with dijkstra; +weights?
51-
- [ ] ++ENH: link graph inputs/outputs to virtual root-nodes, to replace custom visiting algos
52-
(eg unsatisfied-ops, prune-by-outs)
53-
- [ ] FEAT: per-operation EVICTs
54-
(+cat: Compose)
55-
- [ ] ++DROP PARALLEL: always produce a list of "parallelizable batches",
56-
to hook with other executors, and keep here just the single-process implementation.
57-
(+cat: Execution)
58-
59-
- Execute:
60-
- [+] FEAT: +2 callbacks before marshalling: (pre_batch, post_batch) (olds: pre_op, post_op).
61-
PARTIAL: moved existing x2 callbacks to be called before marshalling.
62-
- [?] refact: named_inputs --> sol
63-
- [ ] ENH/REFACT: use Signature.Param from `inspect` module to match needs & zip provides
64-
- [ ] Solution-retriever modifier `opcb()`;
65-
WONTFIX: easier and more generic to access solution from Op-context.
66-
REINSTATED: bc it's simpler and does not have any threading issues....
67-
- [ ] ++FEAT(fnop): vararg(s) for Outs collect all outs to the very end
68-
- [ ] BREAK/FEAT: allow marking SFX-LIST items:
69-
1. Implicits (default), to be checked/resolved downstream (see "FEAT: +2 callbacks"), or
70-
2. Sfx (pure), as they are now).
71-
- [ ] +++BREAK/DROP accessors for `jsonp` (move all functionality in solution)
72-
REJECTED already used for hcat()/vcat()
73-
REINSTATED as "ACCESSOR per jsonp-PART"! (h/vcat should have been like that)
74-
by default applied to the last part (or use ``[acc, ...]`` syntax for specific parts)
75-
76-
- [ ] RENAME ``Solution --> DataTree``
77-
- [ ] move dep_prefixed() as a top-level `modifier` utility (to prefix also Accessors)
78-
- [ ] revive "stable sort paths" branch on solution updates
79-
- [ ] hdf/vdf accessor example-utilities for typical level-0 resolving multi-indexed slices.
80-
- [ ] Drop int/attribute indexers from resolve-jsonp, accessors-per-part can do that.
81-
82-
- **Solution:**
83-
- [+] DROP/ENH: Solution updates GivenInputs only, yet still layers jsonp-refer to its values.
84-
- [+] FEAT: ``pd.concat()`` --> modifier+accessor, to avoid sfxed for multi-column updates.
85-
- [+] FEAT: + `post_callback` with `pre_callback` --> `callbacks` tuple
86-
(+cat: Compose)
87-
- [+] ENH(jsonp): mass-concat (not one-by-one).
88-
- [+] ENH: validate Implicits indeed added (in `post_op_callback`)?
89-
- [ ] FEAT: VALIDATE (by user) items on Solution-Insert (with a `post_callback`?):
90-
- auto-assertions {jsonp--> validators}
91-
- compare overwrites while recomputing
92-
- check implicits exist (??not done above??)
93-
94-
- plot:
95-
- [+] Badges on Data
96-
- [ ] generate legend dynamically
97-
- [ ] SPHINXEXT: autodoc Pipelines & Ops
98-
- [ ] sphinxext: extend standard `doctest` module (instead of sphinx-builder)
99-
100-
- Docs:
101-
- [ ] REFACT: move GitHub organization `pygraphkit --> pygraphtik` (+Travis, +RTD)
102-
- [ ] Merge tutorial (operations + composition)
103-
- [ ] +++DOC: explain Implicits in tutorial; mention ovunque.
104-
- [ ] TCs: Test DEBUG
105-
106-
107-
- DROPPED
108-
- [X] `solution.executed` pre-populated with all operations
109-
- [X] parallel batches restart from last position in steps
110-
- [X] covert custom op classes & modifiers directly into mergeable networkx graphs;
111-
WONTFIX bc foreign function would not work with merged deps.
112-
- [X] conditionals
113-
WONTFIX bc it permits pipelines with too complex execution flow to debug.
114-
- [X] DROP accessors:
115-
REJECTED already used for hcat()/vcat() (but to be dropped for jsnop).
116-
117-
- v9.0.0
118-
- [X] Accept jsonp inputs & outputs,
119-
WONTFIX user's business to expand into given Inputs, Outputs already working.
120-
- [X] REVERT rename subdocs;
121-
WONTFIX bc eventually made it work correctly and added TC.
122-
- [X] REFACT: separate op-decorator from factory
123-
(to facilitate defining conveyor operations):
124-
NO, simplify passing fn=None.
125-
- [X] Nest-rename subdocs: not by default, possible by renamer/nester.
126-
- [X] accessors accept default (not to search x2 contain+get_path)
127-
WONTFIX bc not worth it.
128-
- [X] Simplify Task-context by injecting it in a parametric argument of `fn`.
129-
NO, `opcb` modifier works without `inspect` module.
130-
131-
+ See :gg:`1`.
15+
TODOs
16+
%%%%%
17+
18+
- [+] ENH: planning reports unsatisfied node modus, for plotting
19+
- [+] ENH: Plan accepts previous solutions
20+
21+
- [+] fix: FnOp.compute() should (or should not?) accept Pipeline.compute() args.
22+
- [+] FEAT: ``compute(solution, recompute_from=["input_1"])``
23+
24+
Configs
25+
=======
26+
27+
- [ ] FEAT: drop MANY contextvars with 1 cfg-dict in contextvars?
28+
- [ ] FEAT: drop contextvars, use merging-dicts on (cfg, pipe, op) instead?
29+
BUT then cannot change configs amidst run, unless next todo ...
30+
- [ ] FEAT: configs & op attributes accept 2-tuples: (flag, priority)
31+
default priorities: cfg, fnop
32+
(+cat: Compose)
33+
- [ ] FEAT: Config DEBUG flags:
34+
35+
- [ ] skip - evictions(drop config)
36+
- [ ] keep SFX in outputs
37+
38+
Compose
39+
=======
40+
41+
- [+] feat: add a real ``implicit`` modifier.
42+
- [?] REFACT: separate op-decorator from factory (to facilitate defining conveyor operations).
43+
- [?] `cwd()` modifier:
44+
Isn't already working with relative jsonps?
45+
46+
- `implicit` with only the 2nd arg should work on cwd, no?
47+
48+
- [ ] ++FEAT: Teach pipelines how to accept positional arguments with a tuple with their names
49+
- [ ] ++FEAT: +1 merge method for pipelines: nest=False: treat Pipelines as Operations
50+
(need pipeline feat for positional-args, above).
51+
- [ ] break/enh(modifier): augment modifier table with ``implicits`` for REPR.
52+
- [ ] BREAK(modif): "stuffy" names for ``is_xxx(modifier)`` funcs returning stuff.
53+
- [ ] +++FEAT: Autograph functions by type-annotations
54+
(PY39 :pep:`593` e.g. ``Annotated[int, CWD]``)
55+
- [ ] +++FEAT: Autograph functions by type-annotations
56+
- [ ] modifier || jsonp --> pandalone.pstep (flexible syntax for a modifier on each step)
57+
- [ ] Rename modifier sfx --> token
58+
59+
60+
Planning
61+
========
62+
63+
- [ ] FEAT: break cycles with dijkstra; +weights?
64+
- [ ] ++ENH: link graph inputs/outputs to virtual root-nodes, to replace custom visiting algos
65+
(eg unsatisfied-ops, prune-by-outs)
66+
- [ ] FEAT: per-operation EVICTs
67+
(+cat: Compose)
68+
- [ ] ++DROP PARALLEL: always produce a list of "parallelizable batches",
69+
to hook with other executors, and keep here just the single-process implementation.
70+
(+cat: Execution)
71+
72+
73+
Execute
74+
=======
75+
76+
- [+] FEAT: +2 callbacks before marshalling: (pre_batch, post_batch) (olds: pre_op, post_op).
77+
78+
- PARTIAL: moved existing x2 callbacks to be called before marshalling.
79+
80+
- [?] refact: named_inputs --> sol
81+
- [ ] ENH/REFACT: use Signature.Param from `inspect` module to match needs & zip provides
82+
- [ ] Solution-retriever modifier `opcb()`;
83+
84+
- WONTFIX: easier and more generic to access solution from Op-context.
85+
- REINSTATED: bc it's simpler and does not have any threading issues....
86+
87+
- [ ] ++FEAT(fnop): vararg(s) for Outs collect all outs to the very end
88+
- [ ] BREAK/FEAT: allow marking SFX-LIST items:
89+
90+
1. Implicits (default), to be checked/resolved downstream (see "FEAT: +2 callbacks"), or
91+
2. Sfx (pure), as they are now).
92+
93+
- [ ] +++BREAK/DROP accessors for `jsonp` (move all functionality in solution)
94+
95+
- REJECTED already used for hcat()/vcat()
96+
- REINSTATED as "ACCESSOR per jsonp-PART"! (h/vcat should have been like that)
97+
by default applied to the last part (or use ``[acc, ...]`` syntax for specific parts)
98+
99+
- [ ] RENAME ``Solution --> DataTree``
100+
- [ ] move dep_prefixed() as a top-level `modifier` utility (to prefix also Accessors)
101+
- [ ] revive "stable sort paths" branch on solution updates
102+
- [ ] hdf/vdf accessor example-utilities for typical level-0 resolving multi-indexed slices.
103+
- [ ] Drop int/attribute indexers from resolve-jsonp, accessors-per-part can do that.
104+
- [ ] Interoperate with *Dask,* *Vaex,* *Modin* etc
105+
(see https://12ft.io/proxy?&q=https%3A%2F%2Ftowardsdatascience.com%2F4-libraries-that-can-parallelize-the-existing-pandas-ecosystem-f46e5a257809)
106+
- [ ] replace in-house `jetsam` with :pep:`678` for exception notes on PY3.11+.
107+
108+
Solution
109+
--------
110+
111+
- [+] DROP/ENH: Solution updates GivenInputs only, yet still layers jsonp-refer to its values.
112+
- [+] FEAT: ``pd.concat()`` --> modifier+accessor, to avoid sfxed for multi-column updates.
113+
- [+] FEAT: + `post_callback` with `pre_callback` --> `callbacks` tuple
114+
(+cat: Compose)
115+
116+
- [+] ENH(jsonp): mass-concat (not one-by-one).
117+
- [+] ENH: validate Implicits indeed added (in `post_op_callback`)?
118+
- [ ] FEAT: VALIDATE (by user) items on Solution-Insert (with a `post_callback`?):
119+
- auto-assertions ``{jsonp--> validators}``
120+
- compare overwrites while recomputing
121+
- check implicits exist (??not done above??)
122+
123+
Plotting
124+
========
125+
126+
- [+] Badges on Data
127+
- [ ] generate legend dynamically
128+
- [ ] Draw nested graphs as such (hiding internal nodes)
129+
- [ ] SPHINXEXT: autodoc Pipelines & Ops
130+
- [ ] sphinxext: extend standard ``doctest`` or *doctest* module (instead of sphinx-builder)
131+
- [ ] FEAT/REFACT: dogfood when constructing graphviz graphs.
132+
- [ ] FEAT: add another `dagviz <https://wimyedema.medium.com/drawing-dags-5cadcb452115>`_
133+
plot-backend.
134+
135+
Docs
136+
====
137+
138+
- [ ] REFACT: move GitHub organization `pygraphkit --> pygraphtik` (+Travis, +RTD)
139+
- [ ] Merge tutorial (operations + composition)
140+
- [ ] +++DOC: explain Implicits in tutorial; mention ovunque.
141+
142+
- [ ] TCs: Test DEBUG
143+
144+
145+
DROPPED
146+
=======
147+
148+
- [X] `solution.executed` pre-populated with all operations
149+
- [X] parallel batches restart from last position in steps
150+
- [X] covert custom op classes & modifiers directly into mergeable networkx graphs;
151+
152+
- WONTFIX bc foreign function would not work with merged deps.
153+
154+
- [X] conditionals
155+
156+
- WONTFIX bc it permits pipelines with too complex execution flow to debug.
157+
158+
- [X] DROP accessors:
159+
160+
- REJECTED already used for hcat()/vcat() (but to be dropped for jsnop).
161+
162+
v9.0.0
163+
------
164+
165+
- [X] Accept jsonp inputs & outputs,
166+
167+
- WONTFIX user's business to expand into given Inputs, Outputs already working.
168+
169+
- [X] REVERT rename subdocs;
170+
171+
- WONTFIX bc eventually made it work correctly and added TC.
172+
173+
- [X] REFACT: separate op-decorator from factory
174+
(to facilitate defining conveyor operations):
175+
176+
- NO, simplify passing fn=None.
177+
178+
- [X] Nest-rename subdocs: not by default, possible by renamer/nester.
179+
- [X] accessors accept default (not to search x2 contain+get_path)
180+
181+
- WONTFIX bc not worth it.
182+
183+
- [X] Simplify Task-context by injecting it in a parametric argument of `fn`.
184+
185+
- NO, `opcb` modifier works without `inspect` module.
186+
187+
- See :gg:`1`.
132188

133189

134190
Changelog

0 commit comments

Comments
 (0)