Skip to content

Commit 24d0a20

Browse files
author
Alexander Ororbia
committed
revised/cleaned-up docs
1 parent 4e28ebd commit 24d0a20

File tree

6 files changed

+244
-220
lines changed

6 files changed

+244
-220
lines changed

docs/compartments.md

Lines changed: 37 additions & 42 deletions
Original file line numberDiff line numberDiff line change
@@ -1,49 +1,44 @@
11
# Compartments
22

3-
Inside ngcsimlib there is the global state as the backbone of any given model.
4-
This global state is the culmination of all the dynamic or changing parts of the
5-
model. Each value that builds this state is stored in a special container that
6-
helps track these changes known as a Compartment.
3+
Within ngcsimlib, the global state serves as the backbone of any given model.
4+
This global state is essentially the culmination of all of the dynamic or changing parts of the model itself. Each
5+
value that builds this state is stored within a special "container" that helps track these changes over time -- this
6+
is referred to as a "compartment" (`Compartment`).
77

88
## Practical Information
99

10-
Practically when working with compartments there are a few simple things to keep
11-
in mind and the rest is all behind the scenes bookkeeping. The first piece to
12-
keep in mind is that each compartment holds a value and thus setting a
13-
compartment with `myCompartment = newValue` will not function as intended as
14-
this will overwrite the python object that is the compartment with `newValue`.
15-
Instead, make use of the `.set()` method to update the value stored inside a
16-
compartment so `myCompartment = newValue` becomes `myCompartment.set(newValue)`.
17-
The second piece of information is that to retrieve a value from the compartment
18-
use `myCompartment.get()`. These methods to get and set data inside a
19-
compartment are the two main pieces to remember.
20-
21-
## Technical information
22-
23-
The follow sections are devoted to more technical information regarding how a
24-
compartment functions in the broader scope of ngcsimlib and how to leverage that
25-
information.
26-
27-
### How data is stored
28-
29-
The data stored inside a compartment is not actually stored inside a
30-
compartment. Instead, it is stored inside the global state and each compartment
31-
just holds the path or `key`, in the global state to pull a specific piece of
32-
information. As such it is technically possible to manipulate the value of a
33-
compartment without actually touching the compartment object created in a
34-
component. By default, compartments have safeguards to prevent this from happening
35-
accidentally but directly addressing the compartment in the global state has no
36-
such safeguards.
37-
38-
### What is targeting
39-
40-
As discussed in the model building section there is a concept of wiring together
41-
different compartments of different components. These wires are created through
42-
the concept of targeting. In esscence targeting is just updating the path stored
43-
in a compartment with the path of a different compartment. This means that if
44-
the targeted compartment goes to retireve the value stored in it, it will
45-
actually retrieve the value for the a different compartment. When a compartment
46-
is in this state where it is targeted at another compartment it is set to read
47-
only meaning that it can not modify a different compartment.
10+
Practically, when working with compartments, there are a few simple things to keep in mind despite the fact that most
11+
of ngcsimlib's primary operation is behind-the-scenes bookkeeping. The two main points to note are:
12+
1. Each compartment holds a value and, thus, setting a compartment with `myCompartment = newValue` will not function as
13+
intended since this will overwrite the Python object, i.e., the compartment with `newValue`. Instead, it is
14+
important to make use of the `.set()` method to update the value stored inside a compartment so
15+
`myCompartment = newValue` becomes `myCompartment.set(newValue)`.
16+
2. In order to retrieve a value from a compartment, use `myCompartment.get()`. These methods of getting and setting
17+
data inside a compartment are important to use when both working with and designing a multi-compartment component
18+
(i.e., `Component`).
19+
20+
## Technical Information
21+
22+
The follow sections are devoted to explication of more technical information regarding how a compartment functions
23+
with in the broader scope of ngcsimlib and, furthermore, to explain how to leverage this information.
24+
25+
### How Data is Stored (Within a Model Context)
26+
27+
The data stored inside of a compartment is not actually physically stored within a compartment. Instead, it is stored
28+
inside of the global state and each compartment effectively holds the path or `key` to the right spot in the global
29+
state, allowing it to pull out a specific piece of information. As such, it is technically possible to manipulate the
30+
value of a compartment without actually touching the compartment object itself within any given component. By default,
31+
compartments have in-built safeguards in order to prevent this from happening accidentally; however, directly addressing
32+
the compartment within the global state directly has no such safeguards.
33+
34+
### What is "Targeting"?
35+
36+
As discussed in the model building section, there is notion of "wiring" together different compartments of different
37+
components -- this is at the core of NGC-Learn's and NGC-Sim-Lib's "nodes-and-cables system". These wires are created
38+
through the concept of "targeting,", which is, in essence, just the updating of the path stored within a compartment
39+
using the path of a different compartment. This means that, if the targeted compartment goes to retrieve the value
40+
stored within it, it will actually retrieve the value of a different compartment (as dictated by the target). When a
41+
compartment is in this state -- where it is targeting another compartment -- it is set to read, which only means that
42+
it cannot modify a different compartment.
4843

4944

docs/compiling.md

Lines changed: 31 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -1,35 +1,48 @@
11
# Compiling
2-
The term compiling for ngcsimlib refers to automatic step that happens inside a context that produces a transformed method for all of its components. This step is the most complecated part of the library and generally does not need to be touched or interacted with. Nevertheless this section will cover all the steps that it does at a high level. There is an expectation that the reader has an understanding of python abstract syntax trees, globals, and how to dynamically compile python code to run.
32

4-
## The decorator
5-
In ngcsimlib there is a decorator marked as `@compilable` that is used to add a flag to methods that the user wants to compile. On its own it doesn't do anything but this lets the parser distinguish between method that should be compiled and methods that should be ignored.
3+
The term "compiling" for ngcsimlib refers to automatic step that happens inside of a context that produces a transformed method for all of its components. This step is the most complicated part of the library and, in general, does not need to be touched or interacted with. Nevertheless, this section will cover all of the steps that the ngcsimlib compilation process does at a high level. This section contains advanced technical/developer-level information: there is an expectation that the reader has an understanding of Python abstract syntax trees (ASTs), globals (global variables), and how to dynamically compile Python code to run/execute.
64

7-
## Step by Step
8-
The process started by telling the parser to compile a specific object.
5+
## The Decorator
6+
7+
In ngcsimlib, there is a decorator marked as `@compilable` which is used to add a flag to methods that the user wants to compile. On its own,this will not do anything; however, this decorator lets the parser distinguish between methods that should be compiled and methods that should be ignored (by the overall back-end compilation process).
8+
9+
## The Step-by-Step NGC-Sim-Lib Parsing Process
10+
11+
The process starts by telling the parser to compile a specific object.
912

1013
### Step 1: Compile Children
11-
The first step to compile any object is to make sure that all the compilable objects of the top level object are compiled. So it loops through all the whole object and compiles each part that it finds that is flagged as compilable and is an instance of a class.
14+
15+
The first step to compile any object is to make sure that all of the "compilable" objects of the top level object are essentially compiled. As a result, ngcsimlib will loop through all of the whole objects and will compile each part that it finds that is flagged as compilable (via the decorators mentioned above) and is, furthermore, an instance of a class.
1216

1317
### Step 2: Extract Methods to Compile
14-
While the parser is looping through all the parts of the top level object it is also extracting the methods on the object that are flagged as compilable with the decorator. It stores them for later but this lets the parser only loop over the object once.
18+
19+
While the parser is looping through all of the parts of the top-level object, it is also extracting the methods on/embedded to the object that are flagged as compilable (with the decorator above). ngcsimlib stores them for later; however, this lets the parser know to only loop over the object once.
1520

1621
### Step 3: Parse Each Method
17-
As each method is its own entry point into the compiler the step will run for each method in the top level object
1822

19-
### Step 3a: Set up Transformer
20-
The step sets up a ContextTransformer which is a ask.NodeTransformer and will convert the method from a class method with the use of `self` and other methods that need to be removed into their more context friendly counterpart.
23+
As each method is its own entry-point into the compiler, this step will run for each method in the top-level object.
24+
25+
### Step 3a: Set up a Transformer
2126

22-
### Step 3b: Transform the function
23-
There are quite a few pieces of common python that need to be transformed. This step happens with the overall goal of replacing all object focused parts with a more global view. This means compartment's `.get` and `.set` calls are replaced with direct setting and getting from the global state based on the compartment's target. It also means that all temporally constant values such as `batch_size` are moved into the globals space for that specific file and replaced with the naming convention of `object_path_constant`. One more step that this does is ensure that there is no branching in the code. Specifically if there is a brach such as an if statement it will evaluate it and only keep the branch it will traverse down. This means that there can not be any branch logic based on inputs or computed values (this is a common restriction for just in time compiling).
27+
This step sets up a ContextTransformer, which further makes use of a NodeTransformer, and will convert methods from class methods (with the use of `self`), as well as other methods that need to be removed / ignored, into their more context-friendly counterparts.
28+
29+
### Step 3b: Transform the Function
30+
31+
There are quite a few pieces of common Python that need to be transformed. This step happens with the overall goal of replacing all object-focused parts with a more global view. This means that a compartment's `.get` and `.set` calls are replaced with direct setting and getting from the global state, based on the compartment's target. This also means that all temporally constant values -- such as `batch_size` -- are moved into the globals space for that specific file and ultimately replaced with the naming convention of `object_path_constant`. One more key step that is performed is to ensure that there is no branching in the code. Specifically, if there is a branch, i.e., an if-statement, ngcsimlib will evaluate it and only keep the branch it will traverse down. This means that there cannot be any branch logic based on inputs or computed values (this is a common restriction for just-in-time compiling).
2432

2533
### Step 3c: Parse Sub-Methods
26-
As it is possible to have other class methods that are not marked as entry points for compilation but need to be compiled as step 3b happens it tracks all the sub-methods needed, this step goes through and repeats steps 3a and 3b for each of them with a naming convention similar to the temporally constant values for each method.
2734

28-
### Step 3d: Compile the AST
29-
Once we have all the needed namespace and globals needed to execute the method properly transformed the method is compiled with python and executed.
35+
Since it is possible to have other class methods that are not marked as entry-points for compilation but still need to be compiled, as step 3b happens, ngcsimlib tracks all of the sub-methods required. Notably, this step goes through and repeats steps 3a and 3b for each of the (sub-)methods with a naming convention similar to the temporally constant values for each method.
36+
37+
### Step 3d: Compile the Abstract Syntax Tree (AST)
38+
39+
Once we have all of the namespace and globals needed to execute the properly-transformed method, the method is compiled with Python and finally executed.
3040

3141
### Step 3e: Binding
32-
The final step per method is to bind it to the original method, this replaces the method with an object which when called with act like the normal uncompiled version but has the addition of the `.compiled` attribute which contains all the compiled information to be used later. This allows for the end user to call `myComponent.myMethod.compiled()` and have it run. The exact type for `compiled` value can be found in `ngcsimlib._src.parser.utils:CompiledMethod`.
3342

34-
### Step 4: Finishing Up
35-
Some objects such as the processes have more steps to modify themselves or their compiled methods to align themselves with needed functionality but that is found in each class's expanded `compile` method and should be referred to by looking at those methods specifically.
43+
The final step per method is to bind each to their original method; this replaces each method with an object which, when called, will act like the normal, uncompiled version but has the addition of the `.compiled` attribute. This attribute contains all of the compiled information to be used later (for model / system simulation). This crucially allows for the end user to call `myComponent.myMethod.compiled()` and have it run. The exact type for a `compiled` value can be found in `ngcsimlib._src.parser.utils:CompiledMethod`.
44+
45+
### Step 4: Finishing Up / Final Processing
46+
47+
Some objects, such as the processes, entail additional steps to modify themselves or their compiled methods in order to align themselves with needed functionality. However, this operation/functionality is found within each class's expanded `compile` method and should be referred to by looking at those methods specifically.
48+

docs/components.md

Lines changed: 25 additions & 24 deletions
Original file line numberDiff line numberDiff line change
@@ -1,31 +1,32 @@
11
# Components
22

3-
Living one step above compartments in a model hierachy rests the component. The
4-
components hold a collection of both temporally constant values and dynamic
5-
compartments, in addition they are the lowest place where logic governing the
6-
dynamics of a system are defined. Generally components are going to be the
7-
building blocks that are reused multiple times throughout a model to create a
8-
final dynmical system.
3+
Living one step above compartments in the NGC-Learn dynamical systems hierachy rests the component.
4+
A component (`ngcsimlib.Component`) holds a collection of both temporally constant values as well as dynamic (time-evolving)
5+
compartments. In addition, they are the core place where logic governing the dynamics of a system are
6+
defined. Generally, components serve as the building blocks that are to be reused multiple times
7+
when constructing a complete model of a dynmical system.
98

10-
## Temporally Constant vs Dynamic Compartments
9+
## Temporally Constant versus Dynamic Compartments
1110

12-
One important distinction that needs to be highlighted inside a component is the
13-
difference between a temporally constant value and a dynamic compartment.
14-
Starting with compartments these values that change over time, generally they
15-
will have the type `ngcsimlib.Compartment` and be used to track internal values
16-
of the component. These values can be ones such inputs, decaying values,
17-
counters, etc. The second catagory of values are temporally constant, these are
18-
values that will remain fixed on the model is constructed. These values tend to
19-
include ones such as matrix shapes and coefficients.
11+
One important distinction that needs to be highlighted within a component is the
12+
difference between a temporally constant (or static) value and a dynamic (time-varying) compartment.
13+
Compartments themselves house values that change over time and, generally, they will have the
14+
type `ngcsimlib.Compartment`; note that compartments are to be used to track the internal values
15+
of a component. These internal values can be ones such inputs, decaying values, counters, etc.
16+
The second kind of values found within a component are known as temporally constant values; these
17+
are values (e.g., hyper-parameters, structural parameters, etc.) that will remain fixed / static
18+
with constructed model dynamical system. These types of values tend to include common configuration
19+
and meta-parameter settings, such as matrix shapes and coefficients.
2020

2121
## Defining Compilable Methods
2222

23-
Inside a component it is expected that there are methods used to govern the
24-
temporal dynamics of a system. These compilable methods are decorated
25-
with `@compilable` and are defined like any regular method. Inside a compilable
26-
method there will be access to `self`, meaning that to reference a compartment's
27-
value it is `self.myCompartment.get()`. The only requirement is that any method
28-
decorated can not have a return value, values should be stored inside their
29-
respective compartments. In an external step from defining the component a
30-
transformer will change all of these methods into ones that function with the
31-
rest of ngcsimlib.
23+
Inside of a component, it is expected that there will be methods defined that govern the
24+
temporal dynamics of the system component. These compilable methods are decorated
25+
with `@compilable` and are defined like any other regular (Python) method. Within a compilable
26+
method, there will be access to `self`, which means that, to reference a compartment's
27+
value, one must write out such a call as: `self.myCompartment.get()`. The only requirement is
28+
that any method that is decorated <b>cannot</b> have a return value; values should be stored
29+
inside their respective compartments (by making an appeal to their respective set routine, i.e.,
30+
`self.myCompartment.set(value)`). In an external (compilation) step, outside of the developer's
31+
definition of a component, an ngcsimlib transformer will change/convert all of these (decorated)
32+
methods into ones that function with the rest of the ngcsimlib back-end.

0 commit comments

Comments
 (0)