Merge pull request #40 from NACLab/v3

Merge/transfer from v3 to main branch (roll-out of ngc-sim-lib v3); already obtained approval internally from dev-team.

Author: ago109

.gitignore

@@ -5,3 +5,4 @@ build
 ngcsimlib.egg-info
 
 __pycache__
+/ngcsimlib/old/

CITATION.cff

@@ -3,11 +3,12 @@ message: "NGC-Sim-Lib: A Simulation Framework for Supporting NGC-Learn."
 authors:
   - family-names: Gebhardt, Ororbia
     given-names: William, Alex
+    orcid: https://orcid.org/0009-0008-7456-6556
   - family-names: Ororbia
     given-names: Alexander
     orcid: https://orcid.org/0000-0002-2590-1310
 title: "ngc-sim-lib"
-version: 0.2.0
+version: 3.0.0
 identifiers:
   - type: doi
     value: 10.5281/zenodo.10888211

README.md

@@ -41,7 +41,7 @@ $ pip install --editable . # OR pip install -e .
 </pre>
 
 **Version:**<br>
-1.0.1 <!-- -Beta --> <!-- -Alpha -->
+3.0.0 <!-- -Beta --> <!-- -Alpha -->
 
 Authors:
 William Gebhardt, Alexander G. Ororbia II<br>
@@ -50,10 +50,10 @@ Rochester Institute of Technology, Department of Computer Science
 
 ## <b>Copyright:</b>
 
-Copyright (C) 2023 The Neural Adaptive Computing Laboratory - All Rights Reserved<br>
+Copyright (C) 2021 The Neural Adaptive Computing Laboratory - All Rights Reserved<br>
 You may use, distribute and modify this code under the
 terms of the BSD 3-clause license.
 
 You should have received a copy of the BSD 3-clause license with
-this software.<br>
+this software.<br> 
 If not, please [email us](mailto:[email protected])

docs/compartments.md

@@ -0,0 +1,44 @@
+# Compartments
+
+Within NGC-Sim-Lib, the global state serves as the backbone of any given model. 
+This global state is essentially the culmination of all of the dynamic or changing parts of the model itself. Each 
+value that builds this state is stored within a special "container" that helps track these changes over time -- this 
+is referred to as a `Compartment`.
+
+## Practical Information
+
+Practically, when working with compartments, there are a few simple things to keep in mind despite the fact that most 
+of NGC-Sim-Lib's primary operation is behind-the-scenes bookkeeping. The two main points to note are: 
+1. Each compartment holds a value and, thus, setting a compartment with `myCompartment = newValue` will not function as 
+   intended since this will overwrite the Python object, i.e., the compartment with `newValue`. Instead, it is 
+   important to make use of the `.set()` method to update the value stored inside a compartment so 
+   `myCompartment = newValue` becomes `myCompartment.set(newValue)`.
+2. In order to retrieve a value from a compartment, use `myCompartment.get()`. These methods of getting and setting 
+   data inside a compartment are important to use when both working with and designing a multi-compartment component 
+   (i.e., `Component`).
+
+## Technical Information
+
+The follow sections are devoted to explication of more technical information regarding how a compartment functions 
+with in the broader scope of NGC-Sim-Lib and, furthermore, to explain how to leverage this information.
+
+### How Data is Stored (Within a Model Context)
+
+The data stored inside of a compartment is not actually physically stored within a compartment. Instead, it is stored 
+inside of the global state and each compartment effectively holds the path or `key` to the right spot in the global 
+state, allowing it to pull out a specific piece of information. As such, it is technically possible to manipulate the 
+value of a compartment without actually touching the compartment object itself within any given component. By default, 
+compartments have in-built safeguards in order to prevent this from happening accidentally; however, directly addressing 
+the compartment within the global state directly has no such safeguards.
+
+### What is "Targeting"?
+
+As discussed in the model building section, there is notion of "wiring" together different compartments of different 
+components -- this is at the core of NGC-Learn's and NGC-Sim-Lib's "nodes-and-cables system". These wires are created 
+through the concept of "targeting,", which is, in essence, just the updating of the path stored within a compartment 
+using the path of a different compartment. This means that, if the targeted compartment goes to retrieve the value 
+stored within it, it will actually retrieve the value of a different compartment (as dictated by the target). When a 
+compartment is in this state -- where it is targeting another compartment -- it is set to read-only, which only means that 
+it cannot modify a different compartment.
+
+

docs/compiling.md

@@ -0,0 +1,98 @@
+# Compiling
+
+The term "compiling" for NGC-Sim-Lib refers to automatic step that happens
+inside of a context that produces a transformed method for all of its
+components. This step is the most complicated part of the library and, in
+general, does not need to be touched or interacted with. Nevertheless, this
+section will cover most of the steps that the NGC-Sim-Lib compilation process
+does at a high level. This section contains advanced technical/developer-level
+information: there is an expectation that the reader has an understanding of
+Python abstract syntax trees (ASTs), Python namespaces, and how to
+dynamically compile Python code and execute it.
+
+## The Decorator
+
+In NGC-Sim-Lib, there is a decorator marked as `@compilable` which is used to
+add a flag to methods that the user wants to compile. On its own, this will not
+do anything; however, this decorator lets the parser distinguish between methods
+that should be compiled and methods that should be ignored.
+
+## The Step-by-Step NGC-Sim-Lib Parsing Process
+
+The process starts by telling the parser to compile a specific object.
+
+### Step 1: Compile Children
+
+The first step to compile any object is to make sure that all of the 
+"compilable" objects of the top level object are compiled. As a
+result, NGC-Sim-Lib will loop through all of the whole object and will compile
+each part that it finds that is flagged as compilable (via the decorators
+mentioned above) and is, furthermore, an instance of a class.
+
+### Step 2: Extract Methods to Compile
+
+While the parser is looping through all of the parts of the top-level object, it
+is also extracting the methods on/embedded to the object that are flagged as
+compilable (with the decorator above). NGC-Sim-Lib stores them for later;
+however, this lets the parser only loop over the object once.
+
+### Step 3: Parse Each Method
+
+As each method is its own entry-point into the transformer, this step will run 
+for each method in the top-level object.
+
+### Step 3a: Set up a Transformer
+
+This step sets up a `ContextTransformer`, which further makes use of a
+`ast.NodeTransformer`, and will convert methods from class methods (with the use
+of `self`), as well as other methods that need to be removed / ignored, into
+their more context-friendly counterparts.
+
+### Step 3b: Transform the Function
+
+There are quite a few pieces of common Python that need to be transformed. This
+step happens with the overall goal of replacing all object-focused parts with a
+more global view. This means that a compartment's `.get` and `.set` calls are
+replaced with direct setting and getting from the global state, based on the
+compartment's target. This also means that all temporally constant values --
+such as `batch_size` -- are moved into the globals space for that specific file
+and ultimately replaced with the naming convention of `object_path_constant`.
+One more key step that is performed is to ensure that there is no branching in
+the code. Specifically, if there is a branch, i.e., an if-statement, NGC-Sim-Lib
+will evaluate it and only keep the branch it will traverse down. This means that
+there cannot be any branch logic based on inputs or computed values (this is a
+common restriction for just-in-time compiling).
+
+### Step 3c: Parse Sub-Methods
+
+Since it is possible to have other class methods that are not marked as
+entry-points for compilation but still need to be compiled, as step 3b happens,
+NGC-Sim-Lib tracks all of the sub-methods required. Notably, this step goes
+through and repeats steps 3a and 3b for each of the (sub-)methods with a naming
+convention similar to the temporally constant values for each method.
+
+### Step 3d: Compile the Abstract Syntax Tree (AST)
+
+Once we have all of the namespace and globals needed to execute the
+properly-transformed method, the method is compiled with Python and finally
+executed.
+
+### Step 3e: Binding
+
+The final step per method is to bind each to their original method; this
+replaces each method with an object which, when called, will act like the
+normal, uncompiled version but has the addition of the `.compiled` attribute.
+This attribute contains all of the compiled information to be used later (for
+model / system simulation). This crucially allows for the end user to
+call `myComponent.myMethod.compiled()` and have it run. The exact type for
+a `compiled` value can be found
+in `ngcsimlib._src.parser.utils:CompiledMethod`.
+
+### Step 4: Finishing Up / Final Processing
+
+Some objects, such as the processes, entail additional steps to modify
+themselves or their compiled methods in order to align themselves with needed
+functionality. However, this operation/functionality is found within each
+class's expanded `compile` method and should be referred to by looking at those
+methods specifically.   
+ 

docs/components.md

@@ -0,0 +1,32 @@
+# Components
+
+Living one step above compartments in the NGC-Learn dynamical systems hierachy rests the component. 
+A component (`ngcsimlib.Component`) holds a collection of both temporally constant values as well as dynamic (time-evolving) 
+compartments. In addition, they are the core place where logic governing the dynamics of a system are 
+defined. Generally, components serve as the building blocks that are to be reused multiple times 
+when constructing a complete model of a dynmical system.
+
+## Temporally Constant versus Dynamic Compartments
+
+One important distinction that needs to be highlighted within a component is the 
+difference between a temporally constant value and a dynamic (time-varying) compartment.
+Compartments themselves house values that change over time and, generally, they will have the 
+type `ngcsimlib.Compartment`; note that compartments are to be used to track the internal values 
+of a component. These internal values can be ones such inputs, decaying values, counters, etc. 
+The second kind of values found within a component are known as temporally constant values; these 
+are values (e.g., hyper-parameters, structural parameters, etc.) that will remain fixed 
+within constructed model dynamical system. These types of values tend to include common configuration 
+and meta-parameter settings, such as matrix shapes and coefficients.
+
+## Defining Compilable Methods
+
+Inside of a component, it is expected that there will be methods defined that govern the
+temporal dynamics of the system component. These compilable methods are decorated 
+with `@compilable` and are defined like any other regular (Python) method. Within a compilable
+method, there will be access to `self`, which means that, to reference a compartment's
+value, one must write out such a call as: `self.myCompartment.get()`. The only requirement is 
+that any method that is decorated <b>cannot</b> have a return value; values should be stored 
+inside their respective compartments (by making an appeal to their respective set routine, i.e., 
+`self.myCompartment.set(value)`). In an external (compilation) step, outside of the developer's 
+definition of a component, an NGC-Sim-Lib transformer will change/convert all of these (decorated) 
+methods into ones that function with the rest of the NGC-Sim-Lib back-end.

docs/context.md

@@ -0,0 +1,67 @@
+# Contexts
+
+Contexts, in NGC-Sim-Lib, are the top-level containers that hold everything used to
+define a model / dynamical system. On their own, contexts have no runtime logic; 
+they rely on their internal processes and components to build a complete, working model.
+
+## Defining a Context
+
+To define a context (`ngcsimlib.Context`), NGC-Sim-Lib leverages the `with` block; this 
+means that to create a new context, simply start with the statement 
+`with Context("myContext") as ctx:` and a new context will be created. 
+(<i>Important Note</i>: names are unique; if a context is created with the same name, 
+they will be the same context and, thus, there might be conflicts). 
+A defined context does not do anything on its own.
+
+## Adding Components
+
+To add components to a context, simply initialize components while inside
+the `with` block of the context. Any component defined while inside this block
+will automatically be added and tacked-on to the context object.
+
+## Wiring Components
+
+Inside of a model / dynamical system, components will need to pass data to one 
+another; this is configured within the context. To connect the compartments of 
+two components, follow the pattern: `mySource.output >> myDestination.input`, 
+where `output` and `input` are compartments inside their respective components. 
+This format will ensure that, when processes are being run, the value will 
+flow properly from component to component.
+
+### Operators
+
+There is a special type of wire called an operator; this performs a simple
+operation on the compartment values as the data flows from one component to
+another. Generally, these are use for simple mathematical operations, such as
+negation `Negate(mySource.output) >> myDestination.input` or the summation of
+multiple compartments into
+one `Summation(mySource1.output, mySource2.output, ...) >> myDestination.input`.
+Note that operators can be chained, so it would be possible to negate one or 
+more of the inputs that flow into the summation.
+
+## Adding Processes
+
+To add processes to a context, simply initialize the process and add all of its
+steps while inside the `with`-block of the process.
+
+## Exiting the `with` block
+
+When the context exits the `with`-block, it will re-compile the entire model. 
+Behind the scenes, this is calling `recompile` on the context 
+itself; it is possible to manually trigger the recompile step, but doing so can 
+break certain connections (between components/compartments), so use this 
+functionality sparingly.
+
+## Saving and Loading
+
+The context's one unique job is the handling of the "saving" (serialization) and 
+"loading" (de-serialization) of models to disk. By default, calling 
+`save_to_json(...)` will create the correct file structure as well as the core files 
+needed and load the context in the future. To load / de-serialize a model, 
+calling `Context.load(...)` will load the context in from a directory; something
+important to note is that loading in a context entails effectively 
+recreating the components with their initial values using their arguments as well as
+keywords arguments (excluding those that cannot be serialized). This means that, 
+if you have a trained model, ensure that your components have a save method 
+defined that will handle the saving and loading of all values within their compartments.  
+

docs/global_state.md

@@ -0,0 +1,52 @@
+# The Global State
+
+Since NGC-Sim-Lib is a simulation library focused on temporal models and dynamical 
+systems, or models that change over time there, it is foundational that all models 
+(and their respective elements) have some concept of a "state". These states
+might be comprised of a single value that changes/evolves or of a complex set of values
+that, when combined all together, make up the full dynamical system that underwrites the 
+final model. In both cases, these sets of values are stored in what is known as the 
+<i>global state</i>.
+
+## Interacting with the Global State
+
+Since the global state will contain a large amount of information describing a given
+model, there will be a need to facilitate interaction with and modification of the values 
+contained within the global state. In most use-cases, this is not done directly. The 
+most common way to interact with the global state is through the use of the state-manager. 
+The state-manager exists to provide a set of helper methods for interacting with the
+global state itself. Note that, although the manager is there to assist you, it will not stop
+you from changing the state (or "breaking" the state). When changing the state -- beyond 
+setting it through the specificaiton of <i>processes</i> -- be careful to not add or remove
+anything that is needed for your actual model.
+
+### Adding New Fields to the Global State
+
+If you are new to using NGC-Sim-Lib and looking for a way to add values to the
+global state directly and explicitly, stop for a moment and reconsider. Unless 
+you know exactly what you are doing (i.e., doing core development), it is strongly 
+advised to not manually add values to the global state; instead, work through the 
+mechanisms afforded by compartments and/or components, as these are built to afford you the 
+most common ways for adding fields to the global state itself. The dynamical systems 
+semantics inherent to compartments and components is meant to ensure carefully-constrained 
+design and simulation of flexible models. 
+
+If you actually intend to manually and directly add values to the global state itself, it 
+is done through the use of the `add_key` method. This will create the appropriate key in
+the global state for the given path and name; furthermore, its value can be retrieved
+with `from_key` calls. This value, however, is not linked to a compartment and, therefore, 
+will be hard to get working properly in the compiled methods without some specific references. 
+<i>Please take extra care when working directly and explicitly with the global state.</i>
+
+### Getting the Current State
+
+To get the current state, simply call `global_state_manager.state`; this will give
+you a (shallow) copy of the current state, which means that any modifications made to it will 
+not be reflected in the global state.
+
+### Updating the Global State
+
+To manually update the global state after modifying a local copy; please write an overriding 
+call command: `global_state_manager.state = new_state`. This will update the state with the 
+`.update` call to its underlying dictionaries, which means that a partial state will still update correctly.
+