Architecture thoughts on serialize/deserialize capability

[j-simmons-phd]

With work wrapping up on the project create feature and our moving into work to define a standard set schema queries, I have been thinking we might benefit from having the ability to serialize our queries when we are creating them.

Here are some updated thoughts on what serializing might look like. Note, we will need to add the dendron notes to this thread shortly to consolidate things.

General approach

• I would like to have a library that feels a lot like the js yaml library
• two main functions: dump and load
• dump takes a js object and a model navigation path and instead of returning a string it puts the data onto the fs at the location that matches the model navigation path
• load takes a model navigation path and returns a js object based on fs contents at that location

[j-simmons-phd]

Some additional thought:

• I greatly prefer an implementation that does not need to know the schema in order for the library to function properly
• The dump and load functions must be one-to-one and onto (e.g. for a given js object there is only one way it will serialize, and deserializing the fs representation must return the original js object exactly)
• I think it is critical to be able to lazy load and lazy dump (e.g. only load/dump one level of the model at a time, instead of having to load/dump the full model every time)
• I think it is essential that the implementation create a representation on the fs that is easily understood by users when viewed directly
• What we called building the full model from the fs in older implementations of this tooling would be as simple loading the model from fs for the top level model element (with lazy loading disabled) and then writing the js object to a yaml file
• The implementation should isolate changes to the element that was actually modified (e.g. if an object representing a pizza party has a list of invitees, changes to the list of invitees are specifically about the list of invitees, not the fundamentals of the pizza party like the start and end times; so adding an invitee should only modify the list of invitees on the fs). This is why we want to do things like make list contents a separate file from the object that owns the list. This way adding or subtracting elements (or rearranging their order) only requires committing a new version of the list file

[j-simmons-phd]

• What we called building the full model from the fs in older implementations of this tooling would be as simple loading the model from fs for the top level model element (with lazy loading disabled) and then writing the js object to a yaml file

This statement may not fully cover building the full model, as we have to account for how we will include content from sub-projects and projects brought in via npm install in top level builds. Still, this will at least be a part of the implementation and it will simplify things compared to earlier work.

[capsulecorplab]

Would it make sense to somehow support/account for symbolic links in the YAML data model, perhaps leveraging npm link, to facilitate linking to external model content?

[j-simmons-phd]

I think we should avoid relying on npm inside this library. While we need npm (or one of its competitors) in m30pm, we so far do not need it in yaml-datastore.

This might be more of a question for using yaml-datastore in m30pm. Admittedly, I think there is going to be a lot of work to do where utilizing dependencies in m30pm is concerned and this sounds like something that would come up in discussing that behavior.

[j-simmons-phd]

Oh, one other thing...

I want to be able to do is push "big" strings (things like the njk templates) to separate files when we dump to fs (so that we can edit them as stand alone files) and keep basic strings directly in object files.

In past discussions we have established the idea of scalar values are stored in line and vector/object values are stored in separate files. So, I propose a string be considered scalar of it contains no newlines and vector if it contains any newlines, to enable strings to work as described above and also follow the convention we already established.

[capsulecorplab]

I'm wondering whether it'd be too early to (re)write the specs with OSHW-specific examples, as to support/focus on the classes of data and use cases we can expect to run into with DOF. s.a, the example/spec for an object containing both scalar and vector values as an activitystep

[j-simmons-phd]

I am not quite following the example you are describing at the end of your comment in terms of the mix of scalar/vector data.

[capsulecorplab]

For example, an activityStep (in legacy DOF) is an object that holds both string (scalar) i.e., "summary", lists (vectors) i.e., "requiredParts" & "requiredTools", and a complex string (vector) i.e, "details" https://mach30.github.io/dof/#_activity_step. I'm wondering if it'd be worth coming up with specs for data structures with DOF examples in mind this early.

[j-simmons-phd]

Ah! Yes, something like this is what I was thinking for the jsObjectRepresentative case (or whatever it is named) after we get the smaller cases working. Almost like an integration test if we think of the simpler tests as unit tests.

[j-simmons-phd]

I think this might be useful to create a short GUID like ID for unnamed elements (e.g. elements of lists). https://shortunique.id/

[capsulecorplab]

Right. I'm just pondering how assigning a (GU)ID to an unnamed element can be done in such a way that's 1-to-1 and onto. If a complex string is assigned a GUID on disk, s.a., 550e8400-e29b-41d4-a716-446655440000.txt, and is an element in a list, s.a.,

- ((550e8400-e29b-41d4-a716-446655440000.txt))

then wouldn't the GUID get lost when it's deserialized into a js object?

[j-simmons-phd]

Real quick, for the sake of readability, I would suggest updating the example to use shortid instead of a true full guid (short-ids are long enough that I can't imagine running out of address space for any of our use cases).

https://shortunique.id

Which can for instance give you IDs made of hex digits (say 6 hex digits long): f245a9 or 75aee3. This is one of my preferred formats to consider.

[j-simmons-phd]

One of the things I have not had a chance to show you from my prior work has to do with how id generators like this work. At their heart, they use Random Number Generators (RNGs). And RNGs can be configured to give the same sequence of "random" numbers every time you ask for a sequence. This enables one to spec the use of a specific method for generating ids and a specific configuration, and then everyone who follows that spec will get the same values for ids as they are generated.

This is a bit of a simplification, but it is close enough to demonstrate the point.

Note, that one commit we have not looked at yet demonstrates this repeatability.

[capsulecorplab]

Oh, interesting. I didn't know RNGs could do that...

[j-simmons-phd]

Yup, it is done by setting a value in the RNG called the seed. Not every RNG provides access to set the seed (JS Math.random is one that does not).

But there are JS/TS RNG libraries that are available that do.

There are also many different RNG algorithms with different focuses (speed, size of random number space before a number is repeated, etc).

One algorithm that looks like it would be sufficient for our needs is Xorshift. And there is a zero dependency JS implementation of the 128 bit version of Xorshift (Xorshift128+) here: https://github.com/AndreasMadsen/xorshift. It is old, but I doubt it has needed any updates in the last 4 years.

We could then pair this with our own small implementation of short-unique-id with a fixed seed for repeatable generation of IDs.