Using master branch of MSL - without parts

[HansOlsson]

At the latest meeting there were a discussion about putting references inside the MSL, and one concern was that if people want to get access to the latest build, then adding the references would substantially increase the download size and possibly discourage some users.

TL;DR: It works, but we should have reference-(results) in a separate top-level directory.

My question was if there's a solution with the following constraints:

• Use git
• Not use submodules
• Small size on disk for the latest version - skipping e.g., references
• No/minimal impact in other cases (related to "no submodules")

I know that several companies (like Microsoft) have switched to large mono-repos so I assumed it would be possible now (note that some of the commands are "new").

The preliminary answer seems to be that it seems possible, and we just have to do a minimal restructuring.
The restructuring is just that references (preferably including the existing comparisonSignals.txt) should be outside of the Modelica-directory (so still in the repo on the same branch). No need for a separate sub-module, just a separate directory.

To me that seems like a good solution regardless of this goal, and I don't think it will be that problematic to update our CI/CD scripts handle that.

There are still some things that could be improved, but it is clearly good enough for some uses.

The commands needed to get the relevant parts of the master branch are:

• Sparse cloning without history: git clone --depth=1 --filter=blob:none --sparse https://github.com/modelica/ModelicaStandardLibrary.git; See https://git-scm.com/docs/git-clone - this is also normal to use for CI/CD as I understand.
• Sparse checkout: (inside ModelicaStandardLibrary) git sparse-checkout set Modelica ModelicaReference gives the main parts without ModelicaTest and without ModelicaServices - but they can be added if desired, of course. If we had a directory ReferenceResults that would be skipped as well. See https://git-scm.com/docs/git-sparse-checkout

I checked and we don't need to change anything to support this - except that it would be good to move Modelica/Resources/Reference/Modelica to a separate directory, e.g., ReferenceResults/Modelica and either add the actual results to the same directory or have one sub-directory for what to compare and one for the references.

Notes:

• File history doesn't work well. You can include the history by skipping --depth=1, but git annotate doesn't work well in either case - either you don't get any history or it has to download the history for each command, which even if using less disk-space in practice is slower than getting the entire repository. There might be some better solution, but I anyway usually use GitHub for that.
• The filter-part is the thing that really reduces the size on disk. See https://git-scm.com/docs/git-rev-list#Documentation/git-rev-list.txt---filterfilter-spec
• The sparse-checkout is directory-based and includes all files in the top-level directory (ignore the deprecated option). So Complex.mo, Readme.md etc are always included, and in general all files down to the selected directories
• I haven't checked that you can create a branch, commit and make a pull-request but I don't see why it would be problematic.
• I haven't checked what happens if there are new commits on master-branch, but I assume it works smoothly.

Other related issues that are are mostly solved:

• Libraries (for tables etc) can be handled by automatically building them; tools may need to improve the support for doing that and we should improve makefile support in both the specification and tools.
• ModelicaServices - it's just a matter of using the tool-specific one; so that's entirely a tool-issue.

Have I missed something? Do people agree that the minimal restructuring is worth the effort, if so we can then just document it.

[henrikt-ma]

The restructuring is just that references (preferably including the existing comparisonSignals.txt) should be outside of the Modelica-directory (so still in the repo on the same branch). No need for a separate sub-module, just a separate directory.

To me that seems like a good solution regardless of this goal, and I don't think it will be that problematic to update our CI/CD scripts handle that.

As I recall it, this is what we already decided in the meeting yesterday.

I would prefer if the directory with files related to testing uses a name with lower case initial to make it more easily distinguishable from the directories for Modelica code hierarchies. Perhaps reference-results or references would be good names, unless we foresee the use of this directory for more general test-related content? In particular, I can see how we at some point might want to also include some specification of how to perform the comparison against a particular reference result, and then a name such as tests would make more sense.

Within the directory, the only structure I can see how it would work is the one we already have in use in MAP-LIB_ReferenceResults – up to what we call the files inside the directory corresponding to an example. In my opinion, it is a bit odd that we have standardized names for all files but the CSV result; calling it Rectifier/reference.csv could be a better alternative to calling it Rectifier/Rectifier.csv.

[HansOlsson]

The restructuring is just that references (preferably including the existing comparisonSignals.txt) should be outside of the Modelica-directory (so still in the repo on the same branch). No need for a separate sub-module, just a separate directory.
To me that seems like a good solution regardless of this goal, and I don't think it will be that problematic to update our CI/CD scripts handle that.

As I recall it, this is what we already decided in the meeting yesterday.

That wasn't clear to me, but that's even better.

I would prefer if the directory with files related to testing uses a name with lower case initial to make it more easily distinguishable from the directories for Modelica code hierarchies. Perhaps reference-results or references would be good names, unless we foresee the use of this directory for more general test-related content? In particular, I can see how we at some point might want to also include some specification of how to perform the comparison against a particular reference result, and then a name such as tests would make more sense.

That would work for me as well, one concern is that we have the libraries ModelicaTest (test models) and ModelicaReference (reference for the Modelica Language) in the repo as well - so it's easy to get confused.

Within the directory, the only structure I can see how it would work is the one we already have in use in MAP-LIB_ReferenceResults – up to what we call the files inside the directory corresponding to an example. In my opinion, it is a bit odd that we have standardized names for all files but the CSV result; calling it Rectifier/reference.csv could be a better alternative to calling it Rectifier/Rectifier.csv.

I agree that .../Analog/Examples/Rectifier/Rectifier.csv is odd and .../Analog/Examples/Rectifier/reference.csv seems better. However, it's a bit weird that we have so many (single-file) directories (already an issue for comparisonSignals.txt) so .../Analog/Examples/Rectifier.csv would also work, but I believe there was some previous reason for that choice. It also depends on the overall structure, if we want both reference.csv and comparisonSignals.txt in the same directory I can see some benefit, but if we have them in different directory-structures, I think that the number of single-file directories is excessive.

[henrikt-ma]

I agree that .../Analog/Examples/Rectifier/Rectifier.csv is odd and .../Analog/Examples/Rectifier/reference.csv seems better. However, it's a bit weird that we have so many (single-file) directories (already an issue for comparisonSignals.txt) so .../Analog/Examples/Rectifier.csv would also work, but I believe there was some previous reason for that choice. It also depends on the overall structure, if we want both reference.csv and comparisonSignals.txt in the same directory I can see some benefit, but if we have them in different directory-structures, I think that the number of single-file directories is excessive.

The MAP-LIB_ReferenceResults has 7 files in the directory for an example:

• https://github.com/modelica/MAP-LIB_ReferenceResults/tree/v4.0.0/Modelica/Electrical/Analog/Examples/Rectifier

Some of them don't seem to make sense to keep when we move things to the Modelica repository, but most of them do:

• reference.csv (originally called Rectifier.csv) – obviously
• comparisonSignals.txt – allows temporarily removing the result CSV without losing the list of variables
• creation.md (originally called creation.txt) – good practice to have a file describing how the reference was created
• simulate_passed.log – not strictly needed, but has been found useful in the past (the log should come from the tool used to create the reference, without any specific requirements on the content)
• translate_passed.log – not strictly needed, but has been found useful in the past (the log should come from the tool used to create the reference, without any specific requirements on the content)

Additionally, as mentioned above, I can imagine also having:

• comparison.json – specification of how to compare against reference.csv

This adds up to somewhere between 3 and 6 files per example, so having a directory for the example makes very good sense to me.