Making the repository more flexible (not BELLA specific)

[RemiLehe]

We will need to separate the repository into an open-source repository and several private repositories:

• One private repository per experiment (we could have one open-source template) (e.g., synapse-bella-staging-injector, synapse-bella-ip2)
  - Contains configuration file for this experiment (input/output variables, calibration)
  - Contains simulation scripts and scripts for experimental data extraction

• One open-source repository:
  Dashboard, SFAPI orchestration, ML training
  When building container: user provides a file that point to the private repositories that are to be included

We also need more flexibility regarding which database we use, which NERSC user, how to set passwords for the DB, how to authenticate into the dashboard (in collaboration with ml-exchange?)

[RemiLehe]

As part of this reorganization, we could rename simulation_data to simulation_scripts to make it clear that this folder does not hold data.

[EZoni]

This

As part of this reorganization, we could rename simulation_data to simulation_scripts to make it clear that this folder does not hold data.

was resolved in #300.

[EZoni]

Additional things to do once the open-source repository is published:

• Add CI workflows, e.g.,
  • pre-commit linting and formatting.
  • CodeQL code quality check.
  • Unit tests based on pytest where possible.
  • Dashboard tests if needed (more complex).
• Contribute from forks (like for BLAST codes)?
• Fix copyright notice copied at the beginning of each file, e.g.,
  

  synapse/ml/training_pm.sbatch

  Lines 3 to 7 in 4fd91f1

  	# Copyright 2024-2025 Revathi Jambunathan, Axel Huebl
  	#
  	# This file is part of IFE Superfacility LDRD
  	#
  	# License: BSD-3-Clause-LBNL

  
  I am personally in favor of not listing individual authors per file - I think it serves little to no purpose and is mostly unmaintainable.

[RemiLehe]

Some more thoughts:

• The public repo should have a configuration YAML file that could look like this:

experiments:

    bella-staging-injector:
        repository: git@github.com:BLAST-AI-ML/synapse-bella-staging-injector.git

    bella-ip2:
        repository: git@github.com:BLAST-AI-ML/synapse-bella-staging-ip2.git

Where should this file live?

• This file should not be tracked by git (we should add a .gitignore).

• We should add a Python script that downloads all the required repositories, and would complain if the YAML configuration file is absent. Where should the repositories be downloaded? They should also be listed in the .gitignore.

• When building the Docker image, the script that downloads the required repository could be automatically called.

• We should choose the paths so that they are the same when testing locally and on Spin

[RemiLehe]

Discussed offline with @adhamrait @EZoni :

• We don't need the public repo to have a configuration YAML file ; instead the requested experiments would be chosen by cloning the respective repos in a folder experiments, so that we would have the following structure:

dashboard/
ml/
experiments/
    bella-staging-injector/
    bella-ip2/
    ...

• We will update the publish_container.py to look into experiments, raise an error if no experiment is present, loop through the experiments and switch to the main branch, and run git pull, before doing the rest of its tasks.
• The current folders simulation_scripts and experimental_data, will not be a the root level anymore and instead be moved inside their respective experiments.

[EZoni]

This BELLA-specific code will need to be removed:

• synapse/dashboard/utils.py

  Lines 112 to 119 in 59dded3

  	# load database
  	db_defaults = {
  	"host": "mongodb05.nersc.gov",
  	"port": 27017,
  	"name": "bella_sf",
  	"auth": "bella_sf",
  	"user": "bella_sf_ro",
  	}

• synapse/ml/train_model.py

  Lines 72 to 77 in b315707

  	# Connect to the MongoDB database with read-only access
  	db = pymongo.MongoClient(
  	host="mongodb05.nersc.gov",
  	username="bella_sf_admin",
  	password=re.findall("SF_DB_ADMIN_PASSWORD=\'(.+)\'", db_profile)[0],
  	authSource="bella_sf")["bella_sf"]

• synapse/dashboard/model_manager.py

  Lines 156 to 161 in b315707

  	if self.__is_gaussian_process:
  	# TODO use "exp" only once experimental data is available for all experiments
  	task_tag = "exp" if state.experiment == "ip2" else "sim"
  	output_key = [key for key in output_dict.keys() if task_tag in key][
  	0
  	]

[EZoni]

@RemiLehe

I think this is the remaining BELLA-specific code:

synapse/dashboard/model_manager.py

Lines 156 to 163 in ceea73f

	if self.__is_gaussian_process:
	# TODO use "exp" only once experimental data is available for all experiments
	task_tag = "exp" if state.experiment == "bella-ip2" else "sim"
	output_key = [key for key in output_dict.keys() if task_tag in key][
	0
	]
	elif self.__is_neural_network_ensemble:
	output_key = list(output_dict.keys())[0]

Do you know if this is still needed? Even if so, maybe there's another way to handle this if/else logic without checking the experiment name itself.