adding support for alias files (#557)

* adding support for alias files
an alias file is a yaml file "lookup" alongside a container.yaml
that can be used to provide tag specific commands for a particular
container tag
* updating alias-files -> overrides

Signed-off-by: vsoch <[email protected]>
Co-authored-by: Matthieu Muffato <[email protected]>

Author: vsoch

.github/workflows/test-container.yml

@@ -33,12 +33,15 @@ jobs:
         run: |
           # Continue if we have a changed recipe file
           if [[ "${{ matrix.changed_recipe }}" = registry* ]]; then
-            echo "keepgoing=true" >> $GITHUB_ENV
+            recipe_basename=$(basename "${{ matrix.changed_recipe }}")
+            if [[ "${recipe_basename}" == "container.yaml" ]]; then  
+              echo "keepgoing=true" >> $GITHUB_ENV
+            fi
           fi          
 
       - name: Checkout
         if: ${{ env.keepgoing == 'true' }}
-        uses: actions/checkout@v2        
+        uses: actions/checkout@v3
 
       - name: Make Space For Build
         if: ${{ env.keepgoing == 'true' }}

CHANGELOG.md

@@ -14,6 +14,7 @@ and **Merged pull requests**. Critical items to know are:
 The versions coincide with releases on pip. Only major versions will be released as tags on Github.
 
 ## [0.0.x](https://github.com/singularityhub/singularity-hpc/tree/main) (0.0.x)
+ - adding support for container.yaml overrides (0.0.56)
  - `SINGULARITY_CONTAINER` and `PODMAN_CONTAINER` support (0.0.55)
  - support for views create, delete, edit, install, uninstall (0.0.54)
   - also including cleanup of module in views on uninstall

docs/getting_started/developer-guide.rst

@@ -198,6 +198,50 @@ For each of the above, depending on the prefix of options that you choose, it wi
 This means that if you design a new registry recipe, you should consider how to run it for both kinds of technology. Also note that ``docker_options`` are
 those that will also be used for Podman.
 
+Overrides
+---------
+
+It might be the case that as your containers change over time, the set of any of:
+
+- commands (aliases)
+- docker_script
+- singularity_script
+- environment (env)
+- features
+- description
+
+does too! Or it be the case that you have hundreds of aliases, and want to better organize them separately from the container.yaml. To support this, shpc
+(as of version 0.0.56) has support for an ``overrides`` section in the container.yaml, meaning that you can define pairs of container
+tags and relative path lookups to external files with any of the stated sections. A simple example might look like this:
+
+.. code-block:: yaml
+
+    docker: python
+    url: https://hub.docker.com/_/python
+    maintainer: '@vsoch'
+    description: An interpreted, high-level and general-purpose programming language.
+    latest:
+      3.9.5-alpine: sha256:f189f7366b0d381bf6186b2a2c3d37f143c587e0da2e8dcc21a732bddf4e6f7b
+    tags:
+      3.9.2-alpine: sha256:f046c06388c0721961fe5c9b6184d2f8aeb7eb01b39601babab06cfd975dae01
+    overrides:
+      3.9.2-alpine: aliases/3.9.2-alpine.yaml
+    aliases:
+        python: /usr/local/bin/python
+
+
+Since this file only has aliases, we chose to use a subdirectory called "aliases" to make that clear, however
+the file can have any of the fields mentioned above, and can be organized in any relative path to the container directory that you deem apppropriate.
+Here is what this corresponding file with relative path ``aliases/3.9.2-alpine.yaml`` might look like this:
+
+.. code-block:: yaml
+
+    aliases:
+      python: /alias/path/to/python
+
+Finally, for all fields mentioned above, the format is expected to follow the same convention as above (and it will be validated again on update).
+
+
 Wrapper Script
 --------------
 
@@ -514,6 +558,9 @@ Fields include:
    * - aliases
      - Named entrypoints for container (dict) as described above
      - false
+   * - overrides
+     - Key value pairs to override container.yaml defaults.
+     - false
    * - url
      - Documentation or other url for the container uri
      - false

registry/quay.io/pawsey/openfoam-org/aliases/2.2.0.yaml

@@ -0,0 +1,247 @@
+aliases:
+  Co: Co
+  LTSInterFoam: LTSInterFoam
+  LTSReactingParcelFoam: LTSReactingParcelFoam
+  Lambda2: Lambda2
+  MRFInterFoam: MRFInterFoam
+  MRFMultiphaseInterFoam: MRFMultiphaseInterFoam
+  Mach: Mach
+  PDRFoam: PDRFoam
+  PDRMesh: PDRMesh
+  Pe: Pe
+  Q: Q
+  R: R
+  SRFPimpleFoam: SRFPimpleFoam
+  SRFSimpleFoam: SRFSimpleFoam
+  XiFoam: XiFoam
+  adiabaticFlameT: adiabaticFlameT
+  adjointShapeOptimizationFoam: adjointShapeOptimizationFoam
+  ansysToFoam: ansysToFoam
+  applyBoundaryLayer: applyBoundaryLayer
+  applyWallFunctionBoundaryConditions: applyWallFunctionBoundaryConditions
+  attachMesh: attachMesh
+  autoPatch: autoPatch
+  autoRefineMesh: autoRefineMesh
+  blockMesh: blockMesh
+  boundaryFoam: boundaryFoam
+  boxTurb: boxTurb
+  buoyantBoussinesqPimpleFoam: buoyantBoussinesqPimpleFoam
+  buoyantBoussinesqSimpleFoam: buoyantBoussinesqSimpleFoam
+  buoyantPimpleFoam: buoyantPimpleFoam
+  buoyantSimpleFoam: buoyantSimpleFoam
+  cavitatingDyMFoam: cavitatingDyMFoam
+  cavitatingFoam: cavitatingFoam
+  cfx4ToFoam: cfx4ToFoam
+  changeDictionary: changeDictionary
+  checkMesh: checkMesh
+  chemFoam: chemFoam
+  chemkinToFoam: chemkinToFoam
+  chtMultiRegionFoam: chtMultiRegionFoam
+  chtMultiRegionSimpleFoam: chtMultiRegionSimpleFoam
+  coalChemistryFoam: coalChemistryFoam
+  coldEngineFoam: coldEngineFoam
+  collapseEdges: collapseEdges
+  combinePatchFaces: combinePatchFaces
+  compressibleInterDyMFoam: compressibleInterDyMFoam
+  compressibleInterFoam: compressibleInterFoam
+  compressibleTwoPhaseEulerFoam: compressibleTwoPhaseEulerFoam
+  createBaffles: createBaffles
+  createPatch: createPatch
+  createTurbulenceFields: createTurbulenceFields
+  datToFoam: datToFoam
+  decomposePar: decomposePar
+  deformedGeom: deformedGeom
+  dnsFoam: dnsFoam
+  dsmcFieldsCalc: dsmcFieldsCalc
+  dsmcFoam: dsmcFoam
+  dsmcInitialise: dsmcInitialise
+  electrostaticFoam: electrostaticFoam
+  engineCompRatio: engineCompRatio
+  engineFoam: engineFoam
+  engineSwirl: engineSwirl
+  enstrophy: enstrophy
+  equilibriumCO: equilibriumCO
+  equilibriumFlameT: equilibriumFlameT
+  execFlowFunctionObjects: execFlowFunctionObjects
+  expandDictionary: expandDictionary
+  extrude2DMesh: extrude2DMesh
+  extrudeMesh: extrudeMesh
+  extrudeToRegionMesh: extrudeToRegionMesh
+  faceAgglomerate: faceAgglomerate
+  financialFoam: financialFoam
+  fireFoam: fireFoam
+  flattenMesh: flattenMesh
+  flowType: flowType
+  fluent3DMeshToFoam: fluent3DMeshToFoam
+  fluentMeshToFoam: fluentMeshToFoam
+  foamCalc: foamCalc
+  foamDataToFluent: foamDataToFluent
+  foamDebugSwitches: foamDebugSwitches
+  foamFormatConvert: foamFormatConvert
+  foamHelp: foamHelp
+  foamInfoExec: foamInfoExec
+  foamListTimes: foamListTimes
+  foamMeshToFluent: foamMeshToFluent
+  foamToEnsight: foamToEnsight
+  foamToEnsightParts: foamToEnsightParts
+  foamToGMV: foamToGMV
+  foamToStarMesh: foamToStarMesh
+  foamToSurface: foamToSurface
+  foamToTetDualMesh: foamToTetDualMesh
+  foamToVTK: foamToVTK
+  foamUpgradeCyclics: foamUpgradeCyclics
+  foamUpgradeFvSolution: foamUpgradeFvSolution
+  gambitToFoam: gambitToFoam
+  gmshToFoam: gmshToFoam
+  icoFoam: icoFoam
+  icoUncoupledKinematicParcelDyMFoam: icoUncoupledKinematicParcelDyMFoam
+  icoUncoupledKinematicParcelFoam: icoUncoupledKinematicParcelFoam
+  ideasUnvToFoam: ideasUnvToFoam
+  insideCells: insideCells
+  interDyMFoam: interDyMFoam
+  interFoam: interFoam
+  interMixingFoam: interMixingFoam
+  interPhaseChangeFoam: interPhaseChangeFoam
+  kivaToFoam: kivaToFoam
+  laplacianFoam: laplacianFoam
+  magneticFoam: magneticFoam
+  mapFields: mapFields
+  mdEquilibrationFoam: mdEquilibrationFoam
+  mdFoam: mdFoam
+  mdInitialise: mdInitialise
+  mergeMeshes: mergeMeshes
+  mergeOrSplitBaffles: mergeOrSplitBaffles
+  mhdFoam: mhdFoam
+  mirrorMesh: mirrorMesh
+  mixtureAdiabaticFlameT: mixtureAdiabaticFlameT
+  modifyMesh: modifyMesh
+  moveDynamicMesh: moveDynamicMesh
+  moveEngineMesh: moveEngineMesh
+  moveMesh: moveMesh
+  mshToFoam: mshToFoam
+  multiphaseEulerFoam: multiphaseEulerFoam
+  multiphaseInterFoam: multiphaseInterFoam
+  netgenNeutralToFoam: netgenNeutralToFoam
+  nonNewtonianIcoFoam: nonNewtonianIcoFoam
+  objToVTK: objToVTK
+  pPrime2: pPrime2
+  particleTracks: particleTracks
+  patchAverage: patchAverage
+  patchIntegrate: patchIntegrate
+  patchSummary: patchSummary
+  pdfPlot: pdfPlot
+  pimpleDyMFoam: pimpleDyMFoam
+  pimpleFoam: pimpleFoam
+  pisoFoam: pisoFoam
+  plot3dToFoam: plot3dToFoam
+  polyDualMesh: polyDualMesh
+  porousInterFoam: porousInterFoam
+  porousSimpleFoam: porousSimpleFoam
+  postChannel: postChannel
+  potentialFoam: potentialFoam
+  potentialFreeSurfaceFoam: potentialFreeSurfaceFoam
+  probeLocations: probeLocations
+  ptot: ptot
+  reactingFoam: reactingFoam
+  reactingParcelFilmFoam: reactingParcelFilmFoam
+  reactingParcelFoam: reactingParcelFoam
+  reconstructPar: reconstructPar
+  reconstructParMesh: reconstructParMesh
+  redistributePar: redistributePar
+  refineHexMesh: refineHexMesh
+  refineMesh: refineMesh
+  refineWallLayer: refineWallLayer
+  refinementLevel: refinementLevel
+  removeFaces: removeFaces
+  renumberMesh: renumberMesh
+  rhoCentralDyMFoam: rhoCentralDyMFoam
+  rhoCentralFoam: rhoCentralFoam
+  rhoLTSPimpleFoam: rhoLTSPimpleFoam
+  rhoPimpleFoam: rhoPimpleFoam
+  rhoPimplecFoam: rhoPimplecFoam
+  rhoPorousSimpleFoam: rhoPorousSimpleFoam
+  rhoReactingBuoyantFoam: rhoReactingBuoyantFoam
+  rhoReactingFoam: rhoReactingFoam
+  rhoSimpleFoam: rhoSimpleFoam
+  rhoSimplecFoam: rhoSimplecFoam
+  rotateMesh: rotateMesh
+  sammToFoam: sammToFoam
+  sample: sample
+  scalarTransportFoam: scalarTransportFoam
+  selectCells: selectCells
+  setFields: setFields
+  setSet: setSet
+  setsToZones: setsToZones
+  settlingFoam: settlingFoam
+  shallowWaterFoam: shallowWaterFoam
+  simpleFoam: simpleFoam
+  simpleReactingParcelFoam: simpleReactingParcelFoam
+  singleCellMesh: singleCellMesh
+  smapToFoam: smapToFoam
+  snappyHexMesh: snappyHexMesh
+  solidDisplacementFoam: solidDisplacementFoam
+  solidEquilibriumDisplacementFoam: solidEquilibriumDisplacementFoam
+  sonicDyMFoam: sonicDyMFoam
+  sonicFoam: sonicFoam
+  sonicLiquidFoam: sonicLiquidFoam
+  splitCells: splitCells
+  splitMesh: splitMesh
+  splitMeshRegions: splitMeshRegions
+  sprayEngineFoam: sprayEngineFoam
+  sprayFoam: sprayFoam
+  star3ToFoam: star3ToFoam
+  star4ToFoam: star4ToFoam
+  steadyParticleTracks: steadyParticleTracks
+  stitchMesh: stitchMesh
+  streamFunction: streamFunction
+  stressComponents: stressComponents
+  subsetMesh: subsetMesh
+  surfaceAdd: surfaceAdd
+  surfaceAutoPatch: surfaceAutoPatch
+  surfaceBooleanFeatures: surfaceBooleanFeatures
+  surfaceCheck: surfaceCheck
+  surfaceClean: surfaceClean
+  surfaceCoarsen: surfaceCoarsen
+  surfaceConvert: surfaceConvert
+  surfaceFeatureConvert: surfaceFeatureConvert
+  surfaceFeatureExtract: surfaceFeatureExtract
+  surfaceFind: surfaceFind
+  surfaceInertia: surfaceInertia
+  surfaceMeshConvert: surfaceMeshConvert
+  surfaceMeshConvertTesting: surfaceMeshConvertTesting
+  surfaceMeshExport: surfaceMeshExport
+  surfaceMeshImport: surfaceMeshImport
+  surfaceMeshInfo: surfaceMeshInfo
+  surfaceMeshTriangulate: surfaceMeshTriangulate
+  surfaceOrient: surfaceOrient
+  surfacePointMerge: surfacePointMerge
+  surfaceRedistributePar: surfaceRedistributePar
+  surfaceRefineRedGreen: surfaceRefineRedGreen
+  surfaceSmooth: surfaceSmooth
+  surfaceSplitByPatch: surfaceSplitByPatch
+  surfaceSplitByTopology: surfaceSplitByTopology
+  surfaceSplitNonManifolds: surfaceSplitNonManifolds
+  surfaceSubset: surfaceSubset
+  surfaceToPatch: surfaceToPatch
+  surfaceTransformPoints: surfaceTransformPoints
+  temporalInterpolate: temporalInterpolate
+  tetgenToFoam: tetgenToFoam
+  topoSet: topoSet
+  transformPoints: transformPoints
+  twoLiquidMixingFoam: twoLiquidMixingFoam
+  twoPhaseEulerFoam: twoPhaseEulerFoam
+  uncoupledKinematicParcelFoam: uncoupledKinematicParcelFoam
+  uprime: uprime
+  viewFactorsGen: viewFactorsGen
+  vorticity: vorticity
+  vtkUnstructuredToFoam: vtkUnstructuredToFoam
+  wallFunctionTable: wallFunctionTable
+  wallGradU: wallGradU
+  wallHeatFlux: wallHeatFlux
+  wallShearStress: wallShearStress
+  wdot: wdot
+  writeCellCentres: writeCellCentres
+  writeMeshObj: writeMeshObj
+  yPlusLES: yPlusLES
+  yPlusRAS: yPlusRAS
+  zipUpMesh: zipUpMesh