Add support for additional metapackages

[rhoerr]

A longtime desire has been to support multiple Mage-OS Distribution variants for different use cases, like

• Mage-OS Minimal
• Mage-OS Content Management
• Mage-OS Headless

There are code issues here with modules depending on other modules they probably shouldn't -- mage-os/mageos-magento2#166 addresses that.

But setting that aside: To do this, we'll need to update the release generator to support creating multiple metapackages (different versions of mage-os/product-community-edition) for each variant. Right now the build config and instructions only support the one metapackage explicitly.

[rhoerr]

Pulling this off will require modifying product-community-edition in ways we don't currently.

For example:

The built composer file includes all modules, from the root replace values during the build:

https://github.com/mage-os/magento2-base-composer-json/blob/main/2.4.8-p3/product-community-edition/composer.json

Whereas the template file only includes addon modules:
https://github.com/mage-os/generate-mirror-repo-js/blob/main/resource/composer-templates/magento/product-community-edition/dependencies-template.json

So it'll be necessary to define additional templates with package exclusions, or transformation rules to skip packages, or some similar approach.

Right now the requires are set as referenceRequires + referenceReplaces + templateDependencies, where reference is the built ./composer.json, and template is the dependencies-template.json above.

A transform could do it. In that event, we would still need to add some mechanism to define the additional metapackages and packages to exclude for each. That's going to require maintenance over time. Also would have to be accounted for with the mirror rebuilding.

[fballiano]

could it be done with adding replaces to the default installation? so that it's only the product-community-edition + some replaces (not super clean but the end result should be the same)

[rhoerr]

could it be done with adding replaces to the default installation? so that it's only the product-community-edition + some replaces (not super clean but the end result should be the same)

Interesting idea but I think that would block people from using those packages then. If we say metapackage-minimal replaces module-review:*, then module-review wouldn't get installed, but also it wouldn't be possible to separately require module-review to install it intentionally. As far as composer is concerned it would already be installed. (Happy to be wrong here)

[rhoerr]

My current working theory is to change how we define and build metapackages, so that instead of flags and special cases like this:

generate-mirror-repo-js/src/build-config/packages-config.js

Lines 5 to 6 in 649814e

	magentoCommunityEditionProject: true,
	magentoCommunityEditionMetapackage: true,

... we have some additional packageMeta array or something that builds a metapackage based on certain config/constraints/transforms to be determined. Then we can use that for building the product and project packages, but also any others we see fit (like minimal), using the same mechanism, same code paths, less special cases.

Requires further investigation.

[rhoerr]

Claude analysis (have not evaluated in depth yet):

Based on the codebase analysis, here's my suggested approach to refactoring the metapackage generation system:

1. First, let's identify the key components involved:

• repositoryBuildDefinition has boolean flags magentoCommunityEditionMetapackage and magentoCommunityEditionProject that control metapackage creation
• Metapackage creation logic is in package-modules.js with createMagentoCommunityEditionMetapackage() and createMagentoCommunityEditionProject()
• The package build process happens in processBuildInstructions() in release-build-tools.js
• Templates and dependencies are stored in composer-templates

2. The main issues with the current implementation:

• Hardcoded metapackage types (community edition product and project)
• Fixed dependency resolution from magento2-base
• No way to define custom package subsets
• Tightly coupled package building logic

3. Proposed refactoring approach:

// New type definition
interface MetapackageDefinition {
  name: string;            // Package name (e.g. "mage-os/product-minimal")
  type: string;           // "project" or "metapackage" 
  description: string;    // Package description
  basePackage?: string;   // Optional base package to inherit from
  include?: string[];     // Packages to include
  exclude?: string[];     // Packages to exclude
  transform?: Function[]; // Transform functions to apply
}

// In repository-build-definition.js
class repositoryBuildDefinition {
  metapackages: MetapackageDefinition[] = [];
  // Remove magentoCommunityEditionMetapackage and magentoCommunityEditionProject flags
}

4. Implementation changes:

// In package-modules.js
async function createMetapackage(instruction, release, definition) {
  const {name, type, description, basePackage, include, exclude, transform} = definition;
  
  // Get base dependencies
  let dependencies = {};
  if (basePackage) {
    dependencies = await getBaseDependencies(instruction, basePackage);
  }

  // Filter dependencies
  dependencies = filterDependencies(dependencies, {
    include,
    exclude
  });

  // Create package
  const composerConfig = {
    name,
    type,
    description,
    require: dependencies
  };

  // Apply transforms
  const finalConfig = (transform || []).reduce(
    (config, fn) => fn(config, instruction, release),
    composerConfig
  );

  return createComposerJsonOnlyPackage(
    instruction,
    release, 
    name,
    release.version,
    () => finalConfig
  );
}

5. Update build process:

// In release-build-tools.js
async function processBuildInstructions(instruction, release) {
  // ... existing package building code ...

  // Build metapackages
  for (const metapackage of instruction.metapackages) {
    console.log(`Building metapackage ${metapackage.name}`);
    const built = await createMetapackage(
      instruction,
      release,
      metapackage
    );
    Object.assign(packages, built);
  }
}

6. Example configuration:

// In mageos-release-build-config.js
{
  magento2: {
    repoUrl: '...',
    metapackages: [
      // Original community edition metapackage
      {
        name: 'mage-os/product-community-edition',
        type: 'metapackage',
        description: 'Mage-OS Community Edition',
        basePackage: 'magento2-base',
        transform: [/* existing transforms */]
      },
      // New minimal edition
      {
        name: 'mage-os/product-minimal',
        type: 'metapackage',
        description: 'Minimal Mage-OS Edition',
        basePackage: 'magento2-base',
        exclude: [
          'mage-os/module-page-builder',
          'mage-os/module-adobe-*'
        ]
      }
    ]
  }
}

This approach provides several benefits:

1. Flexible metapackage definitions through configuration
2. Reusable package filtering logic
3. Support for inheritance from base packages
4. Maintains existing transformation capabilities
5. Backwards compatibility by defining the original metapackages in the new format

Changes would also be needed to resource/history/mage-os/ to store historical metapackages, and to the mirror build code to include those. Possibly:

1. Add a helper function to load historical metadata:

// Add near the top with other imports
const historyPath = path.join(__dirname, '../resource/history');

async function loadHistoricalMetadata(vendor, metapackage, version) {
  const filepath = path.join(
    historyPath,
    vendor,
    metapackage.historyPath,
    `${version}.json`
  );

  try {
    const data = await fs.promises.readFile(filepath, 'utf8');
    return JSON.parse(data);
  } catch (error) {
    console.warn(`No historical data found for ${metapackage.name} ${version}`);
    return null;
  }
}

2. Update the historical file structure to support the new format. Create a new file structure for additional metapackages:

mkdir -p /var/www/mage-os/mage-os.generate-mirror-repo-js/resource/history/mage-os/product-minimal/

3. Example historical metadata file for a new minimal product:

{
  "name": "mage-os/product-minimal",
  "type": "metapackage",
  "description": "Mage-OS Minimal Edition",
  "version": "1.0.0",
  "require": {
    "mage-os/magento2-base": "1.0.0",
    "mage-os/module-catalog": "1.0.0",
    "mage-os/module-checkout": "1.0.0"
  },
  "exclude": [
    "mage-os/module-page-builder",
    "mage-os/module-adobe-*"
  ]
}

I also generated mage-os-generate-mirror-repo-js-analysis.md to help with the analysis.

[rhoerr]

I started implementing earlier. There are some gaps in that approach, but it's a good start.

It will take a while finished the changes and then thorough testing.

[rhoerr]

Refactoring is going well. I started some testing earlier. Once I have equivalent behavior for the existing metapackages, I'll focus on adding a mock new one. I figure another 5-10 hours to wrap it all up.

[rhoerr]

Generating a new release with standard and additional metapackages works now. I need to handle historical releases still.