feat(POC): add run time config management service

[Junjiequan]

Description

This PR introduces a runtime configuration system.

Introduced runtime-config module:

• Controller
• Service
• DTO
• MongoDB schema

Added new API endpoints:

• GET /api/v3/runtime-config/:cid -> Public
• PUT /api/v3/runtime-config/:cid -> Only ADMIN users can access

Supported configuration IDs:

• Defaults:
  • frontendConfig
  • frontendTheme
• Can be extended using:
  • CONFIG_SYNC_TO_DB_LIST ENV variable

On application startup:

• backend reads CONFIG_SYNC_TO_DB_LIST
• For each cid in the list:
  • If record does not exist -> create from listed config file or from defaults
  • If record exists -> overwrite using config file
  • iI cid corresponding config file do not exists -> throw log error

📌📌 Config files are always the source of truth 📌📌

Minor admin/config endpoint workflow changes:

• admin/config now internally call runtime-config/frontendConfig
• admin/theme now internally call runtime-config/frontendTheme

Both endpoints still exist.
Only the internal flow changed, with no breaking API changes.

Motivation

This feature adds a DB-backed config store so the frontend config can be edited at runtime without rebuilding. The backend still treats the JSON files as the source of truth and auto-loads them into the Storage collection on startup (POC scope is only frontend config, even though the backend can load any config file). This makes it easier to test UI changes, share configs across environments, and edit them easily through JSONForms.

Fixes

• Bug fixed (#X)

Changes:

• changes made

Tests included

• Included for each change/fix?
• Passing?

Documentation

• swagger documentation updated (required for API changes)
• official documentation updated

official documentation info

[sbliven]

@Junjiequan Can you add an example for how to use this?

[bpedersen2]

Sound good in general.

I guess a filter for certain fields will be needed in the end if we allow be-config changes, e.g. to avoid loss of all admin groups or changes to mongodb settings( I would probably keep all security-related fields as file-only).

Frontend-config should be less critical.

[sbliven]

I think we should discuss the core premise of this POC, which is that the configuration should be stored in the database instead of in files. While it would be convenient to be able to change the configuration on the fly, I'm worried that it will become even harder to track configuration changes. If we do it, I think there should be a clear process to "burn in" the configuration: export non-default settings to an env file, empty the database, and restart the backend with a clean configuration. Otherwise it's hard to keep track of changes.

Can you add a description of the usage to the README or docs? Even if it changes later, it would be helpful to read how the merging works. I think you said that the database is considered canon and overrides anything in the config files or the defaults.

How does this fit with /admin/config? Would this provide the merged "frontend" data?

Likewise, will calls to the configService be merged to the "backend" data? If so, we should double-check that there aren't any remaining direct calls to configuration(), as those would bypass the configService.

I guess you would add more endpoints

• get the full, merged configuration
• get/set individual properties (or would this just be done with PATCH application/json+merge?)
• way to delete configuration changes (resetting from the files or defaults)

[Junjiequan]

I think we should discuss the core premise of this POC, which is that the configuration should be stored in the database instead of in files. While it would be convenient to be able to change the configuration on the fly, I'm worried that it will become even harder to track configuration changes. If we do it, I think there should be a clear process to "burn in" the configuration: export non-default settings to an env file, empty the database, and restart the backend with a clean configuration. Otherwise it's hard to keep track of changes.

We defintelly needs to discuss more about this, I just wanted to pull the idea into code, so we can have something to disccuss upon.

If i understand your proposal correctly, you are suggesting that:

1. the backend always starts with a empty config collection
2. it then loads the config files from disk (mounted or default)
3. creates a fresh config record based on provided files

So, the config file is the source of truth, config record is temporary copy for real time editing. once the backend restarts the temporary state resets and rebuilt from the file again.

Is it correct intepretation?

Can you add a description of the usage to the README or docs? Even if it changes later, it would be helpful to read how the merging works. I think you said that the database is considered canon and overrides anything in the config files or the defaults.

Sure, I will provide it as soon as possible.

How does this fit with /admin/config? Would this provide the merged "frontend" data?

It's up for discussion that wheter if we still want to keep /admin/config after we have the runtimeConfig endpoint.

Likewise, will calls to the configService be merged to the "backend" data? If so, we should double-check that there aren't any remaining direct calls to configuration(), as those would bypass the configService.

The backend data ( environment variables I assume ) is completly seperated logic and should not be relevant to the run time config service at all, they are out of scope this time at least. The run time service serves for frontend related configs only.

I guess you would add more endpoints

get the full, merged configuration
get/set individual properties (or would this just be done with PATCH application/json+merge?)
way to delete configuration changes (resetting from the files or defaults)

I imagine the entry sturcture will look like this:

[
  {
    "_id": "frontendConfig",
    "description": "Some text to be rendered on the UI",
    "data": {
      "...": "raw JSON blob"
    }
  },
  {
    "_id": "frontendTheme",
    "data": {
      "...": "raw JSON blob"
    }
  },
  {
    "_id": "someOtherConfig",
    "data": {
      "...": "raw JSON blob"
    }
  }
]

On the frontend we use jsonforms, which always submits the full JSON object. So, I’m not sure individual property endpoints add much value right now. JSONForms doesn’t send partial updates, so we’d essentially be duplicating logic.
For now, setting/getting the whole config blob seems enough, unless we decide to go with other json editors.

[nitrosx]

@sbliven valid feedback!!!
Let me and Jay brainstorm once or twice more and get back to you.
We can discuss afterwards.

[Junjiequan]

@sbliven @nitrosx
Most of the comments are addressed, documentation included along with updated PR description.
The description field is removed from the runtimeConfig record as it is not going to be used now, but we can add it back when we need it.

[nitrosx]

Minor changes to the swagger controller tags.

Food for thoughts: we should add authorization to the get configuration endpoint, so admins can restrict access to some entries. The authorization is not for the endpoint is for each individual entry.
...but this is for the future

Requested changes are addressed

[Junjiequan]

@sbliven All the previous comments have been addressed. if there's no more comments, I'm planning to merge the PR in this week as this PR has been stalled for quite long time.