Continuous datasets onboarding and updates #93
Description
StatGPT Backend version
Latest.
What is the problem this feature will solve?
The StatGPT system must respond correctly to the following changes in datasets:
- New series added to the indexed dataset.
- The structure of the dataset has changed without changing the version.
- A new version of the dataset has been released.
- A combination of the above.
What is the proposed feature or solution?
Support continuous datasets onboarding and updates with minimal involvement of system administrators in case the changes to datasets are not breaking.
What alternatives have you considered?
The current approach is to manually update the dataset configuration whenever there are changes in the source system. This approach works well when the number of datasets is small, but becomes tedious as they increase.