Move field configs out of the split and back to the sync directory

[imonroe]

READY FOR REVIEW

Summary

• Field storage configs can't live in a config_split because if there is existing data in those fields, when a drush config:import happens, it performs the import based on the config/sync configurations only. When that happens, it finds that there are fields in the database which are not accounted for in the base configuration. So it marks all the entries in those tables as deleted. The next time cron runs, it then removes the data marked for deletion from the database. This causes all those fields to lose all their records.
• After the initial config:import occurs, the config_split is applied, but it doesn't un-mark the data for deletion.
• Thus, even if the data is restored, e.g., from a backup, the next time configuration is imported, the field data will be deleted again.
• This caused a great deal of data loss for the CASBS site, wiping out all the information from eight custom fields. Even though the field tables technically exist, all the records get lost.
• Here, we change the config split to include the field storage configs in the main sync directory to ensure the data is not deleted upon config import.
• Additionally, many configurations had dependencies on configs which would not be imported until after the config_split was applied. This causes installation failures during the ci/cd tests.
• Thus, we also move the minimum number of dependent configs back into the main sync directory, to allow the profile to install correctly during testing.
• We keep the config split in place to keep the base sync directory as clean as possible, but the split is less aggressive.
• The theme customizations were not preserved from the D11 update. Here, we restore those customizations and recompile the theme.

Notes on data recovery in this instance

• I built a copy of the D10 version of the site locally, and I restored from the manual backup taken from production prior to the introduction of the config_split. That got me all the data from the missing tables.
• In my current local D11 version of the site, I imported the latest DB + files from the CASBS production site (data missing from the affected fields).
• I exported the tables related to the affected fields from the D10 version, and imported them into the updated D11 database, restoring the data.
• I did a drush deploy on the D11 version to make sure all the db updates were applied, and all the configuration was restored to D11 status. That gives me a D11 version of the DB including the lost data.
• I pushed the now-fixed database (and files) up to the DEV environment.
• Once the tests passed on this PR, I switched the code on the DEV environment to this branch to ensure it would apply correctly and the lost data was available.
• I found I needed to do a drush @default.dev deploy a couple of times to get everything applied cleanly. Subsequent deployments to the DEV server applied without issue.

Next steps

• Bruce should test his JSONAPI processes against the DEV server to ensure they continue to function.
• Since the DEV server has the only intact version of all the data, we should consider a brief content freeze and promote it to production. This should be done prior to deploying an updated tagged version to PROD.

Review By (Date)

• ASAP

Urgency

• High

Steps to Test

1. Sanity check the updated config_split to ensure there are no remaining issues
2. Ensure tests pass correctly
3. Front end check on the DEV server
4. Make sure Bruce can run his processes correctly

Affected Projects or Products

• Does this PR impact any particular projects, products, or modules?

Associated Issues and/or People

• JIRA ticket
• Other PRs
• Any other contextual information that might be helpful (e.g., description of a bug that this PR fixes, new functionality that it adds, etc.)
• Anyone who should be notified? (@mention them here)

See Also

• PR Checklist

[pookmish]

Field storage configs can't live in a config_split because if there is existing data in those fields

That is not correct. Review how lagunita has many instances. the config split allows us to keep site specific fields in it's own location.

[imonroe]

Field storage configs can't live in a config_split because if there is existing data in those fields

That is not correct. Review how lagunita has many instances. the config split allows us to keep site specific fields in it's own location.

Why was all the data lost from those fields then?

[pookmish]

Why was all the data lost from those fields then?

I can review the situation more another day.

[pookmish]

I think the problem is that the config split needed to be created prior to deploying the code. That would've allowed the system to see the field storage configs in their proper locations and retained the tables in the database.

[imonroe]

That makes sense to me. In that case, I think we should deploy this, restore the lost data, and then re-introduce the field storage splits once it's active on the Acquia environments. That way, the environment knows about the split, and the split files will be included in the unified configuration during future deployments.

[pookmish]

How would deploying this solve the problem? doesn't the application already have the config split on the site? or did you restore from a backup?

If you restored from a backup, just create a db update hook to create the config split. Ensure the uuid and all settings are reflected in the update. Then it should work fine.

[imonroe]

Actually, it looks like just promoting the database will do the trick. Since the database on the DEV server is based on the PROD db with only the messed up tables restored from the backup (see above), it's already aware of the split. I just tested this locally by moving back to the 12x branch and doing a drush deploy, and the deleted fields remained with no issue. One way or another we've got to restore the lost data, and it has to come from a backup, so that's likely the quickest fix. The data loss was reported yesterday, so I'm eager to get them back to a working state quickly. I can open another PR for restoring the theme customizations, and we can just close this one. Any reason why not to do that?

[pookmish]

That seems fine to me. I can't think of a reason not to. The alternative is to restore the prod environment with the backup. Then you could use drush to create the config split and then just drush deploy everything. Sounds like similar outcomes. Keep the db backup just in case we need it.

[imonroe]

Closing in lieu of #93