Crash during column family creation can lead to inconsistent state

[ischeinkman]

Steps to reproduce:

1. Use rocksdb_create_column_family to attempt to create a column family with non-default options set.
2. Crash before the function finishes.
3. On restart, get the latest column family options with rocksdb_load_latest_options
4. Attempt to re-open the database with rocksdb_open_column_families

Expected results

The database is successfully opened, either with the new column family configured with the non-default options applied or without the new column family.

Actual results

The rocksdb_open_column_families emits a Column families not opened error for the new column family.

Cause

Within the database directory we do have a new OPTIONS-$IDX.dbtmp file containing the options for the new instance. I suspect that the MANIFEST file finished serializing before the OPTIONS file due to the OPTIONS file's 2 stage process. Manually running mv OPTIONS-$IDX.dbtmp OPTIONS-$IDX fixed the issue.

Would it be possible to finish serializing the OPTIONS completely before we serialize to MANIFEST?