Skip to content

Allow specifying path to schema-sync folder with SCHEMA_SYNC_PATH #72

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
julbd wants to merge 1 commit into
base: main
Choose a base branch
from
Open

Allow specifying path to schema-sync folder with SCHEMA_SYNC_PATH #72

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 3 additions & 2 deletions README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -59,7 +59,7 @@ volumes:
(re)Build and run your container.

Once it is running, run the following command (from host) to install the extension's columns in the database and add the config folder.

Replace the `my-directus` with the name of your service running directus if it is different

```bash
Expand Down Expand Up @@ -125,6 +125,7 @@ onImport: async (item, itemsSrv) => {
| Variable | Description | Default |
| -------- | ----------- | ------- |
| `SCHEMA_SYNC` | Set to automatically do **IMPORT**, **EXPORT** or **BOTH** | `null` |
| `SCHEMA_SYNC_PATH` | Path to the `schema-sync` folder | (Directus root)/schema-sync |
| `SCHEMA_SYNC_CONFIG` | (optional) An additional config file to use in addition, eg. `test_config.js` | `null` |
| `SCHEMA_SYNC_SPLIT` | (optional) Splits the schema file into multiple files once per collection | `true` |
| `SCHEMA_SYNC_MERGE` | (optional) Only insert and update items found in the import set (including duplicates). Does not remove items in the DB that are not in the import set. | `false` |
Expand All @@ -151,7 +152,7 @@ Besides auto importing and exporting, you can also run the commands manually.
Update the `schema-sync/directus_config.js` file with the following:

Replace `directus_roles`
Add `directus_policies`
Add `directus_policies`
Replace `directus_permissions`
Add `directus_access`

Expand Down
15 changes: 9 additions & 6 deletions src/collectionExporter.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,7 +3,8 @@ import type { ApiExtensionContext } from '@directus/extensions';
import { mkdir, readFile, rm, writeFile } from 'fs/promises';
import { condenseAction } from './condenseAction.js';
import type { CollectionExporterOptions, IExporter, IGetItemsService, ItemsService, ToUpdateItemDiff } from './types';
import { ExportHelper, getDiff, sortObject } from './utils.js';
import { ExportMeta } from './exportMeta.js';
import { getDiff, sortObject, fileExists } from './utils.js';
import { glob } from 'glob';

type PARTIAL_CONFIG = { count: number; groupedBy: string[]; partial: true };
Expand All @@ -18,6 +19,7 @@ const DEFAULT_COLLECTION_EXPORTER_OPTIONS: CollectionExporterOptions = {

class CollectionExporter implements IExporter {
protected _getService: () => Promise<ItemsService>;
protected _exportMeta: ExportMeta;
protected collection: string;

protected options: CollectionExporterOptions;
Expand All @@ -44,13 +46,14 @@ class CollectionExporter implements IExporter {

this.collection = collectionName;

this._exportMeta = new ExportMeta(options.path);
const fileName = this.options.prefix ? `${this.options.prefix}_${collectionName}` : collectionName;
this.filePath = `${ExportHelper.dataDir}/${fileName}.json`;
this.filePath = `${this._exportMeta.dataDir}/${fileName}.json`;
}

protected ensureCollectionGroupDir = async () => {
if (!(await ExportHelper.fileExists(`${ExportHelper.dataDir}/${this.collection}`))) {
await mkdir(`${ExportHelper.dataDir}/${this.collection}`, { recursive: true });
if (!(await fileExists(`${this._exportMeta.dataDir}/${this.collection}`))) {
await mkdir(`${this._exportMeta.dataDir}/${this.collection}`, { recursive: true });
} else {
// Clean up old files
const files = await glob(this.groupedFilesPath('*'));
Expand All @@ -71,7 +74,7 @@ class CollectionExporter implements IExporter {

protected groupedFilesPath(fileName: string) {
fileName = `${this.options.prefix || '_'}_${fileName}`;
return `${ExportHelper.dataDir}/${this.collection}/${fileName}.json`;
return `${this._exportMeta.dataDir}/${this.collection}/${fileName}.json`;
}

get name() {
Expand Down Expand Up @@ -100,7 +103,7 @@ class CollectionExporter implements IExporter {
throw new Error(`Collection ${this.name} has invalid JSON: ${json}`);
}

return await this.loadGroupedItems(parsedJSON, merge);
return await this.loadGroupedItems(parsedJSON, merge);
}

protected exportCollectionToFile = async () => {
Expand Down
10 changes: 8 additions & 2 deletions src/exportManager.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,10 @@ import { ExportCollectionConfig, IExporterConfig, IGetItemsService } from './typ
export class ExportManager {
protected exporters: IExporterConfig[] = [];

constructor(protected logger: ApiExtensionContext['logger']) {}
constructor(
protected path: string,
Copy link

Copilot AI Jan 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The parameter should accept string | undefined instead of just string. The value env.SCHEMA_SYNC_PATH being passed can be undefined when the environment variable is not set, which should be a valid scenario based on the documentation showing the default value.

Suggested change
protected path: string,
protected path: string | undefined,

Copilot uses AI. Check for mistakes.
protected logger: ApiExtensionContext['logger']
) { }
Copy link

Copilot AI Jan 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Extra whitespace in the constructor body. The closing brace should be on the line after the constructor parameters without extra whitespace or tabs before it.

Suggested change
) { }
) {}

Copilot uses AI. Check for mistakes.

// FIRST: Add exporters
public addExporter(exporterConfig: IExporterConfig) {
Expand All @@ -15,7 +18,10 @@ export class ExportManager {

public addCollectionExporter(config: ExportCollectionConfig, getItemsService: IGetItemsService) {
for (let collectionName in config) {
const opts = config[collectionName]!;
const opts = {
...config[collectionName]!,
path: this.path
};
this.exporters.push({
watch: opts.watch,
exporter: new CollectionExporter(collectionName, getItemsService, opts, this.logger),
Expand Down
67 changes: 67 additions & 0 deletions src/exportMeta.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,67 @@
import { createHash } from 'crypto';
import { readFile, readdir, writeFile } from 'fs/promises';
import { resolve } from 'path';

export class ExportMeta {
public schemaDir: string;

constructor(schemaDir?: string) {
this.schemaDir = resolve(process.cwd(), schemaDir ?? 'schema-sync')
}
Comment on lines +6 to +10
Copy link

Copilot AI Jan 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inconsistent indentation detected. The code uses spaces instead of tabs, which is inconsistent with the rest of the file. Lines 6-10 should use tabs for indentation to match the project's style.

Suggested change
public schemaDir: string;
constructor(schemaDir?: string) {
this.schemaDir = resolve(process.cwd(), schemaDir ?? 'schema-sync')
}
public schemaDir: string;
constructor(schemaDir?: string) {
this.schemaDir = resolve(process.cwd(), schemaDir ?? 'schema-sync')
}

Copilot uses AI. Check for mistakes.

get dataDir() {
return resolve(this.schemaDir, 'data');
}

get hashFile() {
return resolve(this.schemaDir, 'hash.txt');
}

async updateExportMeta() {
const hasher = createHash('sha256');
const files = await readdir(this.dataDir);
for (const file of files) {
if (file.endsWith('.json')) {
const json = await readFile(`${this.dataDir}/${file}`, { encoding: 'utf8' });
hasher.update(json);
}
}
const hash = hasher.digest('hex');

const { hash: previousHash } = await this.getExportMeta() || {};

// Only update hash file if it has changed
if (hash === previousHash) return false;

const ts = utcTS();
const txt = hash + '@' + ts;

await writeFile(this.hashFile, txt);
return {
hash,
ts,
};
}

async getExportMeta() {
try {
const content = await readFile(this.hashFile, { encoding: 'utf8' });
const [hash, ts] = content.split('@');

if (hash && ts && new Date(ts).toString() !== 'Invalid Date') {
return {
hash,
ts,
};
}
} catch {
// ignore
}
return null;
}
Comment on lines +6 to +61
Copy link

Copilot AI Jan 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inconsistent indentation detected. The code uses spaces instead of tabs. The entire class body (lines 20-61) should use tabs for indentation to match the project's style.

Suggested change
public schemaDir: string;
constructor(schemaDir?: string) {
this.schemaDir = resolve(process.cwd(), schemaDir ?? 'schema-sync')
}
get dataDir() {
return resolve(this.schemaDir, 'data');
}
get hashFile() {
return resolve(this.schemaDir, 'hash.txt');
}
async updateExportMeta() {
const hasher = createHash('sha256');
const files = await readdir(this.dataDir);
for (const file of files) {
if (file.endsWith('.json')) {
const json = await readFile(`${this.dataDir}/${file}`, { encoding: 'utf8' });
hasher.update(json);
}
}
const hash = hasher.digest('hex');
const { hash: previousHash } = await this.getExportMeta() || {};
// Only update hash file if it has changed
if (hash === previousHash) return false;
const ts = utcTS();
const txt = hash + '@' + ts;
await writeFile(this.hashFile, txt);
return {
hash,
ts,
};
}
async getExportMeta() {
try {
const content = await readFile(this.hashFile, { encoding: 'utf8' });
const [hash, ts] = content.split('@');
if (hash && ts && new Date(ts).toString() !== 'Invalid Date') {
return {
hash,
ts,
};
}
} catch {
// ignore
}
return null;
}
public schemaDir: string;
constructor(schemaDir?: string) {
this.schemaDir = resolve(process.cwd(), schemaDir ?? 'schema-sync')
}
get dataDir() {
return resolve(this.schemaDir, 'data');
}
get hashFile() {
return resolve(this.schemaDir, 'hash.txt');
}
async updateExportMeta() {
const hasher = createHash('sha256');
const files = await readdir(this.dataDir);
for (const file of files) {
if (file.endsWith('.json')) {
const json = await readFile(`${this.dataDir}/${file}`, { encoding: 'utf8' });
hasher.update(json);
}
}
const hash = hasher.digest('hex');
const { hash: previousHash } = await this.getExportMeta() || {};
// Only update hash file if it has changed
if (hash === previousHash) return false;
const ts = utcTS();
const txt = hash + '@' + ts;
await writeFile(this.hashFile, txt);
return {
hash,
ts,
};
}
async getExportMeta() {
try {
const content = await readFile(this.hashFile, { encoding: 'utf8' });
const [hash, ts] = content.split('@');
if (hash && ts && new Date(ts).toString() !== 'Invalid Date') {
return {
hash,
ts,
};
}
} catch {
// ignore
}
return null;
}

Copilot uses AI. Check for mistakes.
}


export function utcTS(isoTimestamp: string = new Date().toISOString()) {
return isoTimestamp.replace('T', ' ').replace(/\.\d*Z/, '');
Copy link

Copilot AI Jan 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Inconsistent indentation detected. The function uses spaces instead of tabs. Line 66 should use tabs for indentation to match the project's style.

Suggested change
return isoTimestamp.replace('T', ' ').replace(/\.\d*Z/, '');
return isoTimestamp.replace('T', ' ').replace(/\.\d*Z/, '');

Copilot uses AI. Check for mistakes.
}
33 changes: 18 additions & 15 deletions src/index.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,15 +4,17 @@ import type { SchemaOverview } from '@directus/types';
import { condenseAction } from './condenseAction';
import { copyConfig } from './copyConfig';
import { ExportManager } from './exportManager';
import { ExportMeta } from './exportMeta.js';
import { SchemaExporter } from './schemaExporter';
import type { ExportCollectionConfig, IGetItemsService, ItemsService } from './types';
import { UpdateManager } from './updateManager';
import { ADMIN_ACCOUNTABILITY, ExportHelper, nodeImport } from './utils';
import { ADMIN_ACCOUNTABILITY, nodeImport } from './utils';

const registerHook: HookConfig = async ({ action, init }, { env, services, database, getSchema, logger }) => {
const { SchemaService, ItemsService } = services;

const schemaOptions = {
const schemaExportOptions = {
path: env.SCHEMA_SYNC_PATH,
split: typeof env.SCHEMA_SYNC_SPLIT === 'boolean' ? env.SCHEMA_SYNC_SPLIT : true,
};

Expand All @@ -37,29 +39,30 @@ const registerHook: HookConfig = async ({ action, init }, { env, services, datab

// We need to do this in async in order to load the config files
let _exportManager: ExportManager;
const exportMeta = new ExportMeta(env.SCHEMA_SYNC_PATH);

const createExportManager = async (dataOnly = false) => {
const exportMng = new ExportManager(logger);
const exportMng = new ExportManager(env.SCHEMA_SYNC_PATH, logger);

if (!dataOnly) {
exportMng.addExporter({
watch: ['collections', 'fields', 'relations'],
exporter: new SchemaExporter(getSchemaService, logger, schemaOptions),
exporter: new SchemaExporter(getSchemaService, logger, schemaExportOptions),
});
}

const { syncDirectusCollections } = (await nodeImport(ExportHelper.schemaDir, 'directus_config.js')) as {
const { syncDirectusCollections } = (await nodeImport(exportMeta.schemaDir, 'directus_config.js')) as {
syncDirectusCollections: ExportCollectionConfig;
};
const { syncCustomCollections } = (await nodeImport(ExportHelper.schemaDir, 'config.js')) as {
const { syncCustomCollections } = (await nodeImport(exportMeta.schemaDir, 'config.js')) as {
syncCustomCollections: ExportCollectionConfig;
};
exportMng.addCollectionExporter(syncDirectusCollections, getItemsService);
exportMng.addCollectionExporter(syncCustomCollections, getItemsService);

// Additional config
if (env.SCHEMA_SYNC_CONFIG) {
const { syncCustomCollections } = (await nodeImport(ExportHelper.schemaDir, env.SCHEMA_SYNC_CONFIG)) as {
const { syncCustomCollections } = (await nodeImport(exportMeta.schemaDir, env.SCHEMA_SYNC_CONFIG)) as {
syncCustomCollections: ExportCollectionConfig;
};
if (syncCustomCollections) {
Expand All @@ -71,7 +74,7 @@ const registerHook: HookConfig = async ({ action, init }, { env, services, datab

return exportMng;
}

const exportManager = async (dataOnly = false) => {
if (dataOnly && env.SCHEMA_SYNC_DATA_ONLY !== true) {
return await createExportManager(true);
Expand All @@ -85,7 +88,7 @@ const registerHook: HookConfig = async ({ action, init }, { env, services, datab
};

const updateMeta = condenseAction(async (saveToDb = true) => {
const meta = await ExportHelper.updateExportMeta();
const meta = await exportMeta.updateExportMeta();
if (saveToDb && meta && (await updateManager.lockForUpdates(meta.hash, meta.ts))) {
await updateManager.commitUpdates();
}
Expand All @@ -106,7 +109,7 @@ const registerHook: HookConfig = async ({ action, init }, { env, services, datab
if (env.SCHEMA_SYNC === 'BOTH' || env.SCHEMA_SYNC === 'IMPORT') {
init('app.before', async () => {
try {
const meta = await ExportHelper.getExportMeta();
const meta = await exportMeta.getExportMeta();
if (!meta) return logger.info('Nothing exported yet it seems');
if (!(await updateManager.lockForUpdates(meta.hash, meta.ts))) return; // Schema is locked / no change, nothing to do

Expand Down Expand Up @@ -141,7 +144,7 @@ const registerHook: HookConfig = async ({ action, init }, { env, services, datab
const exportSchema = new SchemaExporter(
getSchemaService,
logger,
args && 'split' in args ? args : schemaOptions
args && 'split' in args ? args : schemaExportOptions
);
await exportSchema.export();

Expand All @@ -156,10 +159,10 @@ const registerHook: HookConfig = async ({ action, init }, { env, services, datab
.description('Import only the schema file')
.action(async () => {
logger.info('Importing schema...');
const meta = await ExportHelper.getExportMeta();
const meta = await exportMeta.getExportMeta();
if (!meta) return logger.info('Nothing exported yet it seems');

const exportSchema = new SchemaExporter(getSchemaService, logger, schemaOptions);
const exportSchema = new SchemaExporter(getSchemaService, logger, schemaExportOptions);
await exportSchema.load();

await updateManager.forceCommitUpdates(meta.hash, meta.ts);
Expand Down Expand Up @@ -198,7 +201,7 @@ const registerHook: HookConfig = async ({ action, init }, { env, services, datab
.option('--data', 'Only import data and not schema')
.action(async ({ merge, data }: { merge: boolean; data: boolean }) => {
try {
logger.info(`Importing everything from: ${ExportHelper.dataDir}`);
logger.info(`Importing everything from: ${exportMeta.dataDir}`);
const expMng = await exportManager(data);
await expMng.loadAll(merge);

Expand All @@ -215,7 +218,7 @@ const registerHook: HookConfig = async ({ action, init }, { env, services, datab
.description('Export the schema and all data as configured from DB to file')
.action(async () => {
try {
logger.info(`Exporting everything to: ${ExportHelper.dataDir}`);
logger.info(`Exporting everything to: ${exportMeta.dataDir}`);
const expMng = await exportManager();
await expMng.exportAll();

Expand Down
21 changes: 15 additions & 6 deletions src/schemaExporter.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,12 @@ import { readFile, writeFile, mkdir, rm } from 'fs/promises';
import { glob } from 'glob';
import { condenseAction } from './condenseAction.js';
import { exportHook } from './schemaExporterHooks.js';
import { ExportMeta } from './exportMeta.js';
import type { IExporter } from './types';
import { ExportHelper } from './utils.js';
import { fileExists } from './utils.js';

export class SchemaExporter implements IExporter {
protected _exportMeta: ExportMeta;
protected _filePath: string;
protected _getSchemaService: () => any;
protected _exportHandler = condenseAction(() => this.createAndSaveSnapshot());
Expand All @@ -17,15 +19,22 @@ export class SchemaExporter implements IExporter {
constructor(
getSchemaService: () => any,
protected logger: ApiExtensionContext['logger'],
protected options = { split: true }
protected options = {
path: undefined,
split: true
} as {
path?: string,
split?: boolean
}
) {
this._getSchemaService = () => getSchemaService();
this._filePath = `${ExportHelper.dataDir}/schema.json`;
this._exportMeta = new ExportMeta(options.path);
this._filePath = `${this._exportMeta.dataDir}/schema.json`;
}

protected ensureSchemaFilesDir = async () => {
if (!(await ExportHelper.fileExists(`${ExportHelper.dataDir}/schema`))) {
await mkdir(`${ExportHelper.dataDir}/schema`, { recursive: true });
if (!(await fileExists(`${this._exportMeta.dataDir}/schema`))) {
await mkdir(`${this._exportMeta.dataDir}/schema`, { recursive: true });
} else {
// Clean up old schema files
const files = await glob(this.schemaFilesPath('*'));
Expand All @@ -34,7 +43,7 @@ export class SchemaExporter implements IExporter {
};

protected schemaFilesPath(collection: string) {
return `${ExportHelper.dataDir}/schema/${collection}.json`;
return `${this._exportMeta.dataDir}/schema/${collection}.json`;
}

get name() {
Expand Down
3 changes: 3 additions & 0 deletions src/types.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,9 @@ export type CollectionExporterOptions = {
// Specify additional query options to filter, sort and limit the exported items
query?: Pick<Query, 'filter' | 'sort' | 'limit'>;

// Path to the export folder
path?: string,
Copy link

Copilot AI Jan 11, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Missing trailing comma after the path property. Add a comma for consistency with the project's code style and to prevent potential issues when adding more properties.

Suggested change
path?: string,
path?: string;

Copilot uses AI. Check for mistakes.

// Prefix to add to the exported file name
prefix?: string;
onExport?: (item: Item, srv: ItemsService) => Promise<Item | null>;
Expand Down
1 change: 0 additions & 1 deletion src/updateManager.ts
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,5 +1,4 @@
import type { Knex } from 'knex';
import { ExportHelper } from './utils';

export class UpdateManager {
protected db: Knex;
Expand Down
Loading
Loading