Filtering documents by CreatedAt date during export/import with Smuggler

[marcopornic]

Hi RavenDB team,

I have multiple collections containing thousands of documents, and I want to move data from one database to another based on a document’s CreatedAt field (for example, only documents older than 7 days).

I have tried to filter documents during export with Smuggler, but no matter what filter I apply, all documents are exported.

Here is an example of the export code I’m using (in C#):

var exportOptions = new DatabaseSmugglerExportOptions
{
    OperateOnTypes = DatabaseItemType.Documents,
    FilterScript = @"function(doc) {
        var cutoffDate = new Date(Date.now() - 7 * 24 * 60 * 60 * 1000);
        return doc.CreatedAt && new Date(doc.CreatedAt) < cutoffDate;
    }"
};

However, this filter is ignored and all documents are exported.

Is there a recommended way to filter documents during export or import based on a field value?

Is filtering on import with a transform script the best way, or is there a better approach?

Thanks in advance for your help!

[marcopornic]

I eventually found a working solution by using a transform script in the export options instead of a filter script. The transform script allows me to skip documents that do not match my criteria, and it works as expected.

Here's the working code:

var exportOptions = new DatabaseSmugglerExportOptions
{
    Collections = new List<string> { collectionName },
    OperateOnTypes = DatabaseItemType.Documents,
    TransformScript = $@"
        if (this.CreatedAt && this.CreatedAt < '{cutoffIso}')
            return this;
        throw 'skip';"
};

Using this approach, only documents older than the specified date are exported, and the rest are ignored. I can confirm this works well when streaming directly between two databases using:

await store.Smuggler.ExportAsync(exportOptions, destinationStore.Smuggler);

Hopefully this helps others looking to filter documents during export.

Best regards,

[ppekrol]

Hi, your solution with TransformScript is correct.
I'm wondering from where you took the FilterScript approach? There is no such property in the DatabaseSmugglerExportOptions.

[marcopornic]

Hi, you're absolutely right. Thanks for pointing that out.

I double-checked, and indeed I was using TransformScript all along... but I mistakenly referred to it as FilterScript in my original message. At first, I was experimenting with different forms inside the TransformScript, like using the doc() function:

function(doc) {
...
}

Thanks for your feedback 👍