[Ansible] Updates create a backup of the root installation folder

[muued]

Hi,

I love paperless-ng and it works like a charm for me.

Using the ansible route, performing an update currently backs up the root installation folder.
For the default setup (the media folder is not set to a location outside of the root folder), this includes creating a copy of the imported documents, their processed duplicates and thumbnails.
With a relatively fresh setup, this means creating a backup folder sized 2GB.
Personally, I prefer using dedicated backup solutions for purposes like that.

I'd like to discuss ways of improving upon the current situation before creating pull requests

• make the backup mechanism optional via configuration
• from an application maintainer's perspective, what folders are considered to be worth duplicating? i would expect this to be the data directory. what about the database? no backup is created for the postgres database in case one uses that.
• what about configurations where specific parts of paperless-ng are located outside of the root directory (e.g. the data directory)? no backup is created of those directories

Best regards and thanks for an amazing project

[jonaswinkler]

@C0nsultant fyi

[anokfireball]

You are absolutely correct in criticizing the current backup approach during updates.
I myself am not really happy with it, and in my case it works because my media directory is not a subdirectory of the NG root. Exactly the edge-case you pointed out.

The idea was to have a snapshot you could immediately revert back to in case something went wrong, and I would like to keep it this way. Ideally, this would include

• gunicorn.conf.py
• .installed_version
• {{ paperlessng_data_dir }}
• scripts
• src
• {{ paperlessng_staticdir }} (?)
• paperless.conf.template (and/or /etc/paperless.conf)
• {{ paperlessng_virtualenv }}

make the backup mechanism optional via configuration

A good first approach. Though I think the backup-by-default should remain. Opt-out of security rather than opt-in.
Since you appear to largely concerned about the backup size, it might be a good idea to compress it (as it should not have too much binary data when the media directory is excluded).

from an application maintainer's perspective, what folders are considered to be worth duplicating?

See above. I don't think backups of the media directory or the database make sense outside your typical backup schedule, and in particular don't make sense along the update path. The media directory is independent of release AFAICT. The (postgres/mysql) database is versioned s.t. required migrations are automatically performed. For sqlite, input from @jonaswinkler is appreaciated.

what about configurations where specific parts of paperless-ng are located outside of the root directory (e.g. the data directory)?

• PAPERLESS_CONSUMPTION_DIR contents are ephemeral, no need to backup at all
• PAPERLESS_DATA_DIR should probably be included in a backup
• PAPERLESS_MEDIA_ROOT see above, no need to backup during update
• PAPERLESS_STATICDIR see above, should probably be included in a backup
• PAPERLESS_LOGGING_DIR no strong opionion on this one

[muued]

Thanks for the explanation and your view on this.

Regarding the database: the sqlite database resides in the data directory, so it will be backed up.
If one is using a postgresql database, it is of course not part of the backup.
As it is migrated as part of the update, it is questionable whether it will still work with the application version that could be restored from the backup. I don't know the details of The database is versioned, but I can't image the application is rolling back migration changes to get a version it is compatible to?!
However, extending the backup mechanism to a database dump seems like a bit too much.

Regarding the folders: There are some more files and folder flying around the root folder, which were not mentioned, e.g. the docker and docs folders. I'm assuming those could easily be backup up, too. However, since the docs folder is sized 19M and only contains documentation, not application logic, I guess it can safely be left out of a backup.

I will create pull request shortly, which can be the ground work for further discussion.

[anokfireball]

I can't image the application is rolling back migration changes to get a version it is compatible to

According to the Downgrading Paperless documentation it does. Based on the fact that django does revert migrations and the database is versioned through migrations (https://github.com/jonaswinkler/paperless-ng/tree/master/src/documents/migrations), I assume this to be the case. Please correct me if this is wrong, @jonaswinkler.

[anokfireball]

the docker and docs folders

Those (in theory) should not even be part of the ansible installation. I just did not remove them due to their relatively small size (and out of laziness TBH).

[jonaswinkler]

but I can't image the application is rolling back migration changes to get a version it is compatible to?!

The application does not do that automatically. You'll have to revert migrations manually in case of a downgrade.

[muued]

Still pretty amazing.

How would the process for the user look like?

1. Stop paperless
2. Restore the backup
3. Identify installed version (.installed_version contains the commit hash)
4. Resolve that to a release/tag
5. Identify corresponding migration number using the documentation
6. Migrate to the database version needed
7. Start paperless