[BUG] Document not found after update

[gruessung]

Describe the bug
I have updates paperless-ng to 1.4.2. I cannot open any old document since this update. paperless always throw a HTTP 404.
fyi: on my storage folder, there are all documents :)
fyi 2: new documents works great.

To Reproduce
Steps to reproduce the behavior:

1. Upgrade to 1.4.2
2. Open some document which exists before the update

Screenshots

Webserver logs

[2021-05-10 07:18:42,520] [WARNING] [django.request] Not Found: /api/documents/177/preview/
[2021-05-10 07:18:46,981] [WARNING] [django.request] Not Found: /api/documents/177/download/

Relevant information

• Ubuntu Server with docker installation
• browser: Firefox / Chrome
• docker-compose.yml: https://paste.solardorf.eu/lewixuboqa

[jonaswinkler]

• What version did you upgrade from?
• What does your media/documents/archive folder look like?
• What does http://paperless/api/documents/177/metadata/ report?, especially interested in "has_archive_version".

[gruessung]

1. unfortunately I have not remembered
2. the screenshot above shows my "originals" folder. these files are also in "archive" and "thumbnails"

GET /api/documents/177/metadata/

HTTP 200 OK
Allow: GET, HEAD, OPTIONS
Content-Type: application/json
Vary: Accept

{
    "original_checksum": "7f287a8866357c5fd29f0bd4ce9beb79",
    "original_size": null,
    "original_mime_type": "application/pdf",
    "media_filename": "0000177.pdf",
    "has_archive_version": true,
    "original_metadata": null,
    "archive_checksum": "e32d66224bb60e1dec53344c1716bab5",
    "archive_media_filename": "0000177.pdf",
    "archive_size": null,
    "archive_metadata": null
}

[gruessung]

FYI: file permissions are the same (documents before and after upgrade)
FYI 2: i'm able to open the files directly in my filemanager, so the files aren't corrupted.

[gruessung]

do you have any idea? @jonaswinkler

[jonaswinkler]

So far, none. I've got all of the critical migrations tested as well. I'll try some migrations from various versions later today, maybe I'll find something.

[jonaswinkler]

@gruessung I've tested an upgrade from a version as early as 0.9.5 to the current versions, and documents are still in place after the upgrade and accessible with the frontend.

The metadata endpoint does not report a file size, which it only does if it either can't access the file or the file does not exists at the expected location.

Please do the following while paperless is running.

1. In your paperless-ng folder with the docker config files: docker-compose exec webserver python3 manage.py shell - A python prompt >>> should appear.
2. from documents.models import Document + Enter. No output should appear.
3. Document.objects.get(id=177).source_path + Enter. A filename should appear. This is the location where paperless currently expects the file.
4. Exit with CTRL+D or exit().
5. Verify that the path is correct.

If it appears correct, something might not be correct with your docker volumes. If it's wrong, we'll have to investigate further.

[gruessung]

The path seems to be correct:
/usr/src/paperless/src/../media/documents/originals/0000177.pdf

docker-compose.yml:
/home/***/docker/paperless-ng/storage/media:/usr/src/paperless/media

0000177.pdf physical location:
/home/***/docker/paperless-ng/storage/media/documents/originals

Which chmod requires paperless-ng or is there any command to fix the file permissions?
I tried chmod 777 on 0000177.pdf and this does not working

[jonaswinkler]

Please also do docker-compose exec webserver bash and verify the location inside the container.

[gruessung]

Yes, there is any fault:
root@7809002e6c9d:/usr/src/paperless/media/documents/originals# ls 0000184.pdf 0000185.pdf 0000186.pdf 0000187.pdf
Do you have any idea to fix this? Outside from docker all documents has the same chmod and chown.

[jonaswinkler]

These are documents that were not part of your initial screenshot, so I highly suspect that your volumes are not getting mounted properly. I dont think this has anything to do with the actual update of paperless.

[davosian]

Same issue here on v1.4.0: new documents get uploaded fine including thumbnails and actual downloads. Existing documents are missing thumbnails and the files to download are not found (same error message).

I upgraded to 1.4.2 which did not resolve the issue. However, it looks like the problem is some place else and not directly related to 1.4.2.

[jonaswinkler]

• If thumbnails are missing as well, then this is not related to any update. There has never been any change to how thumbnails are stored.
• Thumbnails only get deleted when you delete the associated document in the UI / through the API. The same applies to documents.

Check your docker volume configuration, you can do the same diagnostics as outlined above.

There's also the sanity checker (see documentation) that reports missing and corrupted files.

[davosian]

I found my particular issue: the CIFS volume hosting all documents was not mounted anymore, so definitively not a paperless-ng issue.

[nalybuites]

I am experiencing this (or similar) issue. I recently added a watchtower docker container to my home server. It upgraded Paperless-ng from 1.4.5 to 1.5.0. And now none of my documents exist. My suspicion is that the database got truncated/nullified:

• Superuser no longer existed, had to recreate
• Documents are not in paperless-ng, but do exist in the right locations in the volumes (documents{archive,originals,thumbnails})
• All volumes are CIFS shares on my NAS, all are still properly mounted.
• Looking in the admin panel, there are no documents in the database. Additionally, all of my correspondent and document type records are gone.

[nalybuites]

Ah I think I see the problem in my docker-compose: I've swapped the container and host when specifying the volume for the database data. So upgrading the image does indeed blow away the database... 🤦