[BUG] Upload odt: Error while converting document to PDF #536

telsch · 2021-02-13T14:37:52Z

telsch
Feb 13, 2021

Describe the bug
Uploading an odt Document raising an exception.

To Reproduce

Go to Dashboard
Click on Browse files
Watch docker logs -f paperless_webserver_1
See the exception

Expected behavior
Converting the odt document to PDF.

Webserver logs

15:19:23 [Q] INFO Process-1:1 processing [test.odt]
[2021-02-13 15:19:23,189] [INFO] [paperless.consumer] Consuming test.odt
[2021-02-13 15:19:23,195] [INFO] [paperless.parsing.tika] Sending /usr/src/paperless/src/../consume/test.odt to Tika server
[2021-02-13 15:19:23,247] [INFO] [paperless.parsing.tika] Converting /usr/src/paperless/src/../consume/test.odt to PDF as /tmp/paperless/paperless-30l71j8v/convert.pdf
[2021-02-13 15:19:33,643] [ERROR] [paperless.consumer] Error while consuming document test.odt: Error while converting document to PDF: 504 Server Error: Gateway Timeout for url: http://gotenberg:3000/convert/office
15:19:33 [Q] ERROR Failed [test.odt] - test.odt: Error while consuming document test.odt: Error while converting document to PDF: 504 Server Error: Gateway Timeout for url: http://gotenberg:3000/convert/office : Traceback (most recent call last):
  File "/usr/src/paperless/src/paperless_tika/parsers.py", line 79, in convert_to_pdf
    response.raise_for_status()  # ensure we notice bad responses
  File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 943, in raise_for_status
    raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Timeout for url: http://gotenberg:3000/convert/office

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/asgiref/sync.py", line 208, in main_wrap
    raise exc_info[1]
  File "/usr/src/paperless/src/documents/consumer.py", line 238, in try_consume_file
    document_parser.parse(self.path, mime_type, self.filename)
  File "/usr/src/paperless/src/paperless_tika/parsers.py", line 65, in parse
    self.archive_path = self.convert_to_pdf(document_path, file_name)
  File "/usr/src/paperless/src/paperless_tika/parsers.py", line 82, in convert_to_pdf
    f"Error while converting document to PDF: {err}"
documents.parsers.ParseError: Error while converting document to PDF: 504 Server Error: Gateway Timeout for url: http://gotenberg:3000/convert/office

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.7/site-packages/django_q/cluster.py", line 436, in worker
    res = f(*task["args"], **task["kwargs"])
  File "/usr/src/paperless/src/documents/tasks.py", line 82, in consume_file
    task_id=task_id
  File "/usr/src/paperless/src/documents/consumer.py", line 258, in try_consume_file
    f"Error while consuming document {self.filename}: {e}"
  File "/usr/src/paperless/src/documents/consumer.py", line 70, in _fail
    raise ConsumerError(f"{self.filename}: {log_message or message}")
documents.consumer.ConsumerError: test.odt: Error while consuming document test.odt: Error while converting document to PDF: 504 Server Error: Gateway Timeout for url: http://gotenberg:3000/convert/office

Relevant information

Host OS of the machine running paperless: Archlinux
Browser firefox
Version 1.1.1
Installation method: docker

Workaround

docker exec -it paperless_webserver_1 /bin/bash
sed -i 's/response.raise_for_status/#response.raise_for_status/' /usr/src/paperless/src/paperless_tika/parsers.py

15:21:19 [Q] INFO Process-1:2 processing [test.odt]
[2021-02-13 15:21:19,208] [INFO] [paperless.consumer] Consuming test.odt
[2021-02-13 15:21:19,216] [INFO] [paperless.parsing.tika] Sending /usr/src/paperless/src/../consume/test.odt to Tika server
[2021-02-13 15:21:19,263] [INFO] [paperless.parsing.tika] Converting /usr/src/paperless/src/../consume/test.odt to PDF as /tmp/paperless/paperless-yv6g02gj/convert.pdf
/usr/local/lib/python3.7/site-packages/django/db/models/fields/__init__.py:1370: RuntimeWarning: DateTimeField Document.created received a naive datetime (2021-02-13 14:52:50.218143) while time zone support is active.
  RuntimeWarning)
[2021-02-13 15:21:30,240] [INFO] [paperless.consumer] Document 2021-02-13 test consumption finished
15:21:30 [Q] INFO Processed [test.odt]

Answered by telsch

Feb 13, 2021

Yep this resolved my issue:
DEFAULT_WAIT_TIMEOUT: 30 // cannot be more than 30 seconds

https://thecodingmachine.github.io/gotenberg/#environment_variables.default_wait_timeout

View full answer

jonaswinkler · 2021-02-13T14:47:44Z

jonaswinkler
Feb 13, 2021
Maintainer

This error indicates that paperless cannot reach the gotenberg server. This server is responsible for converting office documents into PDF documents. Without doing that, the front end cannot display these documents, which is a major drawback.

I'd verify that this service is actually running (docker container ls | grep gotenberg) and if not, start it. In case you're using the docker-compose files (you didn't exactly specify that), docker-compose down and up will restart your instance and gotenberg along with it.

Your workaround prevents paperless from aborting consumption in case PDF conversion fails. If my logic is correct, you should not be able to view PDF files for "test.odt" in paperless.

0 replies

telsch · 2021-02-13T15:30:50Z

telsch
Feb 13, 2021
Author

The gotenberg server is up and running. With comment out response.raise_for_status, the PDF conversion succeeded and i can view it in paperless.

I tried before to reach it inside the container and got a response:

curl http://gotenberg:3000/convert/office
{"message":"Method Not Allowed"}

1 reply

jonaswinkler Feb 13, 2021
Maintainer

Oh well, seems I was a bit too quick in deciding that this must be a network issue.

Gotenberg actually has a timeout of 10 seconds by default, and if the conversion does not finish within that timeout, it will report 504 errors.

Is this a particularly large file?

The gotenberg documentation also says that you can adjust this timeout with an environment variable. With docker-compose, doing that requires the following edits in the compose file:

  gotenberg:
    image: thecodingmachine/gotenberg
    restart: unless-stopped
    environment:
      DISABLE_GOOGLE_CHROME: 1
      DEFAULT_WAIT_TIMEOUT: "60" //your timeout in seconds

Not sure if the quotes are required.

telsch · 2021-02-13T16:16:48Z

telsch
Feb 13, 2021
Author

Yep this resolved my issue:
DEFAULT_WAIT_TIMEOUT: 30 // cannot be more than 30 seconds

https://thecodingmachine.github.io/gotenberg/#environment_variables.default_wait_timeout

1 reply

jonaswinkler Feb 13, 2021
Maintainer

I'll add a note to the trouble shooting section, since this works fine for most users with the default settings.

Spoker · 2021-03-03T21:59:18Z

Spoker
Mar 3, 2021

I have the same problem with this timeout. Unfortunately the max is 30 and I have not been able to process a single office document, even very small, single page files.

Error webserver:

formulier_tb2021_volledig.doc: Error while consuming document formulier_tb2021_volledig.doc: Error while converting document to PDF: 504 Server Error: Gateway Timeout for url: http://gotenberg:3000/convert/office : Traceback (most recent call last):
File "/usr/src/paperless/src/paperless_tika/parsers.py", line 79, in convert_to_pdf
response.raise_for_status() # ensure we notice bad responses
File "/usr/local/lib/python3.7/site-packages/requests/models.py", line 943, in raise_for_status
raise HTTPError(http_error_msg, response=self)
requests.exceptions.HTTPError: 504 Server Error: Gateway Timeout for url: http://gotenberg:3000/convert/office

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/asgiref/sync.py", line 208, in main_wrap
raise exc_info[1]
File "/usr/src/paperless/src/documents/consumer.py", line 238, in try_consume_file
document_parser.parse(self.path, mime_type, self.filename)
File "/usr/src/paperless/src/paperless_tika/parsers.py", line 65, in parse
self.archive_path = self.convert_to_pdf(document_path, file_name)
File "/usr/src/paperless/src/paperless_tika/parsers.py", line 82, in convert_to_pdf
f"Error while converting document to PDF: {err}"
documents.parsers.ParseError: Error while converting document to PDF: 504 Server Error: Gateway Timeout for url: http://gotenberg:3000/convert/office

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/django_q/cluster.py", line 436, in worker
res = f(*task["args"], **task["kwargs"])
File "/usr/src/paperless/src/documents/tasks.py", line 81, in consume_file
task_id=task_id
File "/usr/src/paperless/src/documents/consumer.py", line 258, in try_consume_file
f"Error while consuming document {self.filename}: {e}"
File "/usr/src/paperless/src/documents/consumer.py", line 70, in _fail
raise ConsumerError(f"{self.filename}: {log_message or message}")
documents.consumer.ConsumerError: formulier_tb2021_volledig.doc: Error while consuming document formulier_tb2021_volledig.doc: Error while converting document to PDF: 504 Server Error: Gateway Timeout for url: http://gotenberg:3000/convert/office

Output gotenberg:
{"bytes_in":"57510","bytes_out":"36","host":"gotenberg:3000","latency":"30005439534","latency_human":"30.005439534s","level":"error","method":"POST","msg":"request failed","op":"context.Context.LogRequestResult","path":"/convert/office","referer":"","remote_ip":"192.168.32.6","status":504,"time":"2021-03-03T21:45:48Z","trace":"PJdDQ1DtbpRzOKozbw7EN51RUfnmbt2P","uri":"/convert/office","user_agent":"python-requests/2.25.1"}

Any other ideas?
Running on Synology DS918+ with 16 GB Ram.

PS: @jonaswinkler you're doing a great job! Love paperless-ng!

1 reply

jonaswinkler Mar 3, 2021
Maintainer

I really don't know. If the gotenberg server does not respond, there's not much I can do. Try restarting the container maybe?

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[BUG] Upload odt: Error while converting document to PDF #536

Uh oh!

{{title}}

Uh oh!

Replies: 4 comments 3 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

[BUG] Upload odt: Error while converting document to PDF #536

Uh oh!

telsch Feb 13, 2021

Replies: 4 comments · 3 replies

Uh oh!

jonaswinkler Feb 13, 2021 Maintainer

Uh oh!

telsch Feb 13, 2021 Author

Uh oh!

jonaswinkler Feb 13, 2021 Maintainer

Uh oh!

telsch Feb 13, 2021 Author

Uh oh!

jonaswinkler Feb 13, 2021 Maintainer

Uh oh!

Spoker Mar 3, 2021

Uh oh!

jonaswinkler Mar 3, 2021 Maintainer

telsch
Feb 13, 2021

Replies: 4 comments 3 replies

jonaswinkler
Feb 13, 2021
Maintainer

telsch
Feb 13, 2021
Author

jonaswinkler Feb 13, 2021
Maintainer

telsch
Feb 13, 2021
Author

jonaswinkler Feb 13, 2021
Maintainer

Spoker
Mar 3, 2021

jonaswinkler Mar 3, 2021
Maintainer