Work around the 250Mb import limit

[doobes]

Recently, I made an export of my Notebook. All good. Unzipping it, it looks just fine.

In doing a bunch of futzing with my homelab getting Trilium to run under Docker Swarm, I damaged my database.

Restoring a recent database backup, most of the images have been lost. This is a major issue. Trying the other back up databases, they are the same. That's problematic.

Dropping back, to importing the Notebook I exported. The issue: It's ~500Gb and can't be imported.

Seems nonsensical to be able to export a notebook that can't be imported.

Is there a way I can restore my Notebook?

Thanks

Chris

[eliandoran]

Generally it's best to restore from a backup of the SQLite database than with a larger import. Do you have one to restore from?

Also, is the ZIP 500 gigabytes or megabytes? I see you mentioned both units in separate threads. 500 MB sounds feasible, might only require increasing the RAM available to the application.

[doobes]

It's 500 Mb.

The backup databases lost a goodly number of the embedded images. Thus, the desire to do the zip archive.

I need to test this, but it seems like a bug to me.

[perfectra1n]

You can modify the maximum size of a file allowed to be uploaded by setting either of the two environment variables in the docs:

https://docs.triliumnotes.org/user-guide/setup/server#disabling-modifying-the-upload-limit

MAX_ALLOWED_FILE_SIZE_MB
TRILIUM_NO_UPLOAD_LIMIT

[eliandoran]

@doobes ,

Trilium is not disallowing the import, which is ~502Mb, but it's not importing the file.

What happens, more specifically? Are there any errors in the logs?

[doobes]

Nothing happens. I "import" the file, the dialog window closes with no action.

Nothing untoward in the logs.

15:59:33.044 204 POST /api/sync/finished with 0 bytes took 2ms
15:59:33.105 Content hash computation took 23ms
15:59:33.107 Slow 304 GET /api/sync/check with 13750 bytes took 26ms
15:59:34.968 304 GET /api/setup/status with 59 bytes took 3ms
15:59:35.009 200 POST /api/login/sync with 56 bytes took 6ms
15:59:35.050 200 GET /api/sync/changed?instanceId=AC7o7opXNXBs&lastEntityChangeId=148011&logMarkerId=P9Lbyikadr with 73 bytes took 3ms
15:59:35.090 204 POST /api/sync/finished with 0 bytes took 3ms
15:59:35.163 Content hash computation took 35ms
15:59:35.165 Slow 304 GET /api/sync/check with 13750 bytes took 38ms
15:59:54.645 200 GET /api/health-check with 15 bytes took 1ms
16:00:22.318 CSRF token generation: Successful
16:00:22.758 200 GET /api/tree with 7879 bytes took 2ms
16:00:22.768 304 GET /api/keyboard-actions with 21860 bytes took 6ms
16:00:22.772 200 GET /api/options with 9119 bytes took 1ms
16:00:22.777 304 GET /api/autocomplete/notesCount with 3 bytes took 2ms
16:00:22.854 304 GET /api/options/locales with 1113 bytes took 0ms
16:00:23.061 304 GET /api/script/widgets with 2687 bytes took 4ms
16:00:23.103 200 POST /api/tree/load with 4350 bytes took 2ms
16:00:23.267 304 GET /api/keyboard-shortcuts-for-notes with 2 bytes took 1ms
16:00:23.272 200 POST /api/tree/load with 6370 bytes took 2ms
16:00:23.276 304 GET /api/system-checks with 27 bytes took 1ms
16:00:23.281 304 GET /api/sql/schema with 7987 bytes took 3ms
16:00:23.307 304 GET /api/notes/root/blob with 155 bytes took 2ms
16:00:23.372 200 POST /api/tree/load with 4323 bytes took 1ms
16:00:23.382 304 GET /api/recent-changes/undefined with 2 bytes took 5ms
16:00:23.391 304 GET /api/search/note.parents.noteId%3D%22root%22%20%23!archived with 41 bytes took 6ms
16:00:23.395 304 GET /api/notes/root/attachments with 2 bytes took 2ms
16:00:23.401 200 POST /api/tree/load with 21973 bytes took 1ms
16:00:23.406 304 GET /api/note-map/root/backlink-count with 11 bytes took 1ms
16:00:23.434 304 GET /api/search/%23textSnippet with 2 bytes took 2ms
16:00:23.437 304 GET /api/search/%23workspace%20%23!template with 2 bytes took 1ms
16:00:23.443 304 GET /api/autocomplete?query=&activeNoteId=root&fastSearch=true with 1492 bytes took 3ms
16:00:23.453 200 POST /api/tree/load with 14357 bytes took 1ms
16:00:23.533 304 GET /api/notes/sxi3rcausQ7d/blob with 155 bytes took 2ms
16:00:23.537 304 GET /api/notes/jyZwoOlKgs8C/blob with 2493 bytes took 2ms
16:00:23.630 200 POST /api/tree/load with 9400 bytes took 1ms
16:00:23.678 304 GET /api/notes/root with 315 bytes took 1ms
16:00:23.749 304 GET /api/notes/_template_text_snippet with 341 bytes took 0ms
16:00:23.794 304 GET /api/notes/_template_list_view with 326 bytes took 1ms
16:00:23.835 304 GET /api/notes/_template_grid_view with 326 bytes took 0ms
16:00:24.066 JS Info: CKEditor state changed to ready
16:00:24.084 304 GET /api/notes/_template_calendar with 324 bytes took 0ms
16:00:24.129 304 GET /api/notes/_template_table with 318 bytes took 1ms
16:00:24.171 304 GET /api/notes/_template_geo_map with 322 bytes took 1ms
16:00:24.214 304 GET /api/notes/_template_board with 325 bytes took 1ms
16:00:24.256 304 GET /api/notes/_template_presentation with 332 bytes took 1ms
16:00:24.301 304 GET /api/search-templates with 16 bytes took 2ms
16:00:24.342 200 POST /api/tree/load with 5251 bytes took 1ms
16:00:24.384 304 GET /api/notes/aMT5SVr3ow6m with 332 bytes took 0ms
16:00:24.846 200 GET /api/health-check with 15 bytes took 1ms
16:00:25.273 304 GET /api/script/startup with 6468 bytes took 3ms
16:00:25.316 200 POST /api/tree/load with 4568 bytes took 0ms
16:00:25.370 200 POST /api/script/exec with 74 bytes took 5ms
16:00:27.926 304 GET /api/search-templates with 16 bytes took 2ms
16:00:32.964 304 GET /api/setup/status with 59 bytes took 4ms
16:00:33.006 200 POST /api/login/sync with 56 bytes took 6ms
16:00:33.051 200 GET /api/sync/changed?instanceId=lctTy2Gma8KX&lastEntityChangeId=148011&logMarkerId=GlVicp4c9D with 73 bytes took 2ms
16:00:33.092 204 POST /api/sync/finished with 0 bytes took 3ms
16:00:33.171 Content hash computation took 40ms
16:00:33.172 Slow 304 GET /api/sync/check with 13750 bytes took 42ms
16:00:34.965 304 GET /api/setup/status with 59 bytes took 3ms
16:00:35.005 200 POST /api/login/sync with 56 bytes took 6ms
16:00:35.046 200 GET /api/sync/changed?instanceId=AC7o7opXNXBs&lastEntityChangeId=148011&logMarkerId=77hM5OK6gH with 73 bytes took 2ms
16:00:35.088 204 POST /api/sync/finished with 0 bytes took 2ms
16:00:35.161 Content hash computation took 34ms
16:00:35.162 Slow 304 GET /api/sync/check with 13750 bytes took 36ms
16:00:55.042 200 GET /api/health-check with 15 bytes took 0ms
16:01:25.238 200 GET /api/health-check with 15 bytes took 1ms

Cheers,

chris

[perfectra1n]

Yep, that's exactly what I use and I can validate that it works:

        controllers:
          main:
            containers:
              trilium:
                image:
                  repository: ghcr.io/triliumnext/trilium
                  tag: v0.100.0
                  pullPolicy: IfNotPresent
                env:
                  TRILIUM_NO_UPLOAD_LIMIT: "true"

So at least it doesn't seem like the limit is blocking you anymore - but upon file upload/"import", the server attempts to unzip and decompress the file. Then it'll start sending out logs about clients syncing due to the new files uploaded and saved, which yours doesn't seem to be doing.

I upload >2GB files regularly for testing, so I'm not quite sure exactly what might be going wrong in your specific environment...

[doobes]

How is this setting done in the config.ini file?

I've got this, but I'm still getting the 250Mb error:

[Export]
TRILIUM_NO_UPLOAD_LIMIT=true

Thanks

[perfectra1n]

It is not done in the config.ini file, and instead must be done using environment variables:
https://docs.triliumnotes.org/user-guide/setup/server#disabling-modifying-the-upload-limit

[doobes]

Environment values for the local desktop machine, or the server?

I've got it set for the server, but the browser version of Trilium does nothing when importing my file.

The desktop version of Trilium, on the same machine, spits back the 250 MiB limit.

I've read the docs. My comprehension of them is limited as I'm not a dev.

Just trying to recover my notebook.

[perfectra1n]

Trilium runs as a Node process (whether that's in something like a Docker container, or if you run it "directly" via invoking trilium.sh).

That Node process reads the environment variable(s) that are available within the current shell that it's being executed from. As you mentioned above in your Docker Compose file, you're correctly setting:

environment:
  ...
  - TRILIUM_NO_UPLOAD_LIMIT=true
  - MAX_ALLOWED_FILE_SIZE_MB=750

Which is what is needed to increase the file upload limit to 750MB (or just straight up remove the limit, I'm not sure which one takes precedence right now).

If you're trying to import the zip within the Desktop application of Trilium, as that runs its own local "server" (and then syncs with your upstream remote server), you'd need to also set the environment variable for your desktop process.

My suggestion would be (to decrease the layers of obfuscation here) to use a web browser such as Chrome to directly navigate to your remote server (the IP address of the Trilium container running in your Docker Compose stack) and then try to import it via the UI from within your browser. You then wouldn't need to change the maximum upload file limit of your locally running Trilium "instance", as the browser would be "talking" directly to the remote server that has the correct environment variable set.

[doobes]

That's what I've been trying to do.

I've got Trilum reverse proxied via Traefik/cloudflare so I access the browser version at https://trilium.mydomain.com.

The environment values are set as you describe.

When I go to import the zip file, nothing happens.

Thanks

[perfectra1n]

Reverse proxies such as Traefik (and Cloudflare) often have a maximum file size that they handle, so the upload request likely isn't even making it to Trilium. As an example, Cloudflare has a maximum upload limit of 100MB.

You can right click on a webpage to "inspect", then navigate to the "network" tab, then attempt the upload. You'll likely see a request failing after you attempt an upload.

This is outside the scope directly of Trilium though, so you may get better support reaching out on the Cloudflare/Traefik forums.

My suggestion would be to try to remove any/all layers between you and your Trilium server, then to attempt the upload.

[doobes]

If I install Trilium in a linux box that has never had Trilium installed, might I be able to import the file? How do environment values get set on stand alone apps (in Linux)?

Sorry to be a pest.

Chris

[perfectra1n]

There are so many different flavors of terminals, installations, and distros - I'd honestly suggest asking Gemini or ChatGPT on how to do so, using whichever desktop environment, distro, and Trilium installation method you decide on.

If you do get that working, I'd suggest just setting up the "new" installation on that Linux box as an additional "client" of your remote server (select "I already have a remote server..." or whatever it's called). Then with the Trilium process running with the environment variable set, and your "new" box working, I'd try the import.

[doobes]

Ok. I created a docker stack for direct access and the import was successful.

Thank you