Ability to choose workdir for temporary storage

[raybooysen]

The Archive Warrior running writes gigabytes of data before uploading. I would love a way to specify via env variable the location of the temporary storage to a file system or storage device that I'd prefer

[TheTechRobo]

Assuming you're using docker, you can use its -v parameter to bridge the current workdir to wherever you want on the host system.

[raybooysen]

I've been trying with some variants. /home/warrior, /home/warrior/data, /home/warrior/projects. All cause various amounts of errors.

Unsure which I should be using

[viniciushsantana]

I've been considering this issue as well. It seems that the working directories /home/warrior/projects and /home/warrior/data are being utilized in a manner that prevents exposing them as volumes, due to the absence of certain necessary files.

For example, within /home/warrior/data, the binaries wget-at and wget-at-gnutls should be present:

warrior-dockerfile/Dockerfile

Lines 16 to 17 in 86f9433

	&& ln -fs /usr/local/bin/wget-lua /home/warrior/data/wget-at
	COPY --from=atdr.meo.ws/archiveteam/grab-base:gnutls /usr/local/bin/wget-lua /home/warrior/data/wget-at-gnutls

I've discovered a potential workaround for this issue by altering Docker's data-root setting. However, this approach is not ideal and could lead to other complications.

Properly managing working directories and enabling the exposure of volumes would also facilitate the use of tmpfs mounts. This is particularly beneficial for users with ample RAM available who wish to conserve some SSD IOPS.

[raybooysen]

This was my primary usecase. My warriors run for long periods on machines with spare RAM, a tmpfs is a good use case here so avoid the SSD completely for temporal data.

[budde96]

I'm using Podman, you can mount /home/warrior/projects and /home/warrior/data/projects fine and that seems to cover the files you want for persistence and all the big temp files from what I can tell with my tests.

[ericblade]

Agree with others, attempting to use a volume for /home/warrior/data obscures the files that the image puts in there. attempting to use a volume for /home/warrior/projects breaks download/installation of a project with no obvious error, just "installation failed". Attempting to use a volume for a project itself results in a git error about not wanting to traverse across file system boundaries.

I keep my docker data on a "small" 2tb disk, and all my cold storage is elsewhere, so I'd need to restructure how this works to be able to stick a volume in here. Might have to do work on the software itself to get it to be compatible.

now i'm trying /home/warrior/data/projects .. that might work, at least it appears to be more successful than the others, in that it's trying to install the project. Still not being able to persist much of the work directory is not fun.

... that does seem to work after waiting many minutes for it to startup. all the other options i tried failed. at least it gets some of the data off my working disk

[eszpee]

Mapping /home/warrior/data/projects works with docker, thanks ericblade. I used this setup in docker compose:

    volumes:
        - /storage/archiveteam-warrior:/home/warrior/data/projects

[flotwig]

I too had this issue, the warrior has written many terabytes to my SSDs over the years, unnecessarily. So I forked warrior-dockerfile and updated it so that both data and projects can be stored on a tmpfs mount (or any directory of your choosing). It copies data and projects to the chosen run dir at startup, so no files are missing: https://github.com/flotwig/warrior-dockerfile#run-using-tmpfs

Example docker-compose.yml to run with data and projects in tmpfs:

services:
  warrior:
    image: ghcr.io/flotwig/warrior-dockerfile:master
    ports:
      - "8001:8001"
    environment:
      WARRIOR_RUN_DIR: /warrior-tmp
    tmpfs:
      - /warrior-tmp:exec,mode=777,uid=1000,gid=1000

Seems to work well for all current projects available via the warrior.

[jonboy345]

Just in case anyone cares, I figured out how to map the project specific containers to store their downloaded data to a tmp/high endurance disk using Docker Compose:

#SampleProjectGrab
  SampleProject-Grab01:
    image: atdr.meo.ws/archiveteam/SampleProject-grab:latest
    container_name: SampleProject-Grab01
    environment:
      - TZ=America/New_York
    volumes:
      - /path/to/tmp/storage:/grab/data
    restart: unless-stopped
    command: --concurrent numconcurrentjobshere downloadernamehere

Using this in conjunction with the below path mapping to run ATWarrior Docker set to auto, along side some additional containers for long running projects. Warrior works on high priority tasks, while the rest of them churn away on long term projects.

volumes:
      - /path/to/tmp/storage/ATWarrior01:/home/warrior/data/projects

Using TrueNAS Scale Host fwiw.