Fails on seeding new database

[Crowdedlight]

Hi There

Was looking at using this lib for a hobby project, but it seems like when sync a new database it fails to seed the database.

I tried to clone this repo to check if the tests ran and it was something I did wrong. However the unit-tests for sync with ECB fails. It seems to fail on the seed part of making a virtual table from the csv file. The error code from rusqlite I get is "permission denied" when it runs this line: https://github.com/sekunho/moneyman/blob/main/crates/moneyman/src/persistence/seed.rs#L136

I tried to poke about a bit, but I did not manage to figure out why it complains about permission access for the database. The files are created fine on my disk, and I can open both the sqlite database and the csv file. csv file seems to have correct format when comparing with the test-data, and the database is just empty.

Any chance you got an idea what is going on and how to fix it?

[sekunho]

Hi! 👋 Apologies for the trouble. Let me try to sort all of this out for you

Could you provide the following details:

• OS and version
• file perms/ownership of the DB file created, and the CSV file in the data directory you specified (just ls -la in data dir is ok)
• stack trace of the failed test

[Crowdedlight]

Sure!

The moneyman unit-test was tested on Windows 11. The usage of lib in my own project was tested on win11 and in a docker-bookworm container.

I tried to let it use default "home_dir" which on windows becomes /tmp/... and as that did not work I tried to just point it at the same folder as its executed from with:

    #[test]
    fn it_syncs_with_ecb() {
        let rand_str = Alphanumeric.sample_string(&mut rand::thread_rng(), 16);
        let data_dir = env::current_dir()
            .map(|home_dir| home_dir.join(".moneyman"))
            .expect("need a home directory");

        println!("{:?}", data_dir);
        //std::fs::create_dir(&data_dir).expect("failed to create test directory"); // commented out as folder gets created anyway, and if it already exists this will panic.

        ExchangeStore::sync(data_dir.clone()).unwrap();

        assert!(data_dir.join("eurofxref-hist.db3").exists());
    }

This ends up putting the .moneyman dir at: git-repos\moneyman\crates\moneyman\.moneyman
The files have the following permission: (Not sure how to print them on windows, as powershell has some really odd commands for stuff, so here is screenshot)

I can open the .db3 with DB Browser and edit it or make tables fine, and the permissions suggest my account have full permission over it.

Tested by running:

cargo test --package moneyman --lib exchange_store::tests::it_syncs_with_ecb -- --exact

console output:

running 1 test
test exchange_store::tests::it_syncs_with_ecb ... FAILED

failures:

---- exchange_store::tests::it_syncs_with_ecb stdout ----
"C:\\Users\\crow\\Documents\\Github\\moneyman\\crates\\moneyman\\.moneyman"
thread 'exchange_store::tests::it_syncs_with_ecb' panicked at crates\moneyman\src\exchange_store.rs:254:47:
called `Result::unwrap()` on an `Err` value: Seed
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace


failures:
    exchange_store::tests::it_syncs_with_ecb

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 5 filtered out; finished in 1.03s

error: test failed, to rerun pass `-p moneyman --lib`

I am developing on my windows desktop right now, but when done it will be deployed on a linux server. I can try to test it on my linux laptop to check if its a platform problem?
Although i would have assumed running it in the bookworm docker would have worked then.

Let me know what else would be useful from my side :)

[Crowdedlight]

Tested it on my linux machine and it worked fine. So it appears to be a windows problem with permissions somehow. Although not sure what goes wrong specifically, or what it is windows does differently with file permissions when running rusqlite, as the user the is running it, me, has permission to delete and edit the file -_-'

running 1 test
test exchange_store::tests::it_syncs_with_ecb ... ok

test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 5 filtered out; finished in 1.72s


moneyman/crates/moneyman on  main [!?] is 📦 v0.1.2 via 𝗥 v1.82.0 took 2s
➜ ls -al .moneyman
total 4360
drwxrwxr-x 2 fm fm    4096 Feb 10 10:04 .
drwxrwxr-x 5 fm fm    4096 Feb 10 10:02 ..
-rw-rw-r-- 1 fm fm 1811436 Feb 10 10:04 eurofxref-hist.csv
-rw-r--r-- 1 fm fm 2641920 Feb 10 10:04 eurofxref-hist.db3

moneyman/crates/moneyman on  main [!?] is 📦 v0.1.2 via 𝗥 v1.82.0

[sekunho]

Yeah I was able to run it just fine on both macOS, and Linux. I wonder if it's more on a Docker for Windows issue, and how storage is handled? Are you using volumes, or are you just keeping the data within the container? But tbh it does smell like a permission issue since it allows creation of files but disallows the rest.

If it's alright could you share the Dockerfile/compose that you used, and the exact command to execute it? It doesn't have to be the full one, I just need a rough idea of how it is. I could test it out later today, and see if I'm able to replicate the problem on Windows. Though I don't have Win11, only 10. 😓

[Crowdedlight]

I think its a general windows permission problem that is then also carried over to docker for windows. As running the unit-test with:

cargo test --package moneyman --lib exchange_store::tests::it_syncs_with_ecb -- --exact

On native windows11 also has the issue.

I am just not sure what is causing it. Is it downloading the csv file that makes it unable to be opened straight away due to some kind of permission in windows maybe? And when I then try manually its "unlocked" and works fine?

For testing my app that used moneyman as lib, I used this docker-compose and dockerfile:

services:
  bot:
    build: .
    volumes:
      - .moneyman:/app/.moneyman
    env_file:
      - ".env"

Dockerfile

# Leveraging the pre-built Docker images with
# cargo-chef and the Rust toolchain
FROM lukemathwalker/cargo-chef:latest-rust-1-bookworm AS chef
WORKDIR app

FROM chef AS planner
COPY . .
RUN cargo chef prepare --recipe-path recipe.json

FROM chef AS builder
COPY --from=planner /app/recipe.json recipe.json

# install libssl
RUN apt-get update && apt-get install -y pkg-config libssl-dev

# Build dependencies - this is the caching Docker layer!
RUN cargo chef cook --release --recipe-path recipe.json
# Build application
COPY . .
RUN cargo build --release --bin PepperBot

# We do not need the Rust toolchain to run the binary!
FROM debian:bookworm-slim AS runtime
RUN apt-get update && apt-get install -y pkg-config libssl-dev ca-certificates
WORKDIR app
COPY --from=builder /app/target/release/PepperBot /app
ENTRYPOINT ["/app/PepperBot"]

The repo for the app is also public if that helps: https://github.com/Crowdedlight/PepperBot

So mapping the targeted dir directly. It works fine on linux, but not windows.

That being said, I deployed it on linux since I found out it worked on linux and it works fine with the docker-compose/dockerfile there. So while having it work on windows would be great for development ease, I can just use my linux laptop when testing that specific feature.
So you do not have to spend time fixing it on my behalf. I got something that works on linux now :)