Week 1 test result

[kernelogic]

Today I tested singularity v2 data prep on a M1 MacBook Pro running macOS Monterey 12.6.7

My golang was installed with brew but it was on version 1.18. All I had to do is run brew update && brew upgrade go and now it's on golang 1.20

1. Compile fine
2. ./singularity help fine
3. ./singularity admin reset fine, but the database is created in the current folder, rather than on a user home folder like v1 where there was a ~/.singularity
4. I tried ez-prep first, it ran OK, took 10 minutes to prep 80GB worth of cars. However it doesn't seem to store any information in database

5. Then I tried ./singularity dataset create . I got confused at first because now it splits dataset and datasource into two categories. I guess if there's some tip say like "Now you can add datasource to this dataset" after dataset is created, would be useful.
6. Then I tried ./singularity datasource add local . Again I got confused because it exits immediately (unlike ez-prep). I did not know I need to run worker manually. I guess if there's some tip say like "Now you can run worker to start generate cars", would be useful.

7. I noticed in the test plan daggen happens after run worker, my question is does the order matter? Do I have to finish generate all cars before daggen?

8. After I run the dataset-worker, the datasource processing went fine, took same amount of time as ez-prep to process 80 GB worth of cars.
9. One thing I noticed on dataset list command, it doesn't print how many cars are in there. I have to run datasource status to get that number. I think it's beneficial to list cars that active, paused, completed, error like in v1.

[xinaxu]

Thanks for taking a week to test the functionality of Singularity V2

./singularity admin reset fine, but the database is created in the current folder, rather than on a user home folder like v1 where there was a ~/.singularity

This is intentional for two reasons

1. Avoid overriding the existing database by accidentally invoking this command for testing purpose
2. Not all users have a home folder

I tried ez-prep first, it ran OK, took 10 minutes to prep 80GB worth of cars. However it doesn't seem to store any information in database

It is using a different name for database, in case it overrides anything in your default database.

 --database-file value          The database file to store the metadata. To use in memory database, use an empty string. (default: ./ezprep-<name>.db)

I got confused

I believe the new CLI command format will greatly reduce the confusion here. Please let me know otherwise.
#208 (comment)

Now you can run worker to start generate cars

+1 #224

I noticed in the test plan daggen happens after run worker, my question is does the order matter? Do I have to finish generate all cars before daggen

It does matter. daggen is like creating a snapshot of the folder structure for the dataset. You can run daggen over the course of data preparation and create multiple snapshots while the data preparation is ongoing. This would be useful for cases where data preparation will never end (i.e. keep scanning the data source for new files) while you want to create a snapshot every month for the folder structure. Hence the daggen has a higher priority over packing.

One thing I noticed on dataset list command, it doesn't print how many cars are in there. I have to run datasource status to get that number. I think it's beneficial to list cars that active, paused, completed, error like in v1.

With the new CLI design, this will likely becomes singularity data-prep list-jobs and singularity data-prep list-pieces. The first one list packing jobs and their status and the 2nd one list all piece-cids.