Import DP2 release candidate 2 (v30_0_0_rc2)

[nevencaplar]

Import data processing for the release candidate 2 (rc2) for DP2. It should be based on v30.0.0. - the JIRA ticket is at https://rubinobs.atlassian.net/browse/DM-53697

Use the DASH pipeline to import, same procedure as with #1158.

[nevencaplar]

I see that @gitosaurus suggested srun --pty --exclusive --nodes=1 --time=48:00:00 \ --partition=milano --account=rubin:commissioning bash. @smcguire-cmu Do you have rubin:commissioning account? I think you do need it because:

--account rubin:default That’s the preemptible queue. If your goal is to use all the cores that no one else is using, you can use as many cores on rubin:default as you want, and when another job with rubin:commissioning comes along it’ll replace yours automatically. HTCondor will reschedule it somewhere else.

[smcguire-cmu]

I think I have permission, I can run that command and get an allocation so it seems to be working

[smcguire-cmu]

Following on from Derek's nice documentation of running the pipeline, the settings used for this one were:

export REPO=dp2_prep
export INSTRUMENT=LSSTCam
export RUN=20250515-20251214
export VERSION=v30_0_0_rc2
export COLLECTION=DM-53697
export OUTPUT_DIR=/sdf/data/rubin/shared/lsdb_commissioning

Initially this was run with only the default columns, and it took this amount of time for them:
./01-Butler.ipynb 00:01:03 (HH:MM:SS)
./02-Raw_file_sizes.ipynb 00:01:10 (HH:MM:SS)
./03-Import.ipynb 05:06:15 (HH:MM:SS)
./04-Post_processing.ipynb 00:55:48 (HH:MM:SS)
./05-Nesting.ipynb 03:29:17 (HH:MM:SS)
./06.a-Basic_Statistics.ipynb 00:01:24 (HH:MM:SS)
./06.b-ByField.ipynb 00:03:47 (HH:MM:SS)
./07-Generate_collections.ipynb 01:22:17 (HH:MM:SS)

The crossmatch with PS1 failed due to an LSDB error that still needs to be fixed.

After fixing it to run with all columns, I had to run the notebooks one by one, and some of them interactively, so I don't have timing information.

I do have a log file with memory usage over the course of the import, free was logging how much memory was being used every 5 seconds. This has resulted in a huge file that isn't particularly helpful right now but I will try to make some graphs and make sense of it later.

[smcguire-cmu]

I can get timing info from the notebook outputs though, and from that I can say that with all columns, the 03-import notebook, which is the longest, took 5h43m compared to 05:06:15 (HH:MM:SS) with just default

[nevencaplar]

The data has been imported and published to data.lsdb.io.