Add the greedy ambiguity solver to throughput examples #1053

beomki-yeo · 2025-07-02T21:06:49Z

This PR depends on #1057

The greedy ambiguity solver is added to the CUDA full chain

beomki-yeo · 2025-07-04T21:27:08Z

Maybe adding a new algorithm to the chain makes comparison currupted?
BTW, we might need to use different types of markers in the graph as the colors seems overlapping.

I knew that rearrange_tracks is also a bottleneck but didn't expect it would be this much. Good to know..

beomki-yeo · 2025-07-10T00:00:05Z

@stephenswat Can you generate the plot again as the issue on covfie hash has been resolved?

stephenswat · 2025-07-10T08:27:41Z

Running now! Mind you that due to the large number of kernels the ambiguity resolution needs to run, the benchmark takes about 3 hours per commit.

stephenswat · 2025-07-10T14:38:55Z

Hi @beomki-yeo I tried running the multi-threaded CUDA throughput example with this PR and it is giving me segmentation faults. Have you seen this?

beomki-yeo · 2025-07-10T16:27:01Z

I will check

beomki-yeo · 2025-07-10T18:54:55Z

It's running normally for me. Could you share your command?

gcc version: 31.1
CUDA Version: 12.4

byeo@hermes:/mnt/nvme0n1/byeo/projects/traccc/traccc_build$    ./bin/traccc_throughput_mt_cuda \
   --digitization-file=geometries/odd/odd-digi-geometric-config.json \
   --detector-file=geometries/odd/odd-detray_geometry_detray.json \
   --material-file=geometries/odd/odd-detray_material_detray.json \
   --grid-file=geometries/odd/odd-detray_surface_grids_detray.json \
   --use-detray-detector=true \
   --use-acts-geom-source=true \
   --input-directory=odd/odd-simulations-20240509/geant4_ttbar_mu200/ \
   --input-events=10
11:23:54    ThroughputExampleOptions      INFO      
11:23:54    ThroughputExampleOptions      INFO      Running Multi-threaded CUDA GPU throughput tests
11:23:54    ThroughputExampleOptions      INFO      
11:23:54    ThroughputExampleOptions      INFO      Detector Options:
11:23:54    ThroughputExampleOptions      INFO      ├ Detector file:                          geometries/odd/odd-detray_geometry_detray.json
11:23:54    ThroughputExampleOptions      INFO      ├ Material file:                          geometries/odd/odd-detray_material_detray.json
11:23:54    ThroughputExampleOptions      INFO      ├ Surface grid file:                      geometries/odd/odd-detray_surface_grids_detray.json
11:23:54    ThroughputExampleOptions      INFO      ├ Use detray detector:                    true
11:23:54    ThroughputExampleOptions      INFO      └ Digitization file:                      geometries/odd/odd-digi-geometric-config.json
11:23:54    ThroughputExampleOptions      INFO      Magnetic Field Options:
11:23:54    ThroughputExampleOptions      INFO      ├ Read magnetic field from file:          false
11:23:54    ThroughputExampleOptions      INFO      ├ Magnetic field file:                    geometries/odd/odd-bfield.cvf
11:23:54    ThroughputExampleOptions      INFO      ├ Magnetic field file format:             binary
11:23:54    ThroughputExampleOptions      INFO      └ Magnetic field value:                   2 T
11:23:54    ThroughputExampleOptions      INFO      Input Data Options:
11:23:54    ThroughputExampleOptions      INFO      ├ Use ACTS geometry source:               true
11:23:54    ThroughputExampleOptions      INFO      ├ Input data format:                      csv
11:23:54    ThroughputExampleOptions      INFO      ├ Input directory:                        odd/odd-simulations-20240509/geant4_ttbar_mu200/
11:23:54    ThroughputExampleOptions      INFO      ├ Number of input events:                 10
11:23:54    ThroughputExampleOptions      INFO      └ Number of skipped events:               0
11:23:54    ThroughputExampleOptions      INFO      Clusterization Options:
11:23:54    ThroughputExampleOptions      INFO      ├ Threads per partition:                  256
11:23:54    ThroughputExampleOptions      INFO      ├ Target cells per thread:                8
11:23:54    ThroughputExampleOptions      INFO      ├ Max cells per thread:                   16
11:23:54    ThroughputExampleOptions      INFO      └ Scratch space multiplier:               256
11:23:54    ThroughputExampleOptions      INFO      Track Seeding Options:
11:23:54    ThroughputExampleOptions      INFO      Track Finding Options:
11:23:54    ThroughputExampleOptions      INFO      ├ Max branches per seed:                  10
11:23:54    ThroughputExampleOptions      INFO      ├ Max branches at surface:                2
11:23:54    ThroughputExampleOptions      INFO      ├ Track candidate range:                  3:100
11:23:54    ThroughputExampleOptions      INFO      ├ Min step length to next surface:        1.200000 mm
11:23:54    ThroughputExampleOptions      INFO      ├ Max step count to next surface:         100
11:23:54    ThroughputExampleOptions      INFO      ├ Max Chi2:                               10.000000
11:23:54    ThroughputExampleOptions      INFO      ├ Max holes per candidate:                3
11:23:54    ThroughputExampleOptions      INFO      ├ PDG number:                             13
11:23:54    ThroughputExampleOptions      INFO      └ Minimum total track momentum:           0.100000
11:23:54    ThroughputExampleOptions      INFO      Track Propagation Options:
11:23:54    ThroughputExampleOptions      INFO      ├ Navigation:
11:23:54    ThroughputExampleOptions      INFO      │ ├ Min mask tolerance:                   0.000010 mm
11:23:54    ThroughputExampleOptions      INFO      │ ├ Max mask tolerance:                   3.000000 mm
11:23:54    ThroughputExampleOptions      INFO      │ ├ Mask tolerance scalar:                0.050000
11:23:54    ThroughputExampleOptions      INFO      │ ├ Path tolerance:                       1.000000 um
11:23:54    ThroughputExampleOptions      INFO      │ ├ Overstep tolerance:                   -999.999939 um
11:23:54    ThroughputExampleOptions      INFO      │ └ Search window:                        0 x 0
11:23:54    ThroughputExampleOptions      INFO      ├ Transport:
11:23:54    ThroughputExampleOptions      INFO      │ ├ Min step size:                        0.000100 mm
11:23:54    ThroughputExampleOptions      INFO      │ ├ Runge-Kutta tolerance:                0.000100 mm
11:23:54    ThroughputExampleOptions      INFO      │ ├ Max step updates:                     10000
11:23:54    ThroughputExampleOptions      INFO      │ ├ Step size constraint:                 340282346638528859811704183484516925440.000000 mm
11:23:54    ThroughputExampleOptions      INFO      │ ├ Path limit:                           5.000000 m
11:23:54    ThroughputExampleOptions      INFO      │ ├ Min step size:                        0.000100 mm
11:23:54    ThroughputExampleOptions      INFO      │ ├ Enable Bethe energy loss:             true
11:23:54    ThroughputExampleOptions      INFO      │ ├ Enable covariance transport:          true
11:23:54    ThroughputExampleOptions      INFO      │ └ Covariance transport:
11:23:54    ThroughputExampleOptions      INFO      │   ├ Enable energy loss gradient:        false
11:23:54    ThroughputExampleOptions      INFO      │   └ Enable B-field gradient:            false
11:23:54    ThroughputExampleOptions      INFO      └ Geometry context:
11:23:54    ThroughputExampleOptions      INFO      Track Fitting Options:
11:23:54    ThroughputExampleOptions      INFO      ├ Number of iterations:                   1
11:23:54    ThroughputExampleOptions      INFO      ├ Particle hypothesis PDG:                13
11:23:54    ThroughputExampleOptions      INFO      ├ Covariance inflation factor:            1000.000000
11:23:54    ThroughputExampleOptions      INFO      ├ Barcode sequence size factor:           5
11:23:54    ThroughputExampleOptions      INFO      ├ Minimum capacity of barcode sequence:   100
11:23:54    ThroughputExampleOptions      INFO      └ Mask tolerance for the backward filter: 5.000000
11:23:54    ThroughputExampleOptions      INFO      Throughput Measurement Options:
11:23:54    ThroughputExampleOptions      INFO      ├ Cold run events:                        10
11:23:54    ThroughputExampleOptions      INFO      ├ Processed events:                       100
11:23:54    ThroughputExampleOptions      INFO      ├ Log file:                               
11:23:54    ThroughputExampleOptions      INFO      ├ Deterministic ordering:                 false
11:23:54    ThroughputExampleOptions      INFO      └ Random seed:                            time-based
11:23:54    ThroughputExampleOptions      INFO      Multi-Threading Options:
11:23:54    ThroughputExampleOptions      INFO      └ Number of CPU thread:                   1
11:23:54    ThroughputExampleOptions      INFO      
INFO: Reading detector files... Done
INFO: Building detector: Cylindrical detector from DD4hep blueprint... Done
INFO: Checking detector consistency...
WARNING: No material in detector
WARNING: No entries in volume finder
INFO: Consistency check: OK
INFO: Host detector construction complete
INFO: Reading detector files... Done
INFO: Building detector: Cylindrical detector from DD4hep blueprint... Done
INFO: Checking detector consistency...
WARNING: No entries in volume finder
INFO: Consistency check: OK
INFO: Host detector construction complete
11:24:03    ThroughputExample             WARNING   17082 duplicate cells found in /mnt/nvme0n1/byeo/projects/traccc/traccc/data/odd/odd-simulations-20240509/geant4_ttbar_mu200/event000000005-cells.csv
11:24:03    ThroughputExample             WARNING   16264 duplicate cells found in /mnt/nvme0n1/byeo/projects/traccc/traccc/data/odd/odd-simulations-20240509/geant4_ttbar_mu200/event000000001-cells.csv
11:24:03    ThroughputExample             WARNING   18654 duplicate cells found in /mnt/nvme0n1/byeo/projects/traccc/traccc/data/odd/odd-simulations-20240509/geant4_ttbar_mu200/event000000003-cells.csv
11:24:03    ThroughputExample             WARNING   18367 duplicate cells found in /mnt/nvme0n1/byeo/projects/traccc/traccc/data/odd/odd-simulations-20240509/geant4_ttbar_mu200/event000000006-cells.csv
11:24:03    ThroughputExample             WARNING   16623 duplicate cells found in /mnt/nvme0n1/byeo/projects/traccc/traccc/data/odd/odd-simulations-20240509/geant4_ttbar_mu200/event000000008-cells.csv
11:24:03    ThroughputExample             WARNING   17965 duplicate cells found in /mnt/nvme0n1/byeo/projects/traccc/traccc/data/odd/odd-simulations-20240509/geant4_ttbar_mu200/event000000007-cells.csv
11:24:03    ThroughputExample             WARNING   16881 duplicate cells found in /mnt/nvme0n1/byeo/projects/traccc/traccc/data/odd/odd-simulations-20240509/geant4_ttbar_mu200/event000000002-cells.csv
11:24:03    ThroughputExample             WARNING   18943 duplicate cells found in /mnt/nvme0n1/byeo/projects/traccc/traccc/data/odd/odd-simulations-20240509/geant4_ttbar_mu200/event000000000-cells.csv
11:24:03    ThroughputExample             WARNING   19031 duplicate cells found in /mnt/nvme0n1/byeo/projects/traccc/traccc/data/odd/odd-simulations-20240509/geant4_ttbar_mu200/event000000004-cells.csv
11:24:03    ThroughputExample             WARNING   19889 duplicate cells found in /mnt/nvme0n1/byeo/projects/traccc/traccc/data/odd/odd-simulations-20240509/geant4_ttbar_mu200/event000000009-cells.csv
Using CUDA device: NVIDIA A30 [id: 0, bus: 71, device: 0]
Using CUDA device: NVIDIA A30 [id: 0, bus: 71, device: 0]
Warm-up processing [==================================================] 100% [00m:00s]                                                                                                                               
Event processing   [==================================================] 100% [00m:00s]                                                                                                                               
11:24:35 AM ThroughputExample             INFO      Reconstructed track parameters: 510430
11:24:35 AM ThroughputExample             INFO      Time totals:                   File reading  1012 ms
11:24:35 AM ThroughputExample             INFO                  Warm-up processing  3049 ms
11:24:35 AM ThroughputExample             INFO                    Event processing  28801 ms
11:24:35 AM ThroughputExample             INFO      Throughput:            Warm-up processing  304.953 ms/event, 3.2792 events/s
11:24:35 AM ThroughputExample             INFO                    Event processing  288.011 ms/event, 3.47209 events/s

sonarqubecloud · 2025-07-10T19:31:32Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
3.0% Duplication on New Code

See analysis details on SonarQube Cloud

beomki-yeo · 2025-07-24T03:05:20Z

The above benchmark may not be correct due to the bugs reported in #1083
When I eyeballed the profiling results, count_removable_tracks rearrange_tracks and remove_tracks are the main bottleneck.

sonarqubecloud · 2025-08-29T23:33:04Z

Quality Gate passed

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
2.8% Duplication on New Code

See analysis details on SonarQube Cloud

beomki-yeo · 2025-08-30T00:10:19Z

This is the output of single thread chain:

This PR

Using CUDA device: NVIDIA A30 [id: 0, bus: 71, device: 0]
Warm-up processing [==================================================] 100% [00m:00s]                                                                                                                         
Event processing   [==================================================] 100% [00m:00s]                                                                                                                         
Reconstructed track parameters: 1469663
Time totals:
                  File reading  7548 ms
            Warm-up processing  32495 ms
              Event processing  9614 ms
Throughput:
            Warm-up processing  3249.58 ms/event, 0.307732 events/s
              Event processing  96.1429 ms/event, 10.4012 events/s

Main

Using CUDA device: NVIDIA A30 [id: 0, bus: 71, device: 0]
Warm-up processing [==================================================] 100% [00m:00s]                                                                                                                         
Event processing   [==================================================] 100% [00m:00s]                                                                                                                         
Reconstructed track parameters: 1919511
Time totals:
                  File reading  7586 ms
            Warm-up processing  958 ms
              Event processing  8343 ms
Throughput:
            Warm-up processing  95.8378 ms/event, 10.4343 events/s
              Event processing  83.4309 ms/event, 11.986 events/s

As the ambiguity solver still increases the even processing time 15% but I think this number can be tolerated considering the importance of ambiguity resolver. There is still room to improve in the ambiguity solver as well

beomki-yeo · 2025-08-30T03:17:35Z

@stephenswat If you don't mind, could you also generate the plots for throughput and physical performance?

stephenswat · 2025-08-30T08:17:53Z

Running the compute plots for you now. The physics plots are based on the CUDA seeding example, so I cannot make those from this commit unfortunately; the ambiguity resolution would need to be added to the seeding example.

stephenswat · 2025-08-30T08:18:57Z

By the way, on the RTX A5000 we have at CERN the ambiguity resolution tests fail:

[----------] 2 tests from CUDALong/CUDAGreedyResolutionCompareToCPU
[ RUN      ] CUDALong/CUDAGreedyResolutionCompareToCPU.Comparison/0
Event: 0 Seed: 42
 Time for the cpu method 951 ms
 Time for the cuda method 53 ms
/mnt/ssd1/sswatman/traccc/tests/cuda/test_ambiguity_resolution.cpp:922: Failure
Expected equality of these values:
  n_tracks_cpu
    Which is: 1
  res_trk_cands_cuda.capacity()
    Which is: 9851

[  FAILED  ] CUDALong/CUDAGreedyResolutionCompareToCPU.Comparison/0, where GetParam() = (3, 10000, { 3, 500 }, 10000, true) (1235 ms)
[ RUN      ] CUDALong/CUDAGreedyResolutionCompareToCPU.Comparison/1
Event: 0 Seed: 42
 Time for the cpu method 976 ms
 Time for the cuda method 53 ms
/mnt/ssd1/sswatman/traccc/tests/cuda/test_ambiguity_resolution.cpp:922: Failure
Expected equality of these values:
  n_tracks_cpu
    Which is: 1
  res_trk_cands_cuda.capacity()
    Which is: 9861

[  FAILED  ] CUDALong/CUDAGreedyResolutionCompareToCPU.Comparison/1, where GetParam() = (3, 10000, { 3, 500 }, 10000, false) (1361 ms)

Is this something you have seen?

stephenswat · 2025-08-30T08:39:33Z

Performance summary

Here is a summary of the performance effects of this PR:

Graphical

Tabular

Kernel	Reciprocal Throughput			Parallelism
Kernel	`1241c09`	`0817b70`	Delta	`1241c09`	`0817b70`
`propagate_to_next_surface`	12.57 ms	12.56 ms	-0.0%	2.67	2.67
`fit_forward`	5.99 ms	4.63 ms	-22.6%	3.13	3.79
`rearrange_tracks`	—	3.02 ms	—	nan	1.88
`fit_backward`	3.32 ms	2.51 ms	-24.4%	2.35	2.84
`sort_tracks_per_measurement`	—	1.55 ms	—	nan	1.00
`find_tracks`	1.37 ms	1.37 ms	0.5%	1.88	1.88
`ccl_kernel`	826.82 μs	828.99 μs	0.3%	1.37	1.37
`count_doublets`	637.37 μs	633.11 μs	-0.7%	1.61	1.61
`count_triplets`	589.33 μs	590.66 μs	0.2%	1.02	1.02
`find_doublets`	448.55 μs	451.17 μs	0.6%	3.08	3.08
`Thrust::sort`	416.69 μs	440.80 μs	5.8%	7.30	7.94
`update_status`	—	340.72 μs	—	nan	3.83
`fill_inverted_ids`	—	287.32 μs	—	nan	3.81
`find_triplets`	172.38 μs	173.25 μs	0.5%	1.31	1.31
`block_inclusive_scan`	—	123.00 μs	—	nan	11.41
`add_block_offset`	—	90.86 μs	—	nan	11.45
`remove_duplicates`	56.18 μs	56.31 μs	0.2%	17.33	17.34
`select_seeds`	53.31 μs	53.18 μs	-0.2%	1.34	1.34
`remove_tracks`	—	45.17 μs	—	nan	192.00
`unknown`	20.52 μs	40.61 μs	97.9%	2.25	2.87
`fit_prelude`	35.75 μs	26.25 μs	-26.6%	9.38	11.04
`populate_grid`	23.41 μs	23.37 μs	-0.1%	1.22	1.22
`estimate_track_params`	22.75 μs	22.77 μs	0.1%	2.15	2.15
`fill_tracks_per_measurement`	—	22.59 μs	—	nan	6.52
`count_grid_capacities`	22.14 μs	22.09 μs	-0.3%	1.22	1.22
`apply_interaction`	19.16 μs	19.03 μs	-0.7%	7.34	7.32
`update_triplet_weights`	15.01 μs	15.12 μs	0.7%	1.27	1.27
`build_tracks`	12.80 μs	12.81 μs	0.1%	6.34	6.34
`fill_finding_propagation_sort_keys`	12.65 μs	12.73 μs	0.6%	7.99	8.00
`form_spacepoints`	12.35 μs	12.28 μs	-0.5%	1.48	1.49
`sort_updated_tracks`	—	7.42 μs	—	nan	192.00
`fill_track_candidates`	—	6.50 μs	—	nan	7.76
`reduce_triplet_counts`	6.31 μs	6.32 μs	0.1%	3.08	3.08
`fill_finding_duplicate_removal_sort_keys`	5.35 μs	5.36 μs	0.3%	22.03	22.09
`fill_vectors`	—	5.31 μs	—	nan	6.39
`count_shared_measurements`	—	4.60 μs	—	nan	6.42
`fill_unique_meas_id_map`	—	2.44 μs	—	nan	1.55
`make_barcode_sequence`	1.01 μs	1.01 μs	0.2%	3.83	3.83
`scan_block_offsets`	—	870.26 ns	—	nan	1240.28
`DeviceReduceKernel`	—	563.17 ns	—	nan	16.51
`fill_fitting_sort_keys`	386.81 ns	372.41 ns	-3.7%	9.48	11.80
`fill_prefix_sum`	171.92 ns	172.02 ns	0.1%	341.30	341.30
`DeviceSelectSweepKernel`	—	91.95 ns	—	nan	67.25
`DeviceReduceSingleTileKernel`	—	27.08 ns	—	nan	256.01
`DeviceCompactInitKernel`	—	3.25 ns	—	nan	768.00
Total	26.65 ms	30.02 ms	12.7%	2.71	3.14

Important

All metrics in this report are given as reciprocal throughput, not as wallclock runtime.

Warning

At least one kernel incurred a significant performance regression.

Note

This is an automated message produced upon the explicit request of a human being.

beomki-yeo · 2025-09-08T20:34:57Z

Is this something you have seen?

This type of failure in CUDALong is usually from synchronization or uninitialized vector issue. For myself to reproduce the error, Which CUDA and nvidia driver version did you use for the test?

beomki-yeo · 2025-09-09T23:40:10Z

Just as an update, I found that the tests on RTX A6000 are OK with cuda 12.4 and 12.6.

beomki-yeo · 2025-09-17T00:17:05Z

@stephenswat Would you be able to check if your failure is relevant with #1159 ?

beomki-yeo force-pushed the add-ambiguity-solver-to-chain branch 2 times, most recently from 022693e to 126a871 Compare July 2, 2025 22:10

beomki-yeo marked this pull request as draft July 2, 2025 23:06

beomki-yeo force-pushed the add-ambiguity-solver-to-chain branch from 126a871 to d61322d Compare July 3, 2025 19:36

beomki-yeo marked this pull request as ready for review July 3, 2025 19:38

beomki-yeo force-pushed the add-ambiguity-solver-to-chain branch 2 times, most recently from 4c9698f to adcb664 Compare July 3, 2025 19:42

beomki-yeo requested review from krasznaa and stephenswat July 3, 2025 21:16

beomki-yeo force-pushed the add-ambiguity-solver-to-chain branch 2 times, most recently from 62adba1 to 88347ea Compare July 9, 2025 23:59

This comment was marked as outdated.

Sign in to view

beomki-yeo force-pushed the add-ambiguity-solver-to-chain branch 2 times, most recently from b4437b2 to 154f3c7 Compare July 10, 2025 19:30

beomki-yeo marked this pull request as draft August 23, 2025 06:08

beomki-yeo marked this pull request as ready for review August 29, 2025 03:00

beomki-yeo force-pushed the add-ambiguity-solver-to-chain branch from 5e61612 to 4d31318 Compare August 29, 2025 23:09

Add ambiguity resolution to the chain

0817b70

beomki-yeo force-pushed the add-ambiguity-solver-to-chain branch from 4d31318 to 0817b70 Compare August 29, 2025 23:32

beomki-yeo mentioned this pull request Sep 9, 2025

Add ambiguity resoltion algorithm to seeding_example_cuda #1154

Open

Add the greedy ambiguity solver to throughput examples #1053

Are you sure you want to change the base?

Add the greedy ambiguity solver to throughput examples #1053

Uh oh!

Conversation

beomki-yeo commented Jul 2, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

beomki-yeo commented Jul 4, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

beomki-yeo commented Jul 10, 2025

Uh oh!

stephenswat commented Jul 10, 2025

Uh oh!

This comment was marked as outdated.

stephenswat commented Jul 10, 2025

Uh oh!

beomki-yeo commented Jul 10, 2025

Uh oh!

beomki-yeo commented Jul 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sonarqubecloud bot commented Jul 10, 2025

Quality Gate passed

Uh oh!

beomki-yeo commented Jul 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

sonarqubecloud bot commented Aug 29, 2025

Quality Gate passed

Uh oh!

beomki-yeo commented Aug 30, 2025

Uh oh!

beomki-yeo commented Aug 30, 2025

Uh oh!

stephenswat commented Aug 30, 2025

Uh oh!

stephenswat commented Aug 30, 2025

Uh oh!

stephenswat commented Aug 30, 2025

Performance summary

Graphical

Tabular

Uh oh!

beomki-yeo commented Sep 8, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

beomki-yeo commented Sep 9, 2025

Uh oh!

beomki-yeo commented Sep 17, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

beomki-yeo commented Jul 2, 2025 •

edited

Loading

beomki-yeo commented Jul 4, 2025 •

edited

Loading

beomki-yeo commented Jul 10, 2025 •

edited

Loading

beomki-yeo commented Jul 24, 2025 •

edited

Loading

beomki-yeo commented Sep 8, 2025 •

edited

Loading