Best dvc strategy for comparing outs of experiment variations?

[dimitry12]

Here is an illustration of the scenario:

• assume I am doing unsupervised clustering of text/images,
• let's say I have an embedding stage and then clustering stage in the pipeline.
• Also assume for simplicity I have no targets/labels - I'm data-mining.
• I want to run multiple variations of the pipeline.

In this scenario I care about agreement/disagreement across variations. For example, I only want to spend time inspecting the results if they are sufficiently distinct from baseline.

What's the best approach for doing this with dvc? The very basic constraint I am bumping into is the fact that dvc-experiment variations' outs never even exist at the same time in the workspace.

I can only think of taking "outs" of baseline experiment and duplicating them as silver-labels. Then these silver-labels will exist across all variations and I can metric-compare variations-outs against silver-labels. But this gets out of hand fast because as soon as I find sufficiently distinct new variation, I will need a second set of silver-labels.

[aayostem]

@dimitry12 Have you tried using DVC Metrics and Comparisons. Try generate comparison metrics as part of your pipeline

# dvc.yaml
stages:
  embed:
    cmd: python embed.py
    deps:
      - embed.py
      - data/raw
    outs:
      - embeddings.pkl
    metrics:
      - embedding_metrics.json:
          cache: false
  
  cluster:
    cmd: python cluster.py
    deps:
      - cluster.py
      - embeddings.pkl
    outs:
      - clusters.pkl
    metrics:
      - cluster_metrics.json:
          cache: false
  
  compare:
    cmd: python compare.py --baseline clusters_baseline.pkl --current clusters.pkl
    deps:
      - compare.py
      - clusters.pkl
    params:
      - clustering.variation
    metrics:
      - comparison_metrics.json:
          cache: false

[dimitry12]

I use metrics, and I don't know about "comparisons" and I don't see "comparisons" functionality on DVC docs.

In your example, I need to compare clusters.pkl from experiment A and clusters.pkl from experiment B - both produced by the same stage. Sure, if I only have a few experiments I can start spawning file-copies (as you implicitly suggest with clusters_baseline.pkl) but it is impractical at any realistic scale.