Tracking automatically generated parameters

[371258]

We are currently facing an issue with logging of parameters in DVC Live. Our project structure is shown below:

├───src
├───dvclive
│ └───plots
│ └───metrics.json
├───dvc.yaml
├───params.yaml

DVC Version: 3.39.0

We manually modify params.yaml to set parameters for running experiments. This file mainly includes parameters that
adapt data clearing. It does not include any hyperparameters of our model, because those are
defined automatically using Optunas hyperparameter tuning. Nevertheless, we would like to log and track the hyperparameters found by Optuna.

What is the recommended way to do this?

We tried the following:

    with Live() as live:
        live.log_params(model.get_params())

While debugging we can see that the expected file is created in dvclive/params.yaml. But the file is removed automatically
once the DVC pipeline finished.

Any help is highly appreciated.

[dberenbaum]

Could you share the repo, or dvc.yaml structure, and/or code, as well as the versions of dvc and dvclive you are using? It's not clear why the dvclive/params.yaml is being removed when the pipeline finishes.

[371258]

Hi @dberenbaum ,

some further information:

DVC version: 3.39.0
DVCLive version: 3.5.1

dvc.yaml

vars:
- raw_data: data/raw/data.csv
- cleared_data: data/processed/cleared.csv
- train_data: data/processed/train.csv
- test_data: data/processed/test.csv
- scaler: data/processed/scaler.bin
- model: models/model.sav

stages:
  clear:
    cmd: python src/clear.py --RAW_DATA=${raw_data}
    deps:
    - src/clear.py
    - ${raw_data}
    params:
    - shared
    - clear
    outs:
    - ${cleared_data}
  prepare:
    cmd: python src/prepare.py --CLEARED_DATA=${cleared_data}
    deps:
    - src/prepare.py
    - ${cleared_data}
    params:
    - shared
    - prepare
    outs:
    - ${train_data}
    - ${test_data}
    - ${scaler}
  train:
    cmd: python src/train.py --TRAIN_DATA=${train_data} --TEST_DATA=${test_data}
    deps:
    - src/train.py
    - ${train_data}
    - ${test_data}
    params:
    - shared
    - train
    outs:
    - ${model}
    - dvclive/optuna.db
  evaluate:
    cmd: python src/evaluate.py --MODEL=${model} --TEST_DATA=${test_data}
    deps:
    - src/evaluate.py
    - ${model}
    - ${test_data}
    params:
    - shared
    - evaluate

params:
- params.yaml

Script for training stage (train.py) which is trying to log the model parameters (see line 27+28):

import joblib
import pandas as pd
import argparse
import os
import yaml
import xgboost as xgb
import optuna
from sklearn.metrics import recall_score, f1_score
from dvclive import Live

def train_model(train_data_path:str, test_data_path:str):
    
    // read train and test data
    // init x_train, y_train, x_test, y_test 

    hyperparams = tune_model_hyperparams(x_train, y_train, x_test, y_test)

    model = xgb.XGBClassifier()
    model.set_params(**hyperparams)
    model.fit(x_train, y_train)

    os.makedirs(os.path.join("models"), exist_ok=True)

    with open("models/model.sav", 'wb') as f:
        joblib.dump(model, f)

    with Live() as live:
        live.log_params(model.get_params())


def tune_model_hyperparams(x_train:pd.DataFrame, y_train:pd.DataFrame, x_test:pd.DataFrame, y_test:pd.DataFrame):
    optuna.logging.set_verbosity(optuna.logging.WARNING)

    def objective(trial):
        // Code for Optuna hyperparameter tuning objective
        return score

    study_name = "xgb-optuna-hyperparam-tuning"
    db_name = 'optuna.db'
    storage_name = f'sqlite:///dvclive/{db_name}'

    study = optuna.create_study(study_name=study_name, storage=storage_name, direction='maximize', load_if_exists=True)
    study.optimize(objective, n_trials=20)

    return study.best_params


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument("--TRAIN_DATA", default='data/processed/train.csv')
    parser.add_argument("--TEST_DATA", default='data/processed/test.csv')
    args = parser.parse_args()

    train_data_path = args.TRAIN_DATA
    test_data_path = args.TEST_DATA
    
    train_model(train_data_path, test_data_path)

if __name__ == "__main__":
    main()

[dberenbaum]

Thanks for the details! A few things I notice:

1. You may want to write the params and other dvclive outputs to a different directory than where you save the optuna database. It may be that optuna is deleting the files inside dvclive. Could you try to either change the dvclive path (with Live("some_other_dir") as live:) or change the optuna path?
2. At the bottom of the dvc.yaml, it lists the params path as params.yaml instead of dvclive/params.yaml. Not sure whether that was added by you or automatically by dvclive, but there should be a path there to dvclive/params.yaml.
3. You may wish to add the directory you use for DVCLive to the outs section of the train stage (you can add cache: false if you want to keep it in Git instead of the DVC cache). That way, DVC will ensure it exists when the stage completes and will raise an error if not.

[371258]

Thanks a lot for your suggestions! I apologize for the delayed response, but it took me some time to thoroughly test them.

You may want to write the params and other dvclive outputs to a different directory than where you save the optuna database. It may be that optuna is deleting the files inside dvclive. Could you try to either change the dvclive path (with Live("some_other_dir") as live:) or change the optuna path?

Saving the optuna database to a different directory instead of dvclive does not change the behavior: there is still no file dvclive/params.yaml when the DVC pipeline finishes.

At the bottom of the dvc.yaml, it lists the params path as params.yaml instead of dvclive/params.yaml. Not sure whether that was added by you or automatically by dvclive, but there should be a path there to dvclive/params.yaml.

In our current project structure we would expect to see both files:

• params.yaml in the root directory: I added this line manually to the params section of dvc.yaml. The file contains experiment parameters that can be adapted manually e.g. to modify data cleaning. Adapting any parameter within this file will have an effect on the outcome of experiments, since our source code uses this parameters.
• dvclive/params.yaml: should contain the model hyperparameters found by Optuna. Adapting any parameter within this file will not have any effect on the outcome of experiments, since parameters are not used by our source code. When I try to add this file manually to the params section of dvc.yaml then the line is removed automatically when the DVC pipeline finishes.

=> Might there be any issues with reserved file names?

You may wish to add the directory you use for DVCLive to the outs section of the train stage (you can add cache: false if you want to keep it in Git instead of the DVC cache). That way, DVC will ensure it exists when the stage completes and will raise an error if not.

I added dvclive/params.yaml to the outs of the train stage. That successfully prevents the file dvclive/params.yaml from being removed when the pipeline finishes. In addition to that, the file path is added automatically to the params section of dvc.yaml .

Thank you very much, the third suggestion solved our issue.

Nevertheless I would like to ask if there is a different best practice to log "read only" parameters (like the hyperparameters found by optuna in our case).

Thanks again and best regards

[dberenbaum]

Using live.log_params() is the expected way to log these types of parameters, so it looks to me like you are on the right track. I am unable to reproduce your issue in a minimal example, so I'm not sure why it's not working as you expect unless you can provide a simple way to reproduce the issue.

I added dvclive/params.yaml to the outs of the train stage. That successfully prevents the file dvclive/params.yaml from being removed when the pipeline finishes. In addition to that, the file path is added automatically to the params section of dvc.yaml .

I would suggest including all of dvclive as outs of the train stage since you also want to ensure that metrics and other info is tracked as an output of that stage.