[Feature] When using unitxt with lm-evaluation-harness, need to provide a mechanism to specify the local metrics location

[yhwang]

When using unitxt with lm-evaluation-harness, users can only put the local metrics modules in the current working directory. Need a better mechanism to allow users to specify the location of the local metrics modules.

[yhwang]

cc @yoavkatz

[yoavkatz]

You are right. The metric code seems to be always downloaded to HF_HOME . The cache_dir parameter is only used to store temporary reference and predictions.

So I think we have no option but to use the same approach we used in lm eval. Add a prefix to the metric name and store the code under a subdirectory with the same name.

@elronbandel . Please give your input if you see any other option.

[elronbandel]

        self.metric = evaluate.load(
            self.hf_metric_name, experiment_id=str(uuid.uuid4()), download_config=DownloadConfig(cache_dir="./unitxt_cache")
        )

This seems to be download the code to the specified cache_dir

[yoavkatz]

@yhwang - Elron created a PR for the idea above (https://github.com/IBM/unitxt/pull/1603/files)

The benefit of this approach is that you can run unitxt to download the metrics code to the cache while in online mode (and also datasets or metrics) , and then switch to offline mode and everything works. You don't have to explicitly fetch and store the metric code in different directories.
The downside is that the assets are loaded in HF cache format (including SHA for the files ).

To the process is:

env UNITXT_LOCAL_CACHE=./unitxt_cache python prepare/metrics/wer.py
env HF_EVALUATE_OFFLINE=1 UNITXT_LOCAL_CACHE=./unitxt_cache python prepare/metrics/wer.py

Do you think?

[yoavkatz]

@elronbandel - After more checking I see that HF_HOME is still used:

rm -fr ~/.cache/huggingface
env UNITXT_LOCAL_CACHE=./unitxt_cache python prepare/metrics/wer.py
echo HOME
ls -l ~/.cache/huggingface
echo UNITXT_CACHE
ls -l ./unitxt_cache

This is the output:

HOME
total 0
drwxr-xr-x 3 yoavkatz staff 96 Feb 18 13:52 hub
drwxr-xr-x 4 yoavkatz staff 128 Feb 18 13:52 modules
UNITXT_CACHE
total 24
-rw------- 1 yoavkatz staff 4485 Feb 18 13:41 56a97edde9c400b3c6d99f392e976f7b6fe9eeef145ee26fa149d939f5eeab00.7e3cbd712bcd0e3edcbf59696a6efcfd5592df3604dd05cda891d94e17f8cfcc.py
-rw-r--r-- 1 yoavkatz staff 136 Feb 18 13:41 56a97edde9c400b3c6d99f392e976f7b6fe9eeef145ee26fa149d939f5eeab00.7e3cbd712bcd0e3edcbf59696a6efcfd5592df3604dd05cda891d94e17f8cfcc.py.json
-rw-r--r-- 1 yoavkatz staff 0 Feb 18 13:41 56a97edde9c400b3c6d99f392e976f7b6fe9eeef145ee26fa149d939f5eeab00.7e3cbd712bcd0e3edcbf59696a6efcfd5592df3604dd05cda891d94e17f8cfcc.py.lock
drwxr-xr-x 3 yoavkatz staff 96 Feb 18 13:41 wer

[yoavkatz]

This is the full list of generated output:

find ~/.cache/huggingface
/Users/yoavkatz/.cache/huggingface
/Users/yoavkatz/.cache/huggingface/modules
/Users/yoavkatz/.cache/huggingface/modules/init.py
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/metrics
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/metrics/init.py
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/metrics/pycache
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/metrics/pycache/init.cpython-310.pyc
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--wer
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--wer/init.py
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--wer/pycache
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--wer/pycache/init.cpython-310.pyc
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--wer/85bee9e4216a78bb09b2d0d500f6af5c23da58f9210e661add540f5df6630fcd
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--wer/85bee9e4216a78bb09b2d0d500f6af5c23da58f9210e661add540f5df6630fcd/wer.json
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--wer/85bee9e4216a78bb09b2d0d500f6af5c23da58f9210e661add540f5df6630fcd/init.py
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--wer/85bee9e4216a78bb09b2d0d500f6af5c23da58f9210e661add540f5df6630fcd/pycache
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--wer/85bee9e4216a78bb09b2d0d500f6af5c23da58f9210e661add540f5df6630fcd/pycache/init.cpython-310.pyc
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--wer/85bee9e4216a78bb09b2d0d500f6af5c23da58f9210e661add540f5df6630fcd/pycache/wer.cpython-310.pyc
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--wer/85bee9e4216a78bb09b2d0d500f6af5c23da58f9210e661add540f5df6630fcd/wer.py
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/metrics/evaluate-metric--wer.lock
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/init.py
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/pycache
/Users/yoavkatz/.cache/huggingface/modules/evaluate_modules/pycache/init.cpython-310.pyc
/Users/yoavkatz/.cache/huggingface/hub
/Users/yoavkatz/.cache/huggingface/hub/version.txt

[yhwang]

@yoavkatz and @elronbandel thanks for the PR and info. And also tested it out. I will try it. And like you pointed out, the file names are still in SHA format. I will keep thinking of any other way to make it more human-readable, maybe have a mapping list of this in a readable format.

[yoavkatz]

Hi @yhwang - We checked it again - and because HF still uses HF_HOME for some of the files (as seen above), we will just add a new env variable to specify he metric local path, like was done in LM-eval solution,

[elronbandel]

We created a pull request that enables offline mode where all huggingface datasets, metrics and models are taken from pre defined directories. See the PR here:
#1603
@yhwang If you have any suggestions please let us know, otherwise we will merge this and release a new version.

[yhwang]

@elronbandel thanks for the PR. The changes look good to me. Again, on the user end, they still need a way to map a readable list to SHA files if they want to maintain their own offline models/datasets/metrics.

[yoavkatz]

Hi @yhwang . I'm not sure I understand the comment about the SHA file names. In the new approach the users just uses git clone to download the metric from hf into a local directory and then usec env variables to point to that directory. There is no use of the hf caches and their hashing scheme.

[yhwang]

Thanks for correcting me and reminding me to re-visit the PR. I thought it was the old implementation and missed the new change. The new code is perfect! Thanks again to @elronbandel and @yoavkatz