load_qa takes too much time

[yjsoh]

Seems like data = data.map(update) takes too much time. Could it be cached?

From data.py:

def load_qa(..):
   ...
   def update(sample):
        demos = demo_data
        demo_text = ""
        if shots > 0:
            if "popqa" in dataset:
                # popqa only has one split
                demos = demo_data.filter(lambda x: x[key] != sample[key])

            # seed ensures that we get the same demos for the same question
            # hashlib is deterministic while hash() is not in Python>=3.3, the seed has to be a positive integer
            h = (
                int(hashlib.sha256(str(sample[key]).encode("utf-8")).hexdigest(), 16)
                % 2**31
            )
            demos = demos.shuffle(seed=h)
            demos = drop_duplicates(demos, key).select(range(shots))
            demo_text = (
                "\n\n".join(
                    [
                        demo_template.format(
                            **d,
                            documents="\n\n".join(
                                [passage_template.format(**c) for c in d["ctxs"]]
                            ),
                            answer=d["answers"][0],
                        )
                        for d in demos
                    ]
                )
                + "\n\n"
            )
        passage_text = ""
        if len(sample["ctxs"]) > 0:
            passage_text = "\n\n".join(
                [passage_template.format(**c) for c in sample["ctxs"]]
            )
        return {
            "demos": demo_text,
            "context": passage_text,
            "answer": sample["answers"],
        }

    data = data.map(update)

[howard-yen]

Hi, thanks for your interest in our work! Huggingface's dataset supports automatic caching for all mapping operations, so after you do this once, the results should be saved and all subsequent runs would load it (assuming identical hyperparameters like num_max_samples and shots).
For more details, you can check out: https://huggingface.co/docs/datasets/en/cache

If that doesn't work, here are some alternatives:

• In map, you can using multi-processing with data = data.map(update, num_proc=16) or however many CPUs are available on your system, this should make the map significantly faster.
• After doing mapping once, you can explicitly save the data to disk and then change to load_qa method to always load from disk, but make sure that you are using identical hyperparameters in that case for consistency. See here for examples.

Hope this helps and please let me know if you run into any other issues!