About: How can I skip the execution of download() and/or process() ?

[Nokimann]

In Creating “Larger” Datasets, to load an exist dataset, we have to re-define Dataset class without override download() and process().

I wonder if there is another way without re-defining.

[rusty1s]

Can you explain the issue in more detail? In general, downloading and processing are skipped if they are not implemented as part of your Dataset class.

[rusty1s]

So is this issue resolved? Otherwise, I am still unsure about what you are trying to achieve. Can you give a small example/code snippet? Happy to help you out!

[Nokimann]

Let's say we have two custom datasets: one from InMemoryDataset and another from Dataset.

Let's use the two datasets twice. In general,

1. InMemoryDataset
   1st time usage: download() -> process() -> loaded data
   2nd time re-usage later: just load processed data

2. Dataset
   1st time usage: download() -> process() -> loaded data
   2nd time re-usage later: download() -> process() -> loaded data

So, the step download() -> process() is redundant at the 2nd load in the case Dataset. To remove the redundancy, we have to redefine the custom datasets before the 2nd usage.

[rusty1s]

I don't think this is true. The pipeline is the same for both InMemoryDataset and Dataset. As long as you define the raw_file_names and processed_file_names properties in a dataset, downloading and processing will be skipped.

[Nokimann]

I found my fault. The names of saved files are different from the names defined in processed_file_names. So, it repeatedly generates the dataset from scratch.

[rusty1s]

Ah, yes. That makes sense. Glad that this is now resolved :)