LightningDataModule in Colab on a dataset saved on my PC

[asad-ak]

Hello,
I have a question regarding how to go about using the LightningDataModule on a dataset that I have on my PC, every example I found was of MNIST which is downloaded on runtime. I have confusion in how to declare its path and specify it in my LightningDataModule functions.

[akihironitta]

@asad-ak (I might not understand your question, but) if your dataset is available on your machine, then you can just skip downloading it in LightningDataModule and load it in your LightningDataModule. It should be something like:

class MyDataModule(LightningModule):
    def __init__(self, ..., data_dir: str):
        self.data_dir = data_dir

    def train_dataloader(self):
        return DataLoader(MyDataset(self.data_dir))

dm = MyDataModule(..., data_dir="path/to/data/")

[asad-ak]

(https://colab.research.google.com/drive/1fDeXMOoek4E2i0C_CkAXNP8bozMbTknk?usp=sharing)
This is what I am currently working on can you help me understand the problems I have in this?

[akihironitta]

@asad-ak As the error mentions, you're trying to call DataFrame instance, which is not callable, in your setup:

class VIPDataModule(pl.LightningDataModule):
  def __init__(self, train_data:pd.DataFrame, test_data:pd.DataFrame, batch_size=32):
    self.train_data = train_data
    self.test_data = test_data
    self.batch_size = batch_size
    self.transform = transforms.ToTensor()
  
  #17:02
  def setup(self, stage: Optional[str] = None):
    # pre-processing training testing split
    self.train_df = self.train_data(transform=self.transform)
    self.test_df = self.test_data(transform=self.transform)

<ipython-input-18-622e859b64ab> in setup(self, stage)
     11   def setup(self, stage: Optional[str] = None):
     12     # pre-processing training testing split
---> 13     self.train_df = self.train_data(transform=self.transform)
     14     self.test_df = self.test_data(transform=self.transform)
     15 

TypeError: 'DataFrame' object is not callable

[asad-ak]

Alright so I tried another approach and I get a strange error at the end saying:

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.

[akihironitta]

@asad-ak Not sure what you've tried, but maybe https://stackoverflow.com/questions/68166721/cuda-error-device-side-assert-triggered-on-colab might help? Also, I would first make sure that the code works in a CPU environment.

[asad-ak]

The approach is in the same notebook link
https://colab.research.google.com/drive/1fDeXMOoek4E2i0C_CkAXNP8bozMbTknk?usp=sharing