Dataloader:How to load dataset without meeting "DataLoader found invalid type: <class 'builtin_function_or_method'>"error?

[Kevin-shihello-world]

I want to build my dataset to train on PYG it seems every thing goes on well finally till it warns me like this and I think this may help us get deeper information on this question it may be just I used '[]' instead '()':

python
python
 TypeError                                 Traceback (most recent call last)
/tmp/ipykernel_92/3373408995.py in <cell line: 133>()
    135     _, train_acc = test(train_loader)
    136     val_loss, val_acc = test(val_loader)
--> 137     test_loss, test_acc = test(test_loader)
    138     print(f'Epoch: {epoch:03d}, Train Loss: {train_loss:.3f}, '
    139           f'Train Acc: {train_acc:.3f}, Val Loss: {val_loss:.3f}, '

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch/autograd/grad_mode.py in decorate_context(*args, **kwargs)
     25         def decorate_context(*args, **kwargs):
     26             with self.clone():
---> 27                 return func(*args, **kwargs)
     28         return cast(F, decorate_context)
     29 

/tmp/ipykernel_92/3373408995.py in test(loader)
    121     loss_all = 0
    122 
--> 123     for data in loader:
    124         data = data.to(device)
    125         pred, tot_loss, g_g, g_e = model(data.x, data.edge_index, data.batch)

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch/utils/data/dataloader.py in __next__(self)
    528             if self._sampler_iter is None:
    529                 self._reset()
--> 530             data = self._next_data()
    531             self._num_yielded += 1
    532             if self._dataset_kind == _DatasetKind.Iterable and \

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch/utils/data/dataloader.py in _next_data(self)
    568     def _next_data(self):
    569         index = self._next_index()  # may raise StopIteration
--> 570         data = self._dataset_fetcher.fetch(index)  # may raise StopIteration
    571         if self._pin_memory:
    572             data = _utils.pin_memory.pin_memory(data)

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py in fetch(self, possibly_batched_index)
     50         else:
     51             data = self.dataset[possibly_batched_index]
---> 52         return self.collate_fn(data)

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch_geometric/loader/dataloader.py in __call__(self, batch)
     32             return type(elem)(*(self(s) for s in zip(*batch)))
     33         elif isinstance(elem, Sequence) and not isinstance(elem, str):
---> 34             return [self(s) for s in zip(*batch)]
     35 
     36         raise TypeError(f'DataLoader found invalid type: {type(elem)}')

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch_geometric/loader/dataloader.py in <listcomp>(.0)
     32             return type(elem)(*(self(s) for s in zip(*batch)))
     33         elif isinstance(elem, Sequence) and not isinstance(elem, str):
---> 34             return [self(s) for s in zip(*batch)]
     35 
     36         raise TypeError(f'DataLoader found invalid type: {type(elem)}')

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch_geometric/loader/dataloader.py in __call__(self, batch)
     32             return type(elem)(*(self(s) for s in zip(*batch)))
     33         elif isinstance(elem, Sequence) and not isinstance(elem, str):
---> 34             return [self(s) for s in zip(*batch)]
     35 
     36         raise TypeError(f'DataLoader found invalid type: {type(elem)}')

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch_geometric/loader/dataloader.py in <listcomp>(.0)
     32             return type(elem)(*(self(s) for s in zip(*batch)))
     33         elif isinstance(elem, Sequence) and not isinstance(elem, str):
---> 34             return [self(s) for s in zip(*batch)]
     35 
     36         raise TypeError(f'DataLoader found invalid type: {type(elem)}')

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch_geometric/loader/dataloader.py in __call__(self, batch)
     34             return [self(s) for s in zip(*batch)]
     35 
---> 36         raise TypeError(f'DataLoader found invalid type: {type(elem)}')
     37 
     38     def collate(self, batch):  # Deprecated...

TypeError: DataLoader found invalid type: <class 'builtin_function_or_method'>

And this is my original code. The data stored in "/netp/test_1/' + str(i)+"_"+'my_file.npy'['net']['conv1.lin.weight']" are in shape [32 ,128].

pythonfrom typing import Optional, Callable, List
import os.path as osp
import torch
from torch_geometric.data import InMemoryDataset, download_url
from torch_geometric.io import read_planetoid_data

###假设原数据被以tensor形式保存为.npy格式
##tensor to txt:
##import scipy.io as io
##result1 = np.array(result1)
##np.savetxt('npresult1.txt',result1)

from torch_geometric.transforms import NormalizeFeatures
from torch_geometric.data import Data

def read_myds_data(raw_dir_0, raw_dir_1, raw_dir_2,i):
    ##now raw_dir_2 for my_file.npy
    
    ##for i in range(1,10):
    data=[]
    feats = torch.load('/netp/test_1/' + str(i)+"_"+'my_file.npy' )
    j = 0

    feats_n = torch.load(raw_dir_1 + str(i) + raw_dir_2)
    graphs = []
    list_1 = []
    list_2 = []

    list_3 = []
    list_4 = []
    x = torch.ones([32 ,128], dtype=torch.float)
    h = 33##h:number for nodes ;32 for node of conv1.lin_rel.bias while 
        ##0-31 for extracted features for classify and 33 for the first node for conv1.lin_rel.weight 
    h_r = []
   ## h_1 = 3*feats['conv1.lin_rel.weight'].shape[1]##h_1 for edges
    ###

    x = feats['net']['conv1.lin.weight'].t()
    x = x.cuda
    print(feats['net']['conv1.lin.weight'].t().shape[0])
    for i_1 in range(0, feats['net']['conv1.lin.weight'].t().shape[0]):
        ##for here feats[name_0].shape[1] would always be the same as feats[name_2].shape[1] I deal with this 2 in one for loop
        list_1.append(h + i_1)
        list_2.append(h - 1)
    edge_list = torch.Tensor([list_1, list_2]).type(torch.long)
    edge_list_ = torch.Tensor([list_1, list_2]).type(torch.long)
    if i == 1:
        edge_list = edge_list_
    else:
        edge_list = torch.cat((edge_list, edge_list_), dim = -1)
        print("1:", edge_list)
    batch = torch.zeros(12)
    if i == 1:
        batch = torch.zeros(feats['net']['conv1.lin.weight'].t().shape[0])
    else:
        batch_new = (i - 1)*torch.ones(feats['net']['conv1.lin.weight'].t().shape[0])
        batch = torch.cat((batch, batch_new), dim = 0).type(torch.long)##


    ###
    feats_L = torch.load('/netp/test_1_/' + str(i_1)+"_"+'my_file.npy')
    loss = feats_L['loss'] 
    y = loss
    data=Data(x=x,edge_index=edge_list,y = y) #包装成Data类
    #self.data, self.slices = split(data, batch)
    ##self.data, self.slices = self.collate(data)
    return Data(x=x,edge_index=edge_list,y = y)##, slices
class Myds(InMemoryDataset):

    ##url = 'https://github.com/kimiyoung/planetoid/raw/master/data'
    def __init__(self, root: str, name: str):
                 ##, split: str = "public",
                 ##num_train_per_class: int = 20, num_val: int = 500,
                 ##num_test: int = 1000, transform: Optional[Callable] = None,
                 ##pre_transform: Optional[Callable] = None):
        self.name = name

        super().__init__(root)##, transform, pre_transform = None, pre_filter = None) #等同于super(Planetoid,self).__init__(root, transform, pre_transform)
        self.data, self.slices = torch.load(self.processed_paths[0])#InMemoryDataset继承于Dataset类，processed_paths是Dataset类的属性，有self.processed_dir拼接得到
        ###
        self.data, self.slices = torch.load(self.processed_paths[0])#InMemoryDataset继承于Dataset类，processed_paths是Dataset类的属性，有self.processed_dir拼接得到
        
        ###
        '''
        # 将数据集划分为训练、验证和测试集
        self.split = split
        assert self.split in ['public', 'full', 'random']

        if split == 'full':
            data = self.get(0)
            data.train_mask.fill_(True)
            data.train_mask[data.val_mask | data.test_mask] = False
            self.data, self.slices = self.collate([data])

        elif split == 'random':
            data = self.get(0)
            data.train_mask.fill_(False)
            for c in range(self.num_classes):
                idx = (data.y == c).nonzero(as_tuple=False).view(-1)
                idx = idx[torch.randperm(idx.size(0))[:num_train_per_class]]
                data.train_mask[idx] = True

            remaining = (~data.train_mask).nonzero(as_tuple=False).view(-1)
            remaining = remaining[torch.randperm(remaining.size(0))]

            data.val_mask.fill_(False)
            data.val_mask[remaining[:num_val]] = True

            data.test_mask.fill_(False)
            data.test_mask[remaining[num_val:num_val + num_test]] = True

            self.data, self.slices = self.collate([data])
            '''

    @property
    def raw_dir(self) -> str:
        return '/netp/test_1/'

    @property
    def processed_dir(self) -> str:
        return osp.join(self.root, self.name, 'processed')

    @property
    def raw_file_names(self) -> List[str]:
        ##names = 
        ##return [f'ind.{self.name.lower()}.{name}' for name in names]
        return ['/netp/test_1/', 'netp/test_1_/', 'my_file.npy']
    @property
    def processed_file_names(self) -> str:
        return 'data.pt'
    ###
    @property
    def num_node_features(self) -> int:
        r"""Returns the number of features per node in the dataset."""
        data = self[0]
        # Do not fill cache for `InMemoryDataset`:
        if hasattr(self, '_data_list') and self._data_list is not None:
            self._data_list[0] = None
        data = data[0] if isinstance(data, tuple) else data
        if hasattr(data, 'num_node_features'):
            return data.num_node_features
        raise AttributeError(f"'{data.__class__.__name__}' object has no "
                             f"attribute 'num_node_features'")

    @property
    def num_features(self) -> int:
        r"""Returns the number of features per node in the dataset.
        Alias for :py:attr:`~num_node_features`."""
        return 128##self.num_node_features

    @property
    def num_edge_features(self) -> int:
        r"""Returns the number of features per edge in the dataset."""
        data = self[0]
        # Do not fill cache for `InMemoryDataset`:
        if hasattr(self, '_data_list') and self._data_list is not None:
            self._data_list[0] = None
        data = data[0] if isinstance(data, tuple) else data
        if hasattr(data, 'num_edge_features'):
            return data.num_edge_features
        raise AttributeError(f"'{data.__class__.__name__}' object has no "
                             f"attribute 'num_edge_features'")

    

    @property
    def num_classes(self) -> int:
        r"""Returns the number of classes in the dataset."""
        #y = torch.cat([data['y'] for data in self], dim=0)
        # Do not fill cache for `InMemoryDataset`:
        #if hasattr(self, '_data_list') and self._data_list is not None:
         #   self._data_list = self.len() * [None]
        return 5##self._infer_num_classes(y)##?
    '''@property
    def raw_paths(self) -> List[str]:
        r"""The absolute filepaths that must be present in order to skip
        downloading."""
        files = self.raw_file_names
        # Prevent a common source of error in which `file_names` are not
        # defined as a property.
        if isinstance(files, Callable):
            files = files()
        return [osp.join(self.raw_dir, f) for f in to_list(files)]'''
    

    
    
    def __repr__(self) -> str:
        arg_repr = str(len(self)) if len(self) > 1 else ''
        return f'{self.__class__.__name__}({arg_repr})'
    ###
    def download(self):
        pass

    def process(self):
        ###
        
        # Read data into huge `Data` list.
        data = Data
        data_list = []
        i = 0
        for i in range(1,10):
            data_list.append(read_myds_data('/netp/test_1/', 'netp/test_1_/', 'my_file.npy', i))#调用函数读取数据，包装成Data
        #data_list = list(data_list)
        #print("1:",data_list[0])
        self.data, self.slices = self.collate([data_list])
        data = self.data if self.pre_transform is None else self.pre_transform(data)
        slices = self.slices
        torch.save((data, slices), self.processed_paths[0])
        ###

    def __repr__(self) -> str:
        return f'{self.name}()'

And the model trained on this dataset is

python

###
import os.path as osp
from math import ceil

import torch
import torch.nn.functional as F
from torch.nn import Linear

from torch_geometric.datasets import TUDataset
from torch_geometric.loader import DataLoader
from torch_geometric.nn import DenseGraphConv, DMoNPooling, GCNConv
from torch_geometric.utils import to_dense_adj, to_dense_batch

path = osp.join(osp.dirname(osp.realpath('/netp/orginal ds')), '..', 'data', 'PROTEINS')
#dataset = TUDataset(path, name='PROTEINS').shuffle()
###
dataset = Myds("/netp", "try_")
dataset = dataset.shuffle()
n = len(dataset) // 10
test_dataset = dataset[:n]
train_dataset = dataset[n:]
test_loader = DataLoader(test_dataset, batch_size=10)
train_loader = DataLoader(train_dataset, batch_size=17)
###
###avg_num_nodes = int(dataset.data.x.size(0) / len(dataset))
avg_num_nodes = int(len(dataset[0][0]) / len(dataset))
n = (len(dataset) + 9) // 10
test_dataset = dataset[:n]
val_dataset = dataset[n:2 * n]
train_dataset = dataset[2 * n:]
test_loader = DataLoader(test_dataset, batch_size=10)
val_loader = DataLoader(val_dataset, batch_size=10)
train_loader = DataLoader(train_dataset, batch_size=10)


class Net(torch.nn.Module):
    def __init__(self, in_channels, out_channels, hidden_channels=32):
        super().__init__()
        
        self.conv1 = GCNConv(in_channels, hidden_channels)
        num_nodes = ceil(0.5 * avg_num_nodes)
        self.pool1 = DMoNPooling([hidden_channels, hidden_channels], num_nodes)

        self.conv2 = DenseGraphConv(hidden_channels, hidden_channels)
        num_nodes = ceil(0.5 * num_nodes)
        self.pool2 = DMoNPooling([hidden_channels, hidden_channels], num_nodes)

        self.conv3 = DenseGraphConv(hidden_channels, hidden_channels)

        self.lin1 = Linear(hidden_channels, hidden_channels)
        self.lin2 = Linear(hidden_channels, out_channels)

    def forward(self, x, edge_index, batch):
        x = self.conv1(x, edge_index).relu()

        x, mask = to_dense_batch(x, batch)
        adj = to_dense_adj(edge_index, batch)

        _, x, adj, sp1, o1, c1 = self.pool1(x, adj, mask)

        x = self.conv2(x, adj).relu()

        _, x, adj, sp2, o2, c2 = self.pool2(x, adj)

        x = self.conv3(x, adj)

        x = x.mean(dim=1)
        x = self.lin1(x).relu()
        x = self.lin2(x)
        g_g = x
        g_e = edge_index
        return F.log_softmax(x, dim=-1), sp1 + sp2 + o1 + o2 + c1 + c2, g_g, g_e


device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')
model = Net(dataset.num_features, dataset.num_classes).to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=0.001)


def train(train_loader):
    model.train()
    loss_all = 0
    i_1 = 0
    for data in train_loader:
        data = data.to(device)
        optimizer.zero_grad()
        out, tot_loss, g_g, g_e = model(data.x, data.edge_index, data.batch)
        loss = F.nll_loss(out, data.y.view(-1)) + tot_loss
        loss.backward()
        loss_all += data.y.size(0) * float(loss)
        ###
        a = torch.randn(1)
        str(i_1)
        if a < 0:
            
            dict_1 = {'net':model.state_dict(), 'optimizer':optimizer.state_dict(), 'epoch':epoch}
            store = '/netp/test_1/' + str(i_1)+'my_file.npy'          
            torch.save(dict_1, store)
            dict_g = {'g_g': g_g, 'g_e': g_e}
            torch.save(dict_g, store)
                
            optimizer.step() 
            
            i_1 = i_1 + 1   
            dict_2 = {'net':model.state_dict(), 'optimizer':optimizer.state_dict(), 'epoch':epoch}
            store = '/netp/test_1_/' + str(i_1)+'my_file.npy'          
            torch.save(dict_1, store)
            ##loss = loss.cpu()
            dict_3 = {'loss':loss}
            torch.save(loss, store)
            i = 0 
        i_1 = i_1 + 1
    return loss_all / len(train_dataset)


@torch.no_grad()
def test(loader):
    model.eval()
    correct = 0
    loss_all = 0

    for data in loader:
        data = data.to(device)
        pred, tot_loss, g_g, g_e = model(data.x, data.edge_index, data.batch)
        loss = F.nll_loss(pred, data.y.view(-1)) + tot_loss
        loss_all += data.y.size(0) * float(loss)
        correct += int(pred.max(dim=1)[1].eq(data.y.view(-1)).sum())

    return loss_all / len(loader.dataset), correct / len(loader.dataset)


for epoch in range(1, 10):
    train_loss = train(train_loader)
    _, train_acc = test(train_loader)
    val_loss, val_acc = test(val_loader)
    test_loss, test_acc = test(test_loader)
    print(f'Epoch: {epoch:03d}, Train Loss: {train_loss:.3f}, '
          f'Train Acc: {train_acc:.3f}, Val Loss: {val_loss:.3f}, '
          f'Val Acc: {val_acc:.3f}, Test Loss: {test_loss:.3f}, '
          f'Test Acc: {test_acc:.3f}')

I would appreciate it if anyone could offer me an answer. Thank you.

[rusty1s]

Do you have an example I can run on my end? I cannot spot any error in your code, this looks all good. Alternatively, can you try to find out at which element the DataLoader errors out? You can do that by manipulating

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch_geometric/loader/dataloader.py 

[Kevin-shihello-world]

Wow, that's much better I would mark this answer as the answer after you get acknowledged. It does no more show that warning again, thank you for what you have done to help.
And it now warns me as the following, I shall put that to the next question and can you tell me how would that be?

python
/tmp/ipykernel_136/1190763038.py in __init__(self, root, name)
     80 
     81         super().__init__(root)##, transform, pre_transform = None, pre_filter = None) #等同于super(Planetoid,self).__init__(root, transform, pre_transform)
---> 82         self.data, self.slices = torch.load(self.processed_paths[0])#InMemoryDataset继承于Dataset类，processed_paths是Dataset类的属性，有self.processed_dir拼接得到
     83         ###
     84         self.data, self.slices = torch.load(self.processed_paths[0])#InMemoryDataset继承于Dataset类，processed_paths是Dataset类的属性，有self.processed_dir拼接得到

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch/serialization.py in load(f, map_location, pickle_module, **pickle_load_args)
    710                     opened_file.seek(orig_position)
    711                     return torch.jit.load(opened_file)
--> 712                 return _load(opened_zipfile, map_location, pickle_module, **pickle_load_args)
    713         return _legacy_load(opened_file, map_location, pickle_module, **pickle_load_args)
    714 

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch/serialization.py in _load(zip_file, map_location, pickle_module, pickle_file, **pickle_load_args)
   1044     unpickler = UnpicklerWrapper(data_file, **pickle_load_args)
   1045     unpickler.persistent_load = persistent_load
-> 1046     result = unpickler.load()
   1047 
   1048     torch._utils._validate_loaded_sparse_tensors()

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch/serialization.py in persistent_load(saved_id)
   1014         if key not in loaded_storages:
   1015             nbytes = numel * torch._utils._element_size(dtype)
-> 1016             load_tensor(dtype, nbytes, key, _maybe_decode_ascii(location))
   1017 
   1018         return loaded_storages[key]

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch/serialization.py in load_tensor(dtype, numel, key, location)
    999         # stop wrapping with _TypedStorage
   1000         loaded_storages[key] = torch.storage._TypedStorage(
-> 1001             wrap_storage=restore_location(storage, location),
   1002             dtype=dtype)
   1003 

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch/serialization.py in default_restore_location(storage, location)
    174 def default_restore_location(storage, location):
    175     for _, _, fn in _package_registry:
--> 176         result = fn(storage, location)
    177         if result is not None:
    178             return result

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch/serialization.py in _cuda_deserialize(obj, location)
    156                 return storage_type(obj.nbytes())
    157         else:
--> 158             return obj.cuda(device)
    159 
    160 

~/miniconda3/envs/myconda/lib/python3.8/site-packages/torch/_utils.py in _cuda(self, device, non_blocking, **kwargs)
     77         else:
     78             new_type = getattr(torch.cuda, self.__class__.__name__)
---> 79             return new_type(self.size()).copy_(self, non_blocking)
     80 
     81 

RuntimeError: CUDA error: device-side assert triggered
CUDA kernel errors might be asynchronously reported at some other API call,so the stacktrace below might be incorrect.
For debugging consider passing CUDA_LAUNCH_BLOCKING=1.
​

[rusty1s]

Can you try to remove the device transfer (x = x.cuda()) all together?

[Kevin-shihello-world]

I've tried this one and it didn't work. So I tried to replace data in my read_x_data function to dataset[i] where the dataset is from tudataset and it didn't make a difference.
Then I thought I didn't try just loading data_list to the dataloader because I thought it seems it only allows data with little size to be trained in this way so I tried to load data in this way. I used data in my data_list from proteins in the tudataset and it works. The experiment with each data in my data list has twice the size of x and edge_index also works so it seems I should use data lists directly from the beginning.
But what is the reason for this difference? Why load in data list would do while loading it in the dataset would goes wrong? If you answer this question I would open a new question on the PYG and put your answer and your name on it, I put it here for you and other readers to know the context.
Best wishes.

[Kevin-shihello-world]

I've tried this one and it didn't work. So I tried to replace data in my read_x_data function to dataset[i] where the dataset is from tudataset and it didn't make a difference.
Then I thought I didn't try just loading data_list to the dataloader because I thought it seems it only allows data with little size to be trained in this way so I tried to load data in this way. I used data in my data_list from proteins in the tudataset and it works. The experiment with each data in my data list has twice the size of x and edge_index also works so it seems I should use data lists directly from the beginning. But what is the reason for this difference? Why load in data list would do while loading it in the dataset would goes wrong? If you answer this question I would open a new question on the PYG and put your answer and your name on it, I put this reversion here for you and other readers to know the context.
Best wishes.

[rusty1s]

I am not entirely sure, but just by looking at the error message above it looks to be a problem with saving and loading PyTorch CUDA tensors (which is the only difference between using on-the-fly created data_list and our Dataset implementation).