Book-Crossing for Link Prediction

[DPalmz]

I have spent probably more time than I should trying to get this book crossing data bundled up into a Data object so I can start processing it. I've got an edge index and a label index (probably). the problem I'm running into is when I'm encoding the data from the columns for the node features, they become mismatched lengths, which makes tensorizing them very difficult.

from transformers import AutoTokenizer

books =  pd.read_csv("Book-Crossing\BX-Books.csv", encoding='ISO-8859-1', sep=";", on_bad_lines='skip', usecols=['ISBN',	'Book-Title', 'Book-Author', 'Year-Of-Publication',	'Publisher'])
users = pd.read_csv("Book-Crossing\BX-Users.csv", encoding='ISO-8859-1', sep=";", on_bad_lines='skip')
reviews = pd.read_csv("Book-Crossing\BX-Book-Ratings.csv", encoding='ISO-8859-1', sep=";", on_bad_lines='skip')

tokens_df = pd.DataFrame(books)
for i in range(0, len(users.columns)):
    tokens_df[users.columns] = users[users.columns]
tokens = [0] * 271379
tn = torch.empty((6))
tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased", use_fast=True)

for j in range(0, len(tokens_df.columns)):
    tokens_df[tokens_df.columns[j]] = tokens_df[tokens_df.columns[j]].apply(str)
    for i in range(0, 271378):
        tokens[i] = tokenizer.tokenize(tokens_df.loc[i].iat[j])
        
        tokens[i] = torch.tensor(tokenizer.convert_tokens_to_ids(tokens[i] )) 
            
    for i in range (i, 271378): #some of the columns are a few datapoints longer than others, this didn't work though
        
        tokens[i] = torch.empty((6))
    tn = torch.stack(tokens)
    tokens_df[tokens_df.columns[j]] = tn
    
tokens_df.head()

The Raw data is below
Book-Crossing.zip

[rusty1s]

How about:

    for i in range(0, 271378):
        tokens[i] = tokenizer.tokenize(tokens_df.loc[i].iat[j])
        out = torch.tensor(tokenizer.convert_tokens_to_ids(tokens[i]))
        tokens[i] = torch.zeros((9, ), dtype=torch.long)
        tokens[i][:out.size(0)] = out
        print(i, out.shape, tokens[i])

    tn = torch.stack(tokens)

[DPalmz]

That almost works. Your suggestion to add the torch.zeros to the main loop was definitely a good one, thanks. For some reason I'm getting the problem "RuntimeError: The expanded size of the tensor (100) must match the existing size (105) at non-singleton dimension 0. Target sizes: [100]. Tensor sizes: [105]" This data in no way has that many encodings. The only explanation I can think of is that it's taking the torch.zeros size and adding the size of the encoding (5) to it as opposed to assigning it?
I just checked, it happens exactly as j goes from 0 to 1, or as the dataframe switches columns.