Section 05 - code works on notebook but not as an imported module

[NoelTW]

Hi everyone,

This is from 05_pytorch_going_modular_exercise_template.ipynb.
I tried to make the function more handy by returning train_dir and test_dir variables.
Something went wrong with the following code:

# YOUR CODE HERE
%%writefile get_data.py
import os
import zipfile
import requests
from pathlib import Path

# Let's check if the directory has already existed

def get_data(file_url):
    # define pathes
    data_path = Path('data')
    image_path = data_path / file_url.split(r'/')[-1].split('.')[0]
    if image_path.is_dir():
        print(f'{str(image_path)} exists... skip dowload file...')
    else:
        print(f'{str(image_path)} does not exist, create a one')
        image_path.mkdir(parents=True,exist_ok=True)

        with open(data_path / file_url.split(r'/')[-1], 'wb') as f:
            response = requests.get(file_url)
            f.write(response.content)
            print('Zip file donwloaded...')
        
        with zipfile.ZipFile(data_path / file_url.split(r'/')[-1]) as zip_ref:
            zip_ref.extractall(image_path)
            print('Unzipping file...')
        os.remove(data_path / file_url.split(r'/')[-1])

    print('Your data is ready!')
    train_dir = image_path / 'train'
    test_dir = image_path / 'test'

    return train_dir, test_dir
    
if __name__ == '__main__':
    FILE_URL = 'https://github.com/mrdbourke/pytorch-deep-learning/raw/main/data/pizza_steak_sushi_20_percent.zip'
    train_dir, test_dir = get_data(FILE_URL)

This function worked well on a notebook, It can download the file and return train_dir and test_dir.
However, when I tried to use it as an import module, it can not return train_dir test_dir and showed the following error.
I have no idea why Path became None.

import get_data

FILE_URL = 'https://github.com/mrdbourke/pytorch-deep-learning/raw/main/data/pizza_steak_sushi_20_percent.zip'
train_dir, test_dir = get_data.get_data(FILE_URL)

data/pizza_steak_sushi_20_percent does not exist, create a one
Zip file donwloaded...
Unzipping file...
Your data is ready!
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
[<ipython-input-153-ccb7f4e31bb1>](https://localhost:8080/#) in <cell line: 4>()
      2 
      3 FILE_URL = 'https://github.com/mrdbourke/pytorch-deep-learning/raw/main/data/pizza_steak_sushi_20_percent.zip'
----> 4 train_dir, test_dir = get_data.get_data(FILE_URL)
      5 # train_dataloader, test_dataloader, class_names = get_data_loader(train_dir=train_dir,test_dir=test_dir,batch_size=BATCH_SIZE)

TypeError: cannot unpack non-iterable NoneType object

[mrdbourke]

Hi @NoelTW ,

I'm not 100% sure why your code isn't working on your end.

But I can confirm it works on my end.

Perhaps try to remove %%writefile get_data.py from the get_data.py script?

---

For reference, my files are:

• get_data.py

# YOUR CODE HERE
import os
import zipfile
import requests
from pathlib import Path

# Let's check if the directory has already existed

def get_data(file_url):
    # define pathes
    data_path = Path('data')
    image_path = data_path / file_url.split(r'/')[-1].split('.')[0]
    if image_path.is_dir():
        print(f'{str(image_path)} exists... skip dowload file...')
    else:
        print(f'{str(image_path)} does not exist, create a one')
        image_path.mkdir(parents=True,exist_ok=True)

        with open(data_path / file_url.split(r'/')[-1], 'wb') as f:
            response = requests.get(file_url)
            f.write(response.content)
            print('Zip file donwloaded...')
        
        with zipfile.ZipFile(data_path / file_url.split(r'/')[-1]) as zip_ref:
            zip_ref.extractall(image_path)
            print('Unzipping file...')
        os.remove(data_path / file_url.split(r'/')[-1])

    print('Your data is ready!')
    train_dir = image_path / 'train'
    test_dir = image_path / 'test'

    return train_dir, test_dir
    
if __name__ == '__main__':
    FILE_URL = 'https://github.com/mrdbourke/pytorch-deep-learning/raw/main/data/pizza_steak_sushi_20_percent.zip'
    train_dir, test_dir = get_data(FILE_URL)

• test_get_data.py

from get_data import get_data

FILE_URL = 'https://github.com/mrdbourke/pytorch-deep-learning/raw/main/data/pizza_steak_sushi_20_percent.zip'
train_dir, test_dir = get_data(file_url=FILE_URL)

Output:

[NoelTW]

Thank you @mrdbourke

I think I just figured out why my code didn't work.

Because once Colab imports the module, it keeps the first version and sets it as default.

It wouldn't replace overwritten version even when I tried to import again the newer version.

So I just solved this by restarting runtime.

A quick demo: