Verify and fix 'import' issues in notebooks.

[furniturewalatkNIH]

We need to have an automated way to verify the notebooks.

One part of this is the verification of 'imports' in them.

Need to create a workflow/job to run through all notebooks and check/verify each.

[furniturewalatkNIH]

Created new workflow - https://github.com/STRIDES/NIHCloudLabAzure/actions/workflows/verify-imports-jupyter-nbmake.yml

[furniturewalatkNIH]

`============================================================= FAILURES =============================================================
_____________ /workspaces/NIHCloudLabAzure/notebooks/GenAI/notebooks/AzureAIStudio_index_structured_with_console.ipynb _____________
[gw0] linux -- Python 3.10.13 /usr/local/python/3.10.13/bin/python3

#connect to vector store
from azure.search.documents import SearchClient
from azure.core.credentials import AzureKeyCredential

search_client = SearchClient(endpoint, index_name, AzureKeyCredential(index_key))

ModuleNotFoundError Traceback (most recent call last)
Cell In[4], line 2
1 #connect to vector store
----> 2 from azure.search.documents import SearchClient
3 from azure.core.credentials import AzureKeyCredential
5 search_client = SearchClient(endpoint, index_name, AzureKeyCredential(index_key))

ModuleNotFoundError: No module named 'azure'
_______________________ /workspaces/NIHCloudLabAzure/notebooks/GenAI/notebooks/AzureOpenAI_embeddings.ipynb ________________________
[gw0] linux -- Python 3.10.13 /usr/local/python/3.10.13/bin/python3

import os
from openai import AzureOpenAI
import dotenv
import requests
import numpy as np
import pandas as pd

ModuleNotFoundError Traceback (most recent call last)
Cell In[2], line 3
1 import os
2 from openai import AzureOpenAI
----> 3 import dotenv
4 import requests
5 import numpy as np

ModuleNotFoundError: No module named 'dotenv'
_________________________ /workspaces/NIHCloudLabAzure/notebooks/GenAI/notebooks/Pubmed_RAG_chatbot.ipynb __________________________
[gw0] linux -- Python 3.10.13 /usr/local/python/3.10.13/bin/python3

create your SAS token

from datetime import datetime, timedelta
from azure.storage.blob import BlobServiceClient, generate_account_sas, ResourceTypes, AccountSasPermissions
start_time = datetime.utcnow()
expiry_time = start_time + timedelta(hours=2)
sas_token = generate_account_sas(
account_name=storage_account_name,
container_name=container_name,
account_key=key,
resource_types=ResourceTypes(object=True),
permission=AccountSasPermissions(read=True, write=True, delete=True, list=True, add=True, create=True),
expiry=expiry_time,
start=start_time
)

ModuleNotFoundError Traceback (most recent call last)
Cell In[10], line 3
1 # create your SAS token
2 from datetime import datetime, timedelta
----> 3 from azure.storage.blob import BlobServiceClient, generate_account_sas, ResourceTypes, AccountSasPermissions
4 start_time = datetime.utcnow()
5 expiry_time = start_time + timedelta(hours=2)

ModuleNotFoundError: No module named 'azure.storage'
______________________ /workspaces/NIHCloudLabAzure/notebooks/GenAI/notebooks/AzureAIStudio_sql_chatbot.ipynb ______________________
[gw0] linux -- Python 3.10.13 /usr/local/python/3.10.13/bin/python3

import pyodbc

server_name = "<ENTER_DB_SERVER_NAME>"
username = "<ENTER_DB_USERNAME>"
password = "<ENTER_DB_PASSWORD>"
database = "<ENTER_DB_NAME>"
driver= '{ODBC Driver 18 for SQL Server}'

conn = pyodbc.connect(f'DRIVER={driver};PORT=1433;SERVER={server_name}.database.windows.net;PORT=1443;DATABASE={database};UID={username};PWD={password}')

ModuleNotFoundError Traceback (most recent call last)
Cell In[7], line 1
----> 1 import pyodbc
3 server_name = "<ENTER_DB_SERVER_NAME>"
4 username = "<ENTER_DB_USERNAME>"

ModuleNotFoundError: No module named 'pyodbc'
_____________________________ /workspaces/NIHCloudLabAzure/notebooks/pangolin/pangolin_pipeline.ipynb ______________________________
[gw0] linux -- Python 3.10.13 /usr/local/python/3.10.13/bin/python3

#import libraries
import os
from Bio import SeqIO
from Bio import Entrez
import ipyrad as ipa
import toytree

ModuleNotFoundError Traceback (most recent call last)
Cell In[7], line 5
3 from Bio import SeqIO
4 from Bio import Entrez
----> 5 import ipyrad as ipa
6 import toytree

ModuleNotFoundError: No module named 'ipyrad'
========================================================= warnings summary =========================================================
notebooks/GenAI/notebooks/sharepoint_RAG_bot.ipynb::
/home/codespace/.local/lib/python3.10/site-packages/nbformat/init.py:96: MissingIDFieldWarning: Cell is missing an id field, this will become a hard error in future nbformat versions. You may want to use normalize() on your notebooks before validations (available since nbformat 5.1.4). Previous versions of nbformat are fixing this issue transparently, and will stop doing so in the future.
validate(nb)

../../usr/local/python/3.10.13/lib/python3.10/site-packages/_pytest/cacheprovider.py:461
/usr/local/python/3.10.13/lib/python3.10/site-packages/_pytest/cacheprovider.py:461: PytestCacheWarning: could not create cache path /workspaces/NIHCloudLabAzure/.pytest_cache/v/cache/nodeids: [Errno 28] No space left on device: '/workspaces/NIHCloudLabAzure/pytest-cache-files-tklqbmxd'
config.cache.set("cache/nodeids", sorted(self.cached_nodeids))

../../usr/local/python/3.10.13/lib/python3.10/site-packages/_pytest/cacheprovider.py:413
/usr/local/python/3.10.13/lib/python3.10/site-packages/_pytest/cacheprovider.py:413: PytestCacheWarning: could not create cache path /workspaces/NIHCloudLabAzure/.pytest_cache/v/cache/lastfailed: [Errno 28] No space left on device: '/workspaces/NIHCloudLabAzure/pytest-cache-files-b91m919k'
config.cache.set("cache/lastfailed", self.lastfailed)

../../usr/local/python/3.10.13/lib/python3.10/site-packages/_pytest/stepwise.py:57
/usr/local/python/3.10.13/lib/python3.10/site-packages/_pytest/stepwise.py:57: PytestCacheWarning: could not create cache path /workspaces/NIHCloudLabAzure/.pytest_cache/v/cache/stepwise: [Errno 28] No space left on device: '/workspaces/NIHCloudLabAzure/pytest-cache-files-zp6jr5yu'
session.config.cache.set(STEPWISE_CACHE_DIR, [])

-- Docs: https://docs.pytest.org/en/stable/how-to/capture-warnings.html

Learn more about nbmake at https://github.com/treebeardtech/nbmake

===================================================== short test summary info ======================================================
FAILED notebooks/GenAI/notebooks/AzureAIStudio_index_structured_with_console.ipynb::
FAILED notebooks/GenAI/notebooks/AzureOpenAI_embeddings.ipynb::
FAILED notebooks/GenAI/notebooks/Pubmed_RAG_chatbot.ipynb::
FAILED notebooks/GenAI/notebooks/AzureAIStudio_sql_chatbot.ipynb::
FAILED notebooks/pangolin/pangolin_pipeline.ipynb::
======================================= 5 failed, 6 passed, 4 warnings in 759.42s (0:12:39) ========================================`