Function app FileLayoutParsingOther function: Failed to download 'punkt' package

[zzesdc]

Hi Tech team,
I am deploying a secure mode deployment using v1.2. After I uploaded a doc file, it stuck at "FileUploadedFunc - pdf file sent to submit queue. Visible in 52 seconds". When I check the invocations for "FileLayoutParsingOther" section in function app, it shows the error:

I did search for the solution of the same issues. I found adding both "nltk.download('punkt')
nltk.download('punkt_tab')" under utilities.py file. But still no luck to download punkt.

I tried to remove the private endpoint and open the function app inbound traffic to the public but keeping vnet integration. Still no luck.

Thanks for help!

[ruandersMSFT]

I have also experienced this issue while deploying on an internal private network. Our temporarily workaround was to ourselves download and extract the nltk files into the Azure SMB File Share of the Storage Account used by the Function Application. Specifically, we created a nltk_data folder at the root of the SMB File Share, and within nltk_data we have a taggers and tokenizers folder. The tokenizers folder has a punkt and punkt_tab folder in which the extracted files are uploaded. The taggers folder has a averaged_perceptron_tagger folder in which the extracted files of averaged_perceptron_tagger are uploaded. With these files added, the function app was able to run in the private environment.

The code within the solution is attempting to download punkt packages via the nltk.download call and then extract into the folders. I haven't had time to troubleshoot a root cause fix yet in the environment, but similar to our environment you may be behind a firewall or proxy to get to the Internet which also possibly includes the use of internal certificate authority certificates via the proxy. Our environment passes through a proxy that has internal certificates, and I think a root cause in our environment is that the download is not occurring because it can't reach the site due to failed SSL validation of the internal SSL certificates issued by the internal proxy.

Recommend referencing the Installing NLTK Data page for manual installation instructions which equates to the folder creation above. Manual Downloads for extraction:

• averaged_perceptron_tagger
• punkt
• punkt_tab

In order to troubleshoot a root cause, which is likely firewall, proxy or SSL Certificate trust related, access the Kudu App for the function application where you can access the SSH and Bash consoles. Try to do some command such as curl to download the zipped packages above. Verify network routing is working and that the download URLs are reachable. If you identify that there is an SSL Certificate chain trust issue with intermediate certificates, consider if the certificates are added to the Function Application and imported via WEBSITE_LOAD_CERTIFICATES. Because the nltk library download attempt may also be failing ssl validation, consider if addition of ssl create_default_context() with cafile or capath are needed to trust ssl certificates in the application code.

[olafwrieden]

I ran into this issue during a series of deployments recently, it seems to be caused by networking policies of the Azure tenant which prevent the Function App from downloading the punk library at the top of the functions/shared_code/utilities.py file.

The solution proposed above did not end up resolving this problem and presented some "This is a legacy setting"-type of errors on the Function App's App Settings in the Azure Portal. We have identified another workaround that I would like to share for anyone else running into this issue with the current version of PubSec Info Assistant.

Step-by-Step Guidance

1. Download the punkt.zip and punkt_tab.zip manually.
2. Add these two zip files into the functions/shared_code directory as is (no need to unzip). For simplicity we have simply checked this into source control and made these two files part of our codebase but keep in mind that the NLTK still evolves independently and a new version of these two zip files may become available. When this happens, we need to manually download them and replace the one in our codebase with the new version (hopefully not too often).
3. Now that the two dependencies can be referenced locally, we can comment out the line in the functions/shared_code/utilities.py file that attempts to download it:

   import tiktoken
   import nltk
   # Try to download using nltk.download
   # nltk.download('punkt')  <== Note we have commented this line out
   from bs4 import BeautifulSoup
   ...

4. The last part is to reference the zip files from the local directory. For this we made a few additions to the same utilities.py file. In short, we specify where the two zip files are located, we specify a target directory where we want to unzip them to, if those target directories do not yet exist, we create them, otherwise we extract the zips into the respective directories. This guidance came directly from the NLTK manual installation guidance. The first part of our utilities.py now looks like this:

   import tiktoken
   import nltk
   # Try to download using nltk.download
   # nltk.download('punkt')  <== Note we have commented this line out
   from bs4 import BeautifulSoup
   
   # Define directories
   script_dir = os.path.dirname(os.path.abspath(__file__))
   nltk.data.path.insert(0, script_dir)  # Add script_dir as the first directory to search
   
   # Define zip file paths
   local_punkt_zip = os.path.join(script_dir, 'punkt.zip')
   local_punkt_tab_zip = os.path.join(script_dir, 'punkt_tab.zip')
   
   # Define target directories for both default NLTK location and script directory
   nltk_data_dir = nltk.data.path[-1]  # Default NLTK data directory
   target_dirs = [
       # Default NLTK locations
       {
           'tokenizers_dir': os.path.join(nltk_data_dir, 'tokenizers'),
           'punkt_dir': os.path.join(nltk_data_dir, 'tokenizers/punkt'),
           'punkt_tab_dir': os.path.join(nltk_data_dir, 'tokenizers/punkt_tab')
       },
       # Local script directory locations
       {
           'tokenizers_dir': os.path.join(script_dir, 'tokenizers'),
           'punkt_dir': os.path.join(script_dir, 'tokenizers/punkt'),
           'punkt_tab_dir': os.path.join(script_dir, 'tokenizers/punkt_tab')
       }
   ]
   
   # Function to extract zip files
   def extract_nltk_data(zip_path, target_dir):
       if os.path.exists(zip_path):
           with zipfile.ZipFile(zip_path, 'r') as zip_ref:
               zip_ref.extractall(target_dir)
           return True
       return False
   
   # Create directories and extract data for both locations
   for dirs in target_dirs:
       # Create tokenizers directory
       os.makedirs(dirs['tokenizers_dir'], exist_ok=True)
       
       # Extract punkt.zip
       if not os.path.exists(dirs['punkt_dir']):
           if not extract_nltk_data(local_punkt_zip, dirs['tokenizers_dir']):
               logging.warning(f"Could not find punkt.zip in the script directory: {script_dir}")
       
       # Extract punkt_tab.zip
       if not os.path.exists(dirs['punkt_tab_dir']):
           if not extract_nltk_data(local_punkt_tab_zip, dirs['tokenizers_dir']):
               logging.warning(f"Could not find punkt_tab.zip in the script directory: {script_dir}")
   
   # The rest of the file utilities.py continues here... removed for brevity...

5. Be sure to keep the rest of the utilities.py file contents (about line 68 onwards) in place following this modification to the top of the file.

After redeploying the code using the documented make deploy command from the root directory, the documents can now be indexed properly and the "Failed to download 'punkt' package" error is circumvented by referencing the local punkt and punkt_tab directory instead of downloading it on-demand.