prepdocs.py : getting error while running the file.

[namratam30]

Please provide us with the following information:

I am getting below error after updating the open ai in the solution -
Traceback (most recent call last):
2023-10-13T13:55:21.3332834Z File "/home/vsts/work/1/s/scripts/.venv/lib/python3.9/site-packages/tenacity/init.py", line 382, in call
2023-10-13T13:55:21.3333904Z result = fn(*args, **kwargs)
2023-10-13T13:55:21.3334574Z File "/home/vsts/work/1/s/./scripts/prepdocs.py", line 234, in compute_embedding
2023-10-13T13:55:21.3335315Z refresh_openai_token()
2023-10-13T13:55:21.3335943Z File "/home/vsts/work/1/s/./scripts/prepdocs.py", line 312, in refresh_openai_token
2023-10-13T13:55:21.3337486Z if open_ai_token_cache[CACHE_KEY_TOKEN_TYPE] == 'azure' and open_ai_token_cache[CACHE_KEY_CREATED_TIME] + 300 < time.time():
2023-10-13T13:55:21.3338556Z KeyError: 'token_type'
2023-10-13T13:55:21.3338714Z
2023-10-13T13:55:21.3339098Z The above exception was the direct cause of the following exception:
2023-10-13T13:55:21.3339605Z
2023-10-13T13:55:21.3339794Z Traceback (most recent call last):
2023-10-13T13:55:21.3340399Z File "/home/vsts/work/1/s/./scripts/prepdocs.py", line 395, in
2023-10-13T13:55:21.3341154Z index_sections(os.path.basename(filename), sections)
2023-10-13T13:55:21.3341935Z File "/home/vsts/work/1/s/./scripts/prepdocs.py", line 281, in index_sections
2023-10-13T13:55:21.3342639Z for s in sections:
2023-10-13T13:55:21.3343211Z File "/home/vsts/work/1/s/./scripts/prepdocs.py", line 226, in create_sections
2023-10-13T13:55:21.3343998Z section["embedding"] = compute_embedding(content)
2023-10-13T13:55:21.3345299Z File "/home/vsts/work/1/s/scripts/.venv/lib/python3.9/site-packages/tenacity/init.py", line 289, in wrapped_f
2023-10-13T13:55:21.3346383Z return self(f, *args, **kw)
2023-10-13T13:55:21.3347516Z File "/home/vsts/work/1/s/scripts/.venv/lib/python3.9/site-packages/tenacity/init.py", line 379, in call
2023-10-13T13:55:21.3348620Z do = self.iter(retry_state=retry_state)
2023-10-13T13:55:21.3349802Z File "/home/vsts/work/1/s/scripts/.venv/lib/python3.9/site-packages/tenacity/init.py", line 326, in iter
2023-10-13T13:55:21.3350837Z raise retry_exc from fut.exception()
2023-10-13T13:55:21.3351614Z tenacity.RetryError: RetryError[<Future at 0x7fd0b3c4ce50 state=finished raised KeyError>]
2023-10-13T13:55:21.4608265Z ##[error]Bash exited with code '1'.
2023-10-13T13:55:21.4622691Z ##[section]Finishing: Process Documents

---

This issue is for a: (mark with an x)

- [ ] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)

Minimal steps to reproduce

Any log messages given by the failure

Expected/desired behavior

OS and Version?

Windows 7, 8 or 10. Linux (which distribution). macOS (Yosemite? El Capitan? Sierra?)

azd version?

run azd version and copy paste here.

Versions

Mention any other details that might be useful

---

Thanks! We'll be in touch soon.

[pamelafox]

That should have been set here:

azure-search-openai-demo/scripts/prepdocs.py

Line 737 in ca09627

if not args.openaikey:

What kind of OpenAI authentication are you using? Azure key, Azure AD token, or OpenAI key?

[namratam30]

I am using open ai key

[pamelafox]

So you're using openai.com with key, not Azure OpenAI with key?

If so, please make sure you have this fix that passed in all the required OpenAI related parameters:
#796

I think you may be missing openaihost

[namratam30]

I am confused between OpenAI Orgnization and openaihost. Could you please help me how to pass these details-
OPENAI_API_TYPE = "azure"
OPENAI_API_VERSION = "2023-07-01-preview"
OPENAI_API_KEY = "xxxxxxx"
OPENAI_API_BASE = " https://xxxxxx.openai.azure.com/"

[pamelafox]

If you are using an openai.com OpenAI account, then your azd environment variables should look like:

OPENAI_HOST="openai"
OPENAI_API_KEY="key"
OPENAI_ORGANIZATION="org-on-openai-com" (if you have one, you may not)

If you are using an Azure OpenAI account, then you should be logged in locally to your Azure account and not passing any sort of key.

In that case, your .env will look like:

OPENAI_HOST="azure"
OPENAI_ORGANIZATION=""
OPENAI_API_KEY=""
AZURE_OPENAI_SERVICE="cog-j25rgqsibtmlo"
AZURE_OPENAI_EMB_DEPLOYMENT="embedding"
AZURE_OPENAI_EMB_MODEL_NAME="text-embedding-ada-002"
...

[github-actions]

This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this issue will be closed.

[pamelafox]

Thanks for the report and details. The KeyError you hit was in the older scripts/prepdocs.py. The ingestion flow has since moved to app/backend/prepdocs.py and now uses OPENAI_HOST. For OpenAI.com, follow the docs here:
https://github.com/Azure-Samples/azure-search-openai-demo/blob/main/docs/deploy_existing.md#openaicom-openai

I’m going to close this as obsolete. If you can still reproduce on the latest main, please open a new issue with your commit SHA, OPENAI_HOST value, relevant env vars (redacted), and the full stack trace, and I’ll take another look.