0.4 Delta

[dayland]

0.4-Delta Release Notes

What's New:

• Vector Hybrid Search which combines vector similarity with keyword matching to enhance search accuracy.
  • Added document processing pipeline steps to generate embeddings for text-based files. Bring your embedding (Azure Open AI or open-source embedding model).
  • Extended document processing pipeline with richer language detection and translation to avoid common error with OOTB Azure Cognitive Search skillsets
  • Switched to direct search index inserts instead of Azure Cognitive Search Indexer and skillsets
  • Restructured and added vector columns to Azure Cognitive Search Index (expanded JSON into index fields)
  • Update UX to embed users query and execute Vector Hybrid Search with Semantic
  • Added pipeline to process images and store enrichments as keywords in Azure Cognitive Search index which would allow user to do text to image search.
• Added iFrame document and image rendering of source material under citation panel of UX.
• Added support for several new file types using Unstructured.io
  • Text-based: pdf, docx, html, htm, csv, md, pptx, txt, json, xlsx, xml, eml, msg
  • Images: jpg, jpeg, png, gif, bmp, tif, tiff
• Added support for US Sovereign deployments
• Added filtered query support for Azure Cognitive Search index fields
  • Enabled upload to a folder and adding tags to uploaded file in UX
  • Enabled filtering search by "folder" and/or "tags" fields in Adjust panel in UX
• Added function testing of document pre-processing pipelines and embeddings REST APIs
• Added branding updates that allow a warning banner and UX title updates
• Enhanced infrastructure and application logging
  • Detailed chunk-based logging for embeddings and indexing
  • New Azure Workbook to help investigate infrastructure level errors (i.e. App Service not starting up correctly)

What's Changed

• Merge pull request 0.3-Gamma Release Candidate #204 from microsoft/vNext-Dev by @dayland in Merge pull request #204 from microsoft/vNext-Dev #208
• hot fix for missing subtitle in non-pdf by @georearl in hot fix for missing subtitle in non-pdf #210
• Forward integrate HF for non-PDF missing subtitle attribute by @dayland in Forward integrate HF for non-PDF missing subtitle attribute #214
• Update README.md for model version support by @ArpitaisAn0maly in Update README.md for model version support #216
• Geearl/5974 acr bicep by @georearl in Geearl/5974 acr bicep #199
• Gov init by @rohrerb in Gov init #212
• Model check Fix by @rohrerb in Model check Fix #223
• Bash variable if check repair by @rohrerb in Bash variable if check repair #224
• Sovereign Setup Instructions by @rohrerb in Sovereign Setup Instructions #222
• Geearl/5977 container app service by @georearl in Geearl/5977 container app service #205
• Temperature setting by @asbanger in Temperature setting  #215
• udpated default model to gpt-35-turbo-16k by @dayland in udpated default model to gpt-35-turbo-16k #226
• Geearl/5982 appservice debug 2 by @georearl in Geearl/5982 appservice debug 2 #217
• Fix for Deployment Name by @rohrerb in Fix for Deployment Name #229
• Parameter for environment so we can share the vNext.yml by @rohrerb in Parameter for environment so we can share the vNext.yml #228
• Gov Automation Fix. by @rohrerb in Gov Automation Fix. #232
• Some more az login fixes for automation. by @rohrerb in Some more az login fixes for automation.  #233
• Temp Build Fix for Gov by @rohrerb in Temp Build Fix for Gov #234
• Geearl/5972 generate embeddings by @georearl in Geearl/5972 generate embeddings #230
• 6012 Add utils method for extracting html charset else default to utf8 by @ryonsteele in 6012 Add utils method for extracting html charset else default to utf8 #239
• brrohrer/6010 enrichment func gov by @rohrerb in brrohrer/6010 enrichment func gov #237
• AAD Regression Fix by @rohrerb in AAD Regression Fix #240
• Lwilk/5892 image func by @lmwilki in Lwilk/5892 image func #227
• Geearl/6053 large files missing chunks by @georearl in Geearl/6053 large files missing chunks #242
• 5993 Change how page numbers are collected and stored for chunks by @ryonsteele in 5993 Change how page numbers are collected and stored for chunks #243
• requeue logic with max retries by @georearl in requeue logic with max retries #241
• Geearl/6055 missing index key values by @georearl in Geearl/6055 missing index key values #244
• Fix passing of api variable to function by @dayland in Fix passing of api variable to function #245
• brrohrer/6050-GovIngestPipeline by @rohrerb in brrohrer/6050-GovIngestPipeline #247
• Move installation into startup entrypoint script to reduce build by @ryonsteele in Move installation into startup entrypoint script to reduce build #246
• Ryonsteele/6063 default responselength change by @ryonsteele in Ryonsteele/6063 default responselength change #249
• Add build-containers to pipelines by @dayland in Add build-containers to pipelines #250
• updates to process flow diagram for embeddings by @dayland in updates to process flow diagram for embeddings #248
• Aparmar/5661 hybrid search by @ArpitaisAn0maly in Aparmar/5661 hybrid search #255
• 6105/brrohrer stack trace logging by @rohrerb in 6105/brrohrer stack trace logging #253
• Geearl/6070 non pdf by @georearl in Geearl/6070 non pdf #256
• Ryonsteele/int test by @ryonsteele in Ryonsteele/int test #254
• Geearl/6112 missing chunk index by @georearl in Geearl/6112 missing chunk index #257
• Fix embeddings mismatch between pipeline and REST APi by @dayland in Fix embeddings mismatch between pipeline and REST APi #258
• Improvements to functional tests by @dayland in Improvements to functional tests #260
• Branding and UX updates by @dayland in Branding and UX updates #261
• Merging 0.3-Gamma hotfixes into vNext-Dev by @dayland in Merging 0.3-Gamma hotfixes into vNext-Dev #264
• Fix to have non-pdf files honor the DEV_CODE flags by @dayland in Fix to have non-pdf files honor the DEV_CODE flags #265
• Add image enrichment to index for image function by @ryonsteele in Add image enrichment to index for image function #259
• Add more test case files and types by @ryonsteele in Add more test case files and types #266
• Extended logging to UX for uploaded files by @dayland in Extended logging to UX for uploaded files #267
• Enable citations to use new vector index fields by @dayland in Enable citations to use new vector index fields #268
• Ryonsteele/6108 cicd functional by @ryonsteele in Ryonsteele/6108 cicd functional #272
• Aparmar/6151 enable hybrid semantic search by @ArpitaisAn0maly in Aparmar/6151 enable hybrid semantic search #275
• Updates for Sprint 12 arch changes by @dayland in Updates for Sprint 12 arch changes #273
• Ryonsteele/6115 api tests by @ryonsteele in Ryonsteele/6115 api tests #277
• Update to use model name over deployment name by @dayland in Update to use model name over deployment name #279
• Fix to put calculated chunk name in index for images by @dayland in Fix to put calculated chunk name in index for images #280
• Create seperate pipeline ymls for gov and com by @ryonsteele in Create seperate pipeline ymls for gov and com #283
• Updating the GPT-35-Turbo-16K capacity to default by @asbanger in Updating the GPT-35-Turbo-16K capacity to default #284
• Fix for AOAI BICEP deployment by @dayland in Fix for AOAI BICEP deployment #287
• Fixing the GitHub broken Links by @asbanger in Fixing the GitHub broken Links #281
• Ryonsteele/indexing statuslog by @ryonsteele in Ryonsteele/indexing statuslog #289
• Change to create seperate func ASP and resolve enrichment concurrency by @ryonsteele in Change to create seperate func ASP and resolve enrichment concurrency #293
• Geearl/6184 function timeout by @georearl in Geearl/6184 function timeout #294
• ReadMe update for Image Search for Delta release. by @ArpitaisAn0maly in ReadMe update for Image Search for Delta release. #296
• Prompt to delete index on deploy if vector size changes by @ryonsteele in Prompt to delete index on deploy if vector size changes #297
• Vector Search Doc Update to ReadMe by @ArpitaisAn0maly in Vector Search Doc Update to ReadMe #300
• Update preprocessing feature document for unstructured and image by @ryonsteele in Update preprocessing feature document for unstructured and image #298
• Remove stack from upload statuslog by @ryonsteele in Remove stack from upload statuslog #301
• Geearl/6130 azure caps by @georearl in Geearl/6130 azure caps #303
• Fix env variable mispelling by @ryonsteele in Fix env variable mispelling #304
• Dabiscup/applicationtitle2 by @danimal521 in Dabiscup/applicationtitle2 #302
• Resolve startup failure with aoai model usage by @ryonsteele in Resolve startup failure with aoai model usage #307
• brrohrer/6199 - Unstructured JSON Support by @rohrerb in brrohrer/6199 - Unstructured JSON Support #306
• fixes for email processing and minor document updates by @georearl in fixes for email processing and minor document updates #309
• Ryonsteele/6200 fileupload bugfix by @ryonsteele in Ryonsteele/6200 fileupload bugfix #312
• Ryonsteele/6198 workbook infra by @ryonsteele in Ryonsteele/6198 workbook infra #314
• Updaing local.env.example for clarity on embedding models by @ArpitaisAn0maly in Updaing local.env.example for clarity on embedding models #315
• Add new util method for getting chunk path by @ryonsteele in Add new util method for getting chunk path #313
• Logging updates by @ryonsteele in Logging updates #317
• Remove doc and ppt from the list of doctypes in diagram by @ryonsteele in Remove doc and ppt from the list of doctypes in diagram #319
• Add concurrency for backend api and embed api error handling by @ryonsteele in Add concurrency for backend api and embed api error handling #320
• Added UX for folders and tags by @dayland in Added UX for folders and tags #288
• Ltierney/docupdates by @lon-tierney in Ltierney/docupdates #321
• Update local.env.example to match deployment guide for required fields by @dayland in Update local.env.example to match deployment guide for required fields #324
• Vector query fix applied for non semantic hybrid by @ArpitaisAn0maly in Vector query fix applied for non semantic hybrid #325
• Ryonsteele/6218 fix folder tabbing issue by @ryonsteele in Ryonsteele/6218 fix folder tabbing issue #326
• Ryonsteele/6220 lock enrichment versions by @ryonsteele in Ryonsteele/6220 lock enrichment versions #327
• Add known issue for jq params issue to check config and rerun by @ryonsteele in Add known issue for jq params issue to check config and rerun #329
• Update to embedding Model parameter Name to reflect Deployment Name by @ArpitaisAn0maly in Update to embedding Model parameter Name to reflect Deployment Name #330
• Aparmar/delta bm25similarity by @ArpitaisAn0maly in Aparmar/delta bm25similarity #333
• Adding new enrichments to keyword search by @dayland in Adding new enrichments to keyword search #334
• 0.4-Delta Release by @dayland in 0.4-Delta Release #336

New Contributors

• @rohrerb made their first contribution in Gov init #212
• @ryonsteele made their first contribution in 6012 Add utils method for extracting html charset else default to utf8 #239
• @danimal521 made their first contribution in Dabiscup/applicationtitle2 #302

Full Changelog: v0.3-Gamma...v0.4-Delta

---

This discussion was created from the release 0.4 Delta.