diff --git a/docs/cascading-retrieval.ipynb b/docs/cascading-retrieval.ipynb new file mode 100644 index 00000000..251ebfc8 --- /dev/null +++ b/docs/cascading-retrieval.ipynb @@ -0,0 +1,829 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "y4Bo2bsIzS9H" + }, + "source": [ + "# Cascading Retrieval (aka hybrid search)\n", + "\n", + "In this notebook, you'll learn how to use Pinecone to combine the benefits of semantic and lexical search using a technique called cascading retrieval or [hybrid search with two indexes](https://docs.pinecone.io/guides/search/hybrid-search).\n", + "\n", + "Semantic search and lexical search are powerful information retrieval techniques, but each has notable limitations. For example, semantic search can miss results based on exact keyword matches, especially in scenarios involving domain-specific terminology. While lexical search can miss results based on relationships, such as synonyms and paraphrases.\n", + "\n", + "When dense and sparse retrieval are combined along with reranking — an approach we call cascading retrieval — you get up to 48% better performance than either sparse or dense alone.\n", + "\n", + "You can read more about how cascading retrieval works [here](https://www.pinecone.io/blog/cascading-retrieval/).\n", + "\n", + "In the example below, we'll search over financial headlines." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "c801Ix3l1KG6" + }, + "source": [ + "## Setup\n", + "\n", + "First, let's install the necessary libraries, define some helper functions, and set the API keys we will need to use in this notebook." + ] + }, + { + "cell_type": "code", + "execution_count": 1, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "rKDBvbjNzqJG", + "outputId": "11c96b7d-0ce3-48d0-dbf8-3e798fc8bcc9" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Note: you may need to restart the kernel to use updated packages.\n" + ] + } + ], + "source": [ + "%pip install -qU \\\n", + " pinecone~=7.3 \\\n", + " pinecone-notebooks==0.1.1" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "lKEQWQR81HK-" + }, + "source": [ + "---\n", + "\n", + "🚨 _Note: the above `pip install` is formatted for Jupyter notebooks. If running elsewhere you may need to drop the `!`._\n", + "\n", + "---" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "wHLB42qkIJtR" + }, + "source": [ + "### Helper functions\n", + "\n", + "We'll define these helper functions to print out the search results and to deduplicate and merge the dense and sparse results." + ] + }, + { + "cell_type": "code", + "execution_count": 2, + "metadata": { + "id": "c4SrmHP9IMYO" + }, + "outputs": [], + "source": [ + "def print_hits(results):\n", + " for hit in results['result']['hits']:\n", + " print(f\"id: {hit['_id']}, score: {round(hit['_score'], 2)} text: {hit['fields']['chunk_text']}\")\n", + "\n", + "def merge_chunks(h1, h2):\n", + " \"\"\"Get the unique hits from two search results and return them as single array of {'_id', 'chunk_text'} dicts, printing each dict on a new line.\"\"\"\n", + " # Deduplicate by _id\n", + " deduped_hits = {hit['_id']: hit for hit in h1['result']['hits'] + h2['result']['hits']}.values()\n", + " # Sort by _score descending\n", + " sorted_hits = sorted(deduped_hits, key=lambda x: x['_score'], reverse=True)\n", + " # Transform to format for reranking\n", + " result = [{'_id': hit['_id'], 'chunk_text': hit['fields']['chunk_text'], '_score': hit['_score']} for hit in sorted_hits]\n", + " return result" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "UT_6PqpeBOmz" + }, + "source": [ + "### Get and set the Pinecone API key\n", + "\n", + "We will need a free [Pinecone API key](https://docs.pinecone.io/guides/get-started/quickstart). The code below will either help you sign up for a new Pinecone account or authenticate you. Then it will create a new API key and set it as an environment variable. If you are not running in a Colab environment, it will prompt you to enter the API key and then set it in the environment." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 247 + }, + "id": "kraDwoDTBQlQ", + "outputId": "52f267ab-ed79-42b3-8e3a-6b137ab5bef3" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Pinecone API key not found in environment.\n" + ] + } + ], + "source": [ + "import os\n", + "from getpass import getpass\n", + "\n", + "def get_pinecone_api_key():\n", + " \"\"\"\n", + " Get Pinecone API key from environment variable or prompt user for input.\n", + " Returns the API key as a string.\n", + "\n", + " Only necessary for notebooks. When using Pinecone yourself,\n", + " you can use environment variables or the like to set your API key.\n", + " \"\"\"\n", + " api_key = os.environ.get(\"PINECONE_API_KEY\")\n", + "\n", + " if api_key is None:\n", + " try:\n", + " # Try Colab authentication if available\n", + " from pinecone_notebooks.colab import Authenticate\n", + " Authenticate()\n", + " # If successful, key will now be in environment\n", + " api_key = os.environ.get(\"PINECONE_API_KEY\")\n", + " except ImportError:\n", + " # If not in Colab or authentication fails, prompt user for API key\n", + " print(\"Pinecone API key not found in environment.\")\n", + " api_key = getpass(\"Please enter your Pinecone API key: \")\n", + " # Save to environment for future use in session\n", + " os.environ[\"PINECONE_API_KEY\"] = api_key\n", + "\n", + " return api_key\n", + "\n", + "api_key = get_pinecone_api_key()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "HrpnwL3g2mMZ" + }, + "source": [ + "## 2. Create Pinecone index and load data\n", + "\n", + "### Initializing Pinecone" + ] + }, + { + "cell_type": "code", + "execution_count": 4, + "metadata": { + "id": "WIqvwOQG3J5_" + }, + "outputs": [], + "source": [ + "from pinecone import Pinecone\n", + "\n", + "# Initialize client\n", + "\n", + "pc = Pinecone(\n", + " api_key=api_key,\n", + " # You can remove this parameter for your own projects\n", + " source_tag=\"pinecone_examples:docs:cascading_retrieval\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "LY9WhfQ3ErF3" + }, + "source": [ + "### Create Pinecone indexes with integrated embedding\n", + "\n", + "We'll need a dense and a sparse index, both configured to use integrated embedding, allowing us to pass text rather than vectors.\n", + "\n", + "Integrated embedding allows you to specify the creation of a Pinecone index with a specific Pinecone-hosted embedding model, which makes it easy to interact with the index. To learn more about integrated embedding, including what other models are available, check it out [here](https://docs.pinecone.io/guides/get-started/overview#integrated-embedding)." + ] + }, + { + "cell_type": "code", + "execution_count": 5, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "FHP3-OE4zwmr", + "outputId": "bd575404-21fc-4d19-95d3-62d75e6eb9aa" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'index_fullness': 0.0,\n", + " 'metric': 'dotproduct',\n", + " 'namespaces': {},\n", + " 'total_vector_count': 0,\n", + " 'vector_type': 'sparse'}" + ] + }, + "execution_count": 5, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "sparse_index_name = \"cascading-retrieval-sparse-index\"\n", + "\n", + "if not pc.has_index(sparse_index_name):\n", + " pc.create_index_for_model(\n", + " name=sparse_index_name,\n", + " cloud=\"aws\",\n", + " region=\"us-east-1\",\n", + " embed={\n", + " \"model\": \"pinecone-sparse-english-v0\",\n", + " \"field_map\":{\"text\": \"chunk_text\"}\n", + " }\n", + " )\n", + "\n", + "# Initialize index client\n", + "sparse_index = pc.Index(name=sparse_index_name)\n", + "\n", + "# View index stats\n", + "sparse_index.describe_index_stats()" + ] + }, + { + "cell_type": "code", + "execution_count": 6, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "4nZc3kjH4D2m", + "outputId": "48f3d975-67bd-4dd5-d943-5b8f1ba67e9f" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'dimension': 1024,\n", + " 'index_fullness': 0.0,\n", + " 'metric': 'cosine',\n", + " 'namespaces': {},\n", + " 'total_vector_count': 0,\n", + " 'vector_type': 'dense'}" + ] + }, + "execution_count": 6, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dense_index_name = \"cascading-retrieval-dense-index\"\n", + "\n", + "if not pc.has_index(dense_index_name):\n", + " pc.create_index_for_model(\n", + " name=dense_index_name,\n", + " cloud=\"aws\",\n", + " region=\"us-east-1\",\n", + " embed={\n", + " \"model\": \"llama-text-embed-v2\",\n", + " \"field_map\":{\"text\": \"chunk_text\"}\n", + " }\n", + " )\n", + "\n", + "# Initialize index client\n", + "dense_index = pc.Index(name=dense_index_name)\n", + "\n", + "# View index stats\n", + "dense_index.describe_index_stats()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "znQuQjSVK2wl" + }, + "source": [ + "## Creating our dataset\n", + "\n", + "The example data set below contains financial headlines.\n", + "\n", + "Note: This data set has only 96 records, the maximum allowed by the embedding model in one batch. If we had more than 96 records, we'd have to [upsert in batches](https://docs.pinecone.io/guides/index-data/upsert-data#upsert-in-batches).\n" + ] + }, + { + "cell_type": "code", + "execution_count": 7, + "metadata": { + "id": "xvNgZb4XK34g" + }, + "outputs": [], + "source": [ + "# Financial headlines\n", + "data = [\n", + " { \"_id\": \"vec1\", \"chunk_text\": \"Apple Inc. issued a $10 billion corporate bond in 2023.\" },\n", + " { \"_id\": \"vec2\", \"chunk_text\": \"ETFs tracking the S&P 500 outperformed active funds last year.\" },\n", + " { \"_id\": \"vec3\", \"chunk_text\": \"Tesla's options volume surged after the latest earnings report.\" },\n", + " { \"_id\": \"vec4\", \"chunk_text\": \"Dividend aristocrats are known for consistently raising payouts.\" },\n", + " { \"_id\": \"vec5\", \"chunk_text\": \"The Federal Reserve raised interest rates by 0.25% to curb inflation.\" },\n", + " { \"_id\": \"vec6\", \"chunk_text\": \"Unemployment hit a record low of 3.7% in Q4 of 2024.\" },\n", + " { \"_id\": \"vec7\", \"chunk_text\": \"The CPI index rose by 6% in July 2024, raising concerns about purchasing power.\" },\n", + " { \"_id\": \"vec8\", \"chunk_text\": \"GDP growth in emerging markets outpaced developed economies.\" },\n", + " { \"_id\": \"vec9\", \"chunk_text\": \"Amazon's acquisition of MGM Studios was valued at $8.45 billion.\" },\n", + " { \"_id\": \"vec10\", \"chunk_text\": \"Alphabet reported a 20% increase in advertising revenue.\" },\n", + " { \"_id\": \"vec11\", \"chunk_text\": \"ExxonMobil announced a special dividend after record profits.\" },\n", + " { \"_id\": \"vec12\", \"chunk_text\": \"Tesla plans a 3-for-1 stock split to attract retail investors.\" },\n", + " { \"_id\": \"vec13\", \"chunk_text\": \"Credit card APRs reached an all-time high of 22.8% in 2024.\" },\n", + " { \"_id\": \"vec14\", \"chunk_text\": \"A 529 college savings plan offers tax advantages for education.\" },\n", + " { \"_id\": \"vec15\", \"chunk_text\": \"Emergency savings should ideally cover 6 months of expenses.\" },\n", + " { \"_id\": \"vec16\", \"chunk_text\": \"The average mortgage rate rose to 7.1% in December.\" },\n", + " { \"_id\": \"vec17\", \"chunk_text\": \"The SEC fined a hedge fund $50 million for insider trading.\" },\n", + " { \"_id\": \"vec18\", \"chunk_text\": \"New ESG regulations require companies to disclose climate risks.\" },\n", + " { \"_id\": \"vec19\", \"chunk_text\": \"The IRS introduced a new tax bracket for high earners.\" },\n", + " { \"_id\": \"vec20\", \"chunk_text\": \"Compliance with GDPR is mandatory for companies operating in Europe.\" },\n", + " { \"_id\": \"vec21\", \"chunk_text\": \"What are the best-performing green bonds in a rising rate environment?\" },\n", + " { \"_id\": \"vec22\", \"chunk_text\": \"How does inflation impact the real yield of Treasury bonds?\" },\n", + " { \"_id\": \"vec23\", \"chunk_text\": \"Top SPAC mergers in the technology sector for 2024.\" },\n", + " { \"_id\": \"vec24\", \"chunk_text\": \"Are stablecoins a viable hedge against currency devaluation?\" },\n", + " { \"_id\": \"vec25\", \"chunk_text\": \"Comparison of Roth IRA vs 401(k) for high-income earners.\" },\n", + " { \"_id\": \"vec26\", \"chunk_text\": \"Stock splits and their effect on investor sentiment.\" },\n", + " { \"_id\": \"vec27\", \"chunk_text\": \"Tech IPOs that disappointed in their first year.\" },\n", + " { \"_id\": \"vec28\", \"chunk_text\": \"Impact of interest rate hikes on bank stocks.\" },\n", + " { \"_id\": \"vec29\", \"chunk_text\": \"Growth vs. value investing strategies in 2024.\" },\n", + " { \"_id\": \"vec30\", \"chunk_text\": \"The role of artificial intelligence in quantitative trading.\" },\n", + " { \"_id\": \"vec31\", \"chunk_text\": \"What are the implications of quantitative tightening on equities?\" },\n", + " { \"_id\": \"vec32\", \"chunk_text\": \"How does compounding interest affect long-term investments?\" },\n", + " { \"_id\": \"vec33\", \"chunk_text\": \"What are the best assets to hedge against inflation?\" },\n", + " { \"_id\": \"vec34\", \"chunk_text\": \"Can ETFs provide better diversification than mutual funds?\" },\n", + " { \"_id\": \"vec35\", \"chunk_text\": \"Unemployment hit at 2.4% in Q3 of 2024.\" },\n", + " { \"_id\": \"vec36\", \"chunk_text\": \"Unemployment is expected to hit 2.5% in Q3 of 2024.\" },\n", + " { \"_id\": \"vec37\", \"chunk_text\": \"In Q3 2025 unemployment for the prior year was revised to 2.2%\"},\n", + " { \"_id\": \"vec38\", \"chunk_text\": \"Emerging markets witnessed increased foreign direct investment as global interest rates stabilized.\" },\n", + " { \"_id\": \"vec39\", \"chunk_text\": \"The rise in energy prices significantly impacted inflation trends during the first half of 2024.\" },\n", + " { \"_id\": \"vec40\", \"chunk_text\": \"Labor market trends show a declining participation rate despite record low unemployment in 2024.\" },\n", + " { \"_id\": \"vec41\", \"chunk_text\": \"Forecasts of global supply chain disruptions eased in late 2024, but consumer prices remained elevated due to persistent demand.\" },\n", + " { \"_id\": \"vec42\", \"chunk_text\": \"Tech sector layoffs in Q3 2024 have reshaped hiring trends across high-growth industries.\" },\n", + " { \"_id\": \"vec43\", \"chunk_text\": \"The U.S. dollar weakened against a basket of currencies as the global economy adjusted to shifting trade balances.\" },\n", + " { \"_id\": \"vec44\", \"chunk_text\": \"Central banks worldwide increased gold reserves to hedge against geopolitical and economic instability.\" },\n", + " { \"_id\": \"vec45\", \"chunk_text\": \"Corporate earnings in Q4 2024 were largely impacted by rising raw material costs and currency fluctuations.\" },\n", + " { \"_id\": \"vec46\", \"chunk_text\": \"Economic recovery in Q2 2024 relied heavily on government spending in infrastructure and green energy projects.\" },\n", + " { \"_id\": \"vec47\", \"chunk_text\": \"The housing market saw a rebound in late 2024, driven by falling mortgage rates and pent-up demand.\" },\n", + " { \"_id\": \"vec48\", \"chunk_text\": \"Wage growth outpaced inflation for the first time in years, signaling improved purchasing power in 2024.\" },\n", + " { \"_id\": \"vec49\", \"chunk_text\": \"China's economic growth in 2024 slowed to its lowest level in decades due to structural reforms and weak exports.\" },\n", + " { \"_id\": \"vec50\", \"chunk_text\": \"AI-driven automation in the manufacturing sector boosted productivity but raised concerns about job displacement.\" },\n", + " { \"_id\": \"vec51\", \"chunk_text\": \"The European Union introduced new fiscal policies in 2024 aimed at reducing public debt without stifling growth.\" },\n", + " { \"_id\": \"vec52\", \"chunk_text\": \"Record-breaking weather events in early 2024 have highlighted the growing economic impact of climate change.\" },\n", + " { \"_id\": \"vec53\", \"chunk_text\": \"Cryptocurrencies faced regulatory scrutiny in 2024, leading to volatility and reduced market capitalization.\" },\n", + " { \"_id\": \"vec54\", \"chunk_text\": \"The global tourism sector showed signs of recovery in late 2024 after years of pandemic-related setbacks.\" },\n", + " { \"_id\": \"vec55\", \"chunk_text\": \"Trade tensions between the U.S. and China escalated in 2024, impacting global supply chains and investment flows.\" },\n", + " { \"_id\": \"vec56\", \"chunk_text\": \"Consumer confidence indices remained resilient in Q2 2024 despite fears of an impending recession.\" },\n", + " { \"_id\": \"vec57\", \"chunk_text\": \"Startups in 2024 faced tighter funding conditions as venture capitalists focused on profitability over growth.\" },\n", + " { \"_id\": \"vec58\", \"chunk_text\": \"Oil production cuts in Q1 2024 by OPEC nations drove prices higher, influencing global energy policies.\" },\n", + " { \"_id\": \"vec59\", \"chunk_text\": \"The adoption of digital currencies by central banks increased in 2024, reshaping monetary policy frameworks.\" },\n", + " { \"_id\": \"vec60\", \"chunk_text\": \"Healthcare spending in 2024 surged as governments expanded access to preventive care and pandemic preparedness.\" },\n", + " { \"_id\": \"vec61\", \"chunk_text\": \"The World Bank reported declining poverty rates globally, but regional disparities persisted.\" },\n", + " { \"_id\": \"vec62\", \"chunk_text\": \"Private equity activity in 2024 focused on renewable energy and technology sectors amid shifting investor priorities.\" },\n", + " { \"_id\": \"vec63\", \"chunk_text\": \"Population aging emerged as a critical economic issue in 2024, especially in advanced economies.\" },\n", + " { \"_id\": \"vec64\", \"chunk_text\": \"Rising commodity prices in 2024 strained emerging markets dependent on imports of raw materials.\" },\n", + " { \"_id\": \"vec65\", \"chunk_text\": \"The global shipping industry experienced declining freight rates in 2024 due to overcapacity and reduced demand.\" },\n", + " { \"_id\": \"vec66\", \"chunk_text\": \"Bank lending to small and medium-sized enterprises surged in 2024 as governments incentivized entrepreneurship.\" },\n", + " { \"_id\": \"vec67\", \"chunk_text\": \"Renewable energy projects accounted for a record share of global infrastructure investment in 2024.\" },\n", + " { \"_id\": \"vec68\", \"chunk_text\": \"Cybersecurity spending reached new highs in 2024, reflecting the growing threat of digital attacks on infrastructure.\" },\n", + " { \"_id\": \"vec69\", \"chunk_text\": \"The agricultural sector faced challenges in 2024 due to extreme weather and rising input costs.\" },\n", + " { \"_id\": \"vec70\", \"chunk_text\": \"Consumer spending patterns shifted in 2024, with a greater focus on experiences over goods.\" },\n", + " { \"_id\": \"vec71\", \"chunk_text\": \"The economic impact of the 2008 financial crisis was mitigated by quantitative easing policies.\" },\n", + " { \"_id\": \"vec72\", \"chunk_text\": \"In early 2024, global GDP growth slowed, driven by weaker exports in Asia and Europe.\" },\n", + " { \"_id\": \"vec73\", \"chunk_text\": \"The historical relationship between inflation and unemployment is explained by the Phillips Curve.\" },\n", + " { \"_id\": \"vec74\", \"chunk_text\": \"The World Trade Organization's role in resolving disputes was tested in 2024.\" },\n", + " { \"_id\": \"vec75\", \"chunk_text\": \"The collapse of Silicon Valley Bank raised questions about regulatory oversight in 2024.\" },\n", + " { \"_id\": \"vec76\", \"chunk_text\": \"The cost of living crisis has been exacerbated by stagnant wage growth and rising inflation.\" },\n", + " { \"_id\": \"vec77\", \"chunk_text\": \"Supply chain resilience became a top priority for multinational corporations in 2024.\" },\n", + " { \"_id\": \"vec78\", \"chunk_text\": \"Consumer sentiment surveys in 2024 reflected optimism despite high interest rates.\" },\n", + " { \"_id\": \"vec79\", \"chunk_text\": \"The resurgence of industrial policy in Q1 2024 focused on decoupling critical supply chains.\" },\n", + " { \"_id\": \"vec80\", \"chunk_text\": \"Technological innovation in the fintech sector disrupted traditional banking in 2024.\" },\n", + " { \"_id\": \"vec81\", \"chunk_text\": \"The link between climate change and migration patterns is increasingly recognized.\" },\n", + " { \"_id\": \"vec82\", \"chunk_text\": \"Renewable energy subsidies in 2024 reduced the global reliance on fossil fuels.\" },\n", + " { \"_id\": \"vec83\", \"chunk_text\": \"The economic fallout of geopolitical tensions was evident in rising defense budgets worldwide.\" },\n", + " { \"_id\": \"vec84\", \"chunk_text\": \"The IMF's 2024 global outlook highlighted risks of stagflation in emerging markets.\" },\n", + " { \"_id\": \"vec85\", \"chunk_text\": \"Declining birth rates in advanced economies pose long-term challenges for labor markets.\" },\n", + " { \"_id\": \"vec86\", \"chunk_text\": \"Digital transformation initiatives in 2024 drove productivity gains in the services sector.\" },\n", + " { \"_id\": \"vec87\", \"chunk_text\": \"The U.S. labor market's resilience in 2024 defied predictions of a severe recession.\" },\n", + " { \"_id\": \"vec88\", \"chunk_text\": \"New fiscal measures in the European Union aimed to stabilize debt levels post-pandemic.\" },\n", + " { \"_id\": \"vec89\", \"chunk_text\": \"Venture capital investments in 2024 leaned heavily toward AI and automation startups.\" },\n", + " { \"_id\": \"vec90\", \"chunk_text\": \"The surge in e-commerce in 2024 was facilitated by advancements in logistics technology.\" },\n", + " { \"_id\": \"vec91\", \"chunk_text\": \"The impact of ESG investing on corporate strategies has been a major focus in 2024.\" },\n", + " { \"_id\": \"vec92\", \"chunk_text\": \"Income inequality widened in 2024 despite strong economic growth in developed nations.\" },\n", + " { \"_id\": \"vec93\", \"chunk_text\": \"The collapse of FTX highlighted the volatility and risks associated with cryptocurrencies.\" },\n", + " { \"_id\": \"vec94\", \"chunk_text\": \"Cyberattacks targeting financial institutions in 2024 led to record cybersecurity spending.\" },\n", + " { \"_id\": \"vec95\", \"chunk_text\": \"Automation in agriculture in 2024 increased yields but displaced rural workers.\" },\n", + " { \"_id\": \"vec96\", \"chunk_text\": \"New trade agreements signed 2022 will make an impact in 2024\"},\n", + "]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "H8QBxyB6wK8q" + }, + "source": [ + "### Upserting data into the Pinecone indexes\n", + "\n", + "Here, we embed and upsert the data into Pinecone. During the upsert process, a vector embedding is created for each record using the embedding model we specified on index creation. These vector embeddings are then stored in the index with any additional info, or metadata, we specify. Read more about metadata [here](https://docs.pinecone.io/guides/index-data/indexing-overview#metadata).\n", + "\n", + "We specify a namespace called \"headlines\", which is a higher level unit of organization when interacting with Pinecone but has some important benefits. Querying by namespace performs a sort of broad filter to only records that exist in that namespace. This can be used for isolating customer data for multi-tenancy. And when you divide records into namespaces in a logical way, you speed up queries by ensuring only relevant records are scanned. You can learn more about namespaces [here](https://docs.pinecone.io/guides/index-data/indexing-overview#namespaces)." + ] + }, + { + "cell_type": "code", + "execution_count": 8, + "metadata": { + "id": "xf1qe2tL0PYf" + }, + "outputs": [], + "source": [ + "namespace = \"headlines\"\n", + "\n", + "sparse_index.upsert_records(records = data, namespace = namespace)" + ] + }, + { + "cell_type": "code", + "execution_count": 12, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "aR4xA1Ou5W_A", + "outputId": "08804990-20ca-416c-e099-072debd5c9b7" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'index_fullness': 0.0,\n", + " 'metric': 'dotproduct',\n", + " 'namespaces': {'headlines': {'vector_count': 96}},\n", + " 'total_vector_count': 96,\n", + " 'vector_type': 'sparse'}" + ] + }, + "execution_count": 12, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "sparse_index.describe_index_stats()" + ] + }, + { + "cell_type": "code", + "execution_count": 10, + "metadata": { + "id": "O2egFf8JG5ih" + }, + "outputs": [], + "source": [ + "dense_index.upsert_records(records = data, namespace = namespace)" + ] + }, + { + "cell_type": "code", + "execution_count": 13, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "dtx2tVEB6UiC", + "outputId": "dc47eb45-1611-4999-d70c-7b924d982a48" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'dimension': 1024,\n", + " 'index_fullness': 0.0,\n", + " 'metric': 'cosine',\n", + " 'namespaces': {'headlines': {'vector_count': 96}},\n", + " 'total_vector_count': 96,\n", + " 'vector_type': 'dense'}" + ] + }, + "execution_count": 13, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "dense_index.describe_index_stats()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "FrhzSdvV6hI5" + }, + "source": [ + "## 3. Running queries\n", + "\n", + "Now that our indexes are populated we can begin making queries.\n", + "\n", + "Since we're using Pinecone's integrated embedding, we can query our indexes with the text we want to search for. The search query is vectorized using the same embedding model we specified earlier and then results are returned based on either cosine similarity (dense index) or dotproduct score (sparse index).\n", + "\n", + "We'll search across the sparse and dense indexes with the same query. Note that the resulting scores for dense vs sparse search are on a different scale due to the distance metrics, making them difficult to compare. We will resolve this using reranking in the next step." + ] + }, + { + "cell_type": "code", + "execution_count": 14, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "bW16fiUuHdyX", + "outputId": "731a6c26-3e78-4dd5-a908-32c102b7cf25" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "id: vec35, score: 0.51 text: Unemployment hit at 2.4% in Q3 of 2024.\n", + "id: vec36, score: 0.48 text: Unemployment is expected to hit 2.5% in Q3 of 2024.\n", + "id: vec6, score: 0.41 text: Unemployment hit a record low of 3.7% in Q4 of 2024.\n", + "id: vec46, score: 0.41 text: Economic recovery in Q2 2024 relied heavily on government spending in infrastructure and green energy projects.\n", + "id: vec42, score: 0.41 text: Tech sector layoffs in Q3 2024 have reshaped hiring trends across high-growth industries.\n", + "id: vec87, score: 0.4 text: The U.S. labor market's resilience in 2024 defied predictions of a severe recession.\n", + "id: vec45, score: 0.36 text: Corporate earnings in Q4 2024 were largely impacted by rising raw material costs and currency fluctuations.\n", + "id: vec92, score: 0.36 text: Income inequality widened in 2024 despite strong economic growth in developed nations.\n", + "id: vec55, score: 0.35 text: Trade tensions between the U.S. and China escalated in 2024, impacting global supply chains and investment flows.\n", + "id: vec94, score: 0.35 text: Cyberattacks targeting financial institutions in 2024 led to record cybersecurity spending.\n", + "id: vec48, score: 0.35 text: Wage growth outpaced inflation for the first time in years, signaling improved purchasing power in 2024.\n", + "id: vec58, score: 0.35 text: Oil production cuts in Q1 2024 by OPEC nations drove prices higher, influencing global energy policies.\n", + "id: vec56, score: 0.35 text: Consumer confidence indices remained resilient in Q2 2024 despite fears of an impending recession.\n", + "id: vec62, score: 0.35 text: Private equity activity in 2024 focused on renewable energy and technology sectors amid shifting investor priorities.\n", + "id: vec40, score: 0.34 text: Labor market trends show a declining participation rate despite record low unemployment in 2024.\n", + "id: vec7, score: 0.34 text: The CPI index rose by 6% in July 2024, raising concerns about purchasing power.\n", + "id: vec52, score: 0.34 text: Record-breaking weather events in early 2024 have highlighted the growing economic impact of climate change.\n", + "id: vec79, score: 0.33 text: The resurgence of industrial policy in Q1 2024 focused on decoupling critical supply chains.\n", + "id: vec13, score: 0.33 text: Credit card APRs reached an all-time high of 22.8% in 2024.\n", + "id: vec78, score: 0.33 text: Consumer sentiment surveys in 2024 reflected optimism despite high interest rates.\n" + ] + } + ], + "source": [ + "query = \"Q3 2024 us economic data\"\n", + "\n", + "dense_results = dense_index.search(\n", + " namespace=namespace,\n", + " query={\n", + " \"top_k\": 20,\n", + " \"inputs\": {\n", + " 'text': query\n", + " }\n", + " }\n", + ")\n", + "\n", + "print_hits(dense_results)" + ] + }, + { + "cell_type": "code", + "execution_count": 15, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "P4rCS-F_MwF-", + "outputId": "ed37e14d-b8f0-47a7-815d-774c1876bcd4" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "id: vec35, score: 7.06 text: Unemployment hit at 2.4% in Q3 of 2024.\n", + "id: vec46, score: 7.04 text: Economic recovery in Q2 2024 relied heavily on government spending in infrastructure and green energy projects.\n", + "id: vec36, score: 6.97 text: Unemployment is expected to hit 2.5% in Q3 of 2024.\n", + "id: vec42, score: 6.96 text: Tech sector layoffs in Q3 2024 have reshaped hiring trends across high-growth industries.\n", + "id: vec49, score: 6.66 text: China's economic growth in 2024 slowed to its lowest level in decades due to structural reforms and weak exports.\n", + "id: vec63, score: 6.48 text: Population aging emerged as a critical economic issue in 2024, especially in advanced economies.\n", + "id: vec92, score: 5.72 text: Income inequality widened in 2024 despite strong economic growth in developed nations.\n", + "id: vec52, score: 5.6 text: Record-breaking weather events in early 2024 have highlighted the growing economic impact of climate change.\n", + "id: vec89, score: 4.01 text: Venture capital investments in 2024 leaned heavily toward AI and automation startups.\n", + "id: vec62, score: 4.0 text: Private equity activity in 2024 focused on renewable energy and technology sectors amid shifting investor priorities.\n", + "id: vec57, score: 3.93 text: Startups in 2024 faced tighter funding conditions as venture capitalists focused on profitability over growth.\n", + "id: vec69, score: 3.9 text: The agricultural sector faced challenges in 2024 due to extreme weather and rising input costs.\n", + "id: vec37, score: 3.89 text: In Q3 2025 unemployment for the prior year was revised to 2.2%\n", + "id: vec60, score: 3.82 text: Healthcare spending in 2024 surged as governments expanded access to preventive care and pandemic preparedness.\n", + "id: vec51, score: 3.78 text: The European Union introduced new fiscal policies in 2024 aimed at reducing public debt without stifling growth.\n", + "id: vec55, score: 3.77 text: Trade tensions between the U.S. and China escalated in 2024, impacting global supply chains and investment flows.\n", + "id: vec70, score: 3.76 text: Consumer spending patterns shifted in 2024, with a greater focus on experiences over goods.\n", + "id: vec90, score: 3.71 text: The surge in e-commerce in 2024 was facilitated by advancements in logistics technology.\n", + "id: vec87, score: 3.69 text: The U.S. labor market's resilience in 2024 defied predictions of a severe recession.\n", + "id: vec78, score: 3.67 text: Consumer sentiment surveys in 2024 reflected optimism despite high interest rates.\n" + ] + } + ], + "source": [ + "sparse_results = sparse_index.search(\n", + " namespace=namespace,\n", + " query={\n", + " \"top_k\": 20,\n", + " \"inputs\": {\n", + " \"text\": query\n", + " }\n", + " }\n", + ")\n", + "\n", + "print_hits(sparse_results)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "LMjExxSH1ERT" + }, + "source": [ + "### Merge results\n", + "\n", + "We now merge the results and de-duplicate records." + ] + }, + { + "cell_type": "code", + "execution_count": 16, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "JhKrGMBcy9qX", + "outputId": "addf1517-1422-4d6a-d61e-4ee5e53482d9" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Query Q3 2024 us economic data\n", + "-----\n", + "vec49 - 6.66 - China's economic growth in 2024 slowed to its lowest level in decades due to structural reforms and weak exports.\n", + "vec63 - 6.48 - Population aging emerged as a critical economic issue in 2024, especially in advanced economies.\n", + "vec89 - 4.01 - Venture capital investments in 2024 leaned heavily toward AI and automation startups.\n", + "vec57 - 3.93 - Startups in 2024 faced tighter funding conditions as venture capitalists focused on profitability over growth.\n", + "vec69 - 3.9 - The agricultural sector faced challenges in 2024 due to extreme weather and rising input costs.\n", + "vec37 - 3.89 - In Q3 2025 unemployment for the prior year was revised to 2.2%\n", + "vec60 - 3.82 - Healthcare spending in 2024 surged as governments expanded access to preventive care and pandemic preparedness.\n", + "vec51 - 3.78 - The European Union introduced new fiscal policies in 2024 aimed at reducing public debt without stifling growth.\n", + "vec70 - 3.76 - Consumer spending patterns shifted in 2024, with a greater focus on experiences over goods.\n", + "vec90 - 3.71 - The surge in e-commerce in 2024 was facilitated by advancements in logistics technology.\n", + "vec35 - 0.51 - Unemployment hit at 2.4% in Q3 of 2024.\n", + "vec36 - 0.48 - Unemployment is expected to hit 2.5% in Q3 of 2024.\n", + "vec6 - 0.41 - Unemployment hit a record low of 3.7% in Q4 of 2024.\n", + "vec46 - 0.41 - Economic recovery in Q2 2024 relied heavily on government spending in infrastructure and green energy projects.\n", + "vec42 - 0.41 - Tech sector layoffs in Q3 2024 have reshaped hiring trends across high-growth industries.\n", + "vec87 - 0.4 - The U.S. labor market's resilience in 2024 defied predictions of a severe recession.\n", + "vec45 - 0.36 - Corporate earnings in Q4 2024 were largely impacted by rising raw material costs and currency fluctuations.\n", + "vec92 - 0.36 - Income inequality widened in 2024 despite strong economic growth in developed nations.\n", + "vec55 - 0.35 - Trade tensions between the U.S. and China escalated in 2024, impacting global supply chains and investment flows.\n", + "vec94 - 0.35 - Cyberattacks targeting financial institutions in 2024 led to record cybersecurity spending.\n", + "vec48 - 0.35 - Wage growth outpaced inflation for the first time in years, signaling improved purchasing power in 2024.\n", + "vec58 - 0.35 - Oil production cuts in Q1 2024 by OPEC nations drove prices higher, influencing global energy policies.\n", + "vec56 - 0.35 - Consumer confidence indices remained resilient in Q2 2024 despite fears of an impending recession.\n", + "vec62 - 0.35 - Private equity activity in 2024 focused on renewable energy and technology sectors amid shifting investor priorities.\n", + "vec40 - 0.34 - Labor market trends show a declining participation rate despite record low unemployment in 2024.\n", + "vec7 - 0.34 - The CPI index rose by 6% in July 2024, raising concerns about purchasing power.\n", + "vec52 - 0.34 - Record-breaking weather events in early 2024 have highlighted the growing economic impact of climate change.\n", + "vec79 - 0.33 - The resurgence of industrial policy in Q1 2024 focused on decoupling critical supply chains.\n", + "vec13 - 0.33 - Credit card APRs reached an all-time high of 22.8% in 2024.\n", + "vec78 - 0.33 - Consumer sentiment surveys in 2024 reflected optimism despite high interest rates.\n" + ] + } + ], + "source": [ + "merged_results = merge_chunks(sparse_results, dense_results)\n", + "\n", + "print(\"Query\", query)\n", + "print('-----')\n", + "for row in merged_results:\n", + " print(f\"{row['_id']} - {round(row['_score'], 2)} - {row['chunk_text']}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2tvHzIah-Wbq" + }, + "source": [ + "### Rerank the results\n", + "\n", + "Here, we rerank the merged and de-duplicated results against the query based on a unified relevance score and then return a smaller set (top_n of 10) of the most highly relevant results." + ] + }, + { + "cell_type": "code", + "execution_count": 17, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "LPd15tpE-dvF", + "outputId": "a7f68160-ed5f-4a18-eaa4-0617251caa11" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "Query Q3 2024 us economic data\n", + "-----\n", + "vec36 - 0.84 - Unemployment is expected to hit 2.5% in Q3 of 2024.\n", + "vec35 - 0.76 - Unemployment hit at 2.4% in Q3 of 2024.\n", + "vec48 - 0.33 - Wage growth outpaced inflation for the first time in years, signaling improved purchasing power in 2024.\n", + "vec37 - 0.25 - In Q3 2025 unemployment for the prior year was revised to 2.2%\n", + "vec42 - 0.22 - Tech sector layoffs in Q3 2024 have reshaped hiring trends across high-growth industries.\n", + "vec87 - 0.21 - The U.S. labor market's resilience in 2024 defied predictions of a severe recession.\n", + "vec63 - 0.08 - Population aging emerged as a critical economic issue in 2024, especially in advanced economies.\n", + "vec92 - 0.08 - Income inequality widened in 2024 despite strong economic growth in developed nations.\n", + "vec46 - 0.06 - Economic recovery in Q2 2024 relied heavily on government spending in infrastructure and green energy projects.\n", + "vec6 - 0.06 - Unemployment hit a record low of 3.7% in Q4 of 2024.\n" + ] + } + ], + "source": [ + "final_results = pc.inference.rerank(\n", + " model=\"bge-reranker-v2-m3\",\n", + " query=query,\n", + " documents=merged_results,\n", + " rank_fields=[\"chunk_text\"],\n", + " top_n=10,\n", + " return_documents=True,\n", + " parameters={\n", + " \"truncate\": \"END\"\n", + " }\n", + ")\n", + "\n", + "print(\"Query\", query)\n", + "print('-----')\n", + "for row in final_results.data:\n", + " print(f\"{row['document']['_id']} - {round(row['score'], 2)} - {row['document']['chunk_text']}\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "ztTacxqLA2h1" + }, + "source": [ + "## 4. Demo cleanup\n", + "\n", + "When you're done, delete the indexes to save resources.\n", + "\n", + "Congrats, you've just implemented cascading retrieval or hybrid search with Pinecone!" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "acjL3tu4A7Zd" + }, + "outputs": [], + "source": [ + "pc.delete_index(name=sparse_index_name)\n", + "\n", + "pc.delete_index(name=dense_index_name)" + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "kernelspec": { + "display_name": ".venv", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.12.9" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +} diff --git a/docs/lexical-search.ipynb b/docs/lexical-search.ipynb new file mode 100644 index 00000000..57557cee --- /dev/null +++ b/docs/lexical-search.ipynb @@ -0,0 +1,683 @@ +{ + "cells": [ + { + "cell_type": "markdown", + "metadata": { + "id": "k7Lc9I6taO3k" + }, + "source": [ + "[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/pinecone-io/examples/blob/master/docs/lexical-search.ipynb) [![Open nbviewer](https://raw.githubusercontent.com/pinecone-io/examples/master/assets/nbviewer-shield.svg)](https://nbviewer.org/github/pinecone-io/examples/blob/master/docs/lexical-search.ipynb)\n", + "\n", + "# Lexical Search\n", + "\n", + "In this notebook, you'll learn how to use Pinecone for lexical (keyword) search.\n", + "\n", + "Lexical search is a form of retrieval that allows you to find records that most exactly match the words or phrases in a query. Lexical search uses sparse vectors, which have a very large number of dimensions, where only a small proportion of values are non-zero. The dimensions represent words from a dictionary, and the values represent the contextual importance of these words in the document. Words are scored independently and then summed, with the most similar records scored highest.\n", + "\n", + "You can read more about how sparse retrieval works [here](https://www.pinecone.io/learn/sparse-retrieval/).\n", + "\n", + "While semantic search (over a dense index) allows you to find records that are most similar in meaning and context to a query, lexical search (over a sparse index) lets you search for exact token matches like keywords, acronyms, stock tickers, or even proprietary domain terminology, like a product name.\n", + "\n", + "In the example below, we'll search over financial headlines to find news related to Apple." + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "hrSfFiIC5roI" + }, + "source": [ + "## 1. Setup\n", + "\n", + "First, let's install the necessary libraries, define some helper functions, and set the API keys we will need to use in this notebook." + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "q03L1BYEZQfe", + "outputId": "7ade39e8-9dfa-4ad2-a81a-6a82e5c7b730" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "\u001b[?25l \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/491.4 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K \u001b[91m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m\u001b[90m╺\u001b[0m \u001b[32m481.3/491.4 kB\u001b[0m \u001b[31m17.5 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\r\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m491.4/491.4 kB\u001b[0m \u001b[31m10.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25h\u001b[?25l \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m0.0/587.6 kB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\r\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m587.6/587.6 kB\u001b[0m \u001b[31m30.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m240.0/240.0 kB\u001b[0m \u001b[31m17.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[2K \u001b[90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\u001b[0m \u001b[32m65.5/65.5 kB\u001b[0m \u001b[31m5.2 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n", + "\u001b[?25h" + ] + } + ], + "source": [ + "%pip install -qU \\\n", + " pinecone~=7.3 \\\n", + " pinecone-notebooks==0.1.1" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "_xmobBcgqKEL" + }, + "source": [ + "---\n", + "\n", + "🚨 _Note: the above `pip install` is formatted for Jupyter notebooks. If running elsewhere you may need to drop the `!`._\n", + "\n", + "---" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "WeMh_4aKh-nJ" + }, + "source": [ + "### Helper functions\n", + "\n", + "We'll define this helper function to print out the search results." + ] + }, + { + "cell_type": "code", + "execution_count": 82, + "metadata": { + "id": "2pquxRO60gBg" + }, + "outputs": [], + "source": [ + "def print_hits(results):\n", + " for hit in results['result']['hits']:\n", + " print(f\"id: {hit['_id']}, score: {round(hit['_score'], 2)} text: {hit['fields']['chunk_text']}\")\n", + "\n", + " if len(results['result']['hits']) == 0:\n", + " print(\"No results found\")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "CjD2KYv10ha9" + }, + "source": [ + "### Get and set the Pinecone API key\n", + "\n", + "We will need a free [Pinecone API key](https://docs.pinecone.io/guides/get-started/quickstart). The code below will either help you sign up for a new Pinecone account or authenticate you. Then it will create a new API key and set it as an environment variable. If you are not running in a Colab environment, it will prompt you to enter the API key and then set it in the environment." + ] + }, + { + "cell_type": "code", + "execution_count": 3, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/", + "height": 247 + }, + "id": "rPq3DJN5K7dw", + "outputId": "78c47aa5-ad7f-4d17-b938-cd9013e80094" + }, + "outputs": [ + { + "data": { + "text/html": [ + "" + ], + "text/plain": [ + "" + ] + }, + "metadata": {}, + "output_type": "display_data" + } + ], + "source": [ + "import os\n", + "from getpass import getpass\n", + "\n", + "def get_pinecone_api_key():\n", + " \"\"\"\n", + " Get Pinecone API key from environment variable or prompt user for input.\n", + " Returns the API key as a string.\n", + "\n", + " Only necessary for notebooks. When using Pinecone yourself,\n", + " you can use environment variables or the like to set your API key.\n", + " \"\"\"\n", + " api_key = os.environ.get(\"PINECONE_API_KEY\")\n", + "\n", + " if api_key is None:\n", + " try:\n", + " # Try Colab authentication if available\n", + " from pinecone_notebooks.colab import Authenticate\n", + " Authenticate()\n", + " # If successful, key will now be in environment\n", + " api_key = os.environ.get(\"PINECONE_API_KEY\")\n", + " except ImportError:\n", + " # If not in Colab or authentication fails, prompt user for API key\n", + " print(\"Pinecone API key not found in environment.\")\n", + " api_key = getpass(\"Please enter your Pinecone API key: \")\n", + " # Save to environment for future use in session\n", + " os.environ[\"PINECONE_API_KEY\"] = api_key\n", + "\n", + " return api_key\n", + "\n", + "api_key = get_pinecone_api_key()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "CHsyZ3Dj0lDv" + }, + "source": [ + "## 2. Create Pinecone index and load data\n", + "\n", + "### Initializing Pinecone" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "mc66NEBAcQHY" + }, + "outputs": [], + "source": [ + "from pinecone import Pinecone\n", + "\n", + "# Initialize client\n", + "\n", + "pc = Pinecone(\n", + " api_key=api_key,\n", + " # You can remove this parameter for your own projects\n", + " source_tag=\"pinecone_examples:docs:lexical_search\"\n", + ")" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "8GIyxWIqK7dw" + }, + "source": [ + "### Create a Pinecone index with integrated embedding\n", + "\n", + "Lexical search requires three pieces: a processed data source (chunks, or records in Pinecone), an embedding model, and a vector database.\n", + "\n", + "Integrated embedding allows you to specify the creation of a Pinecone index with a specific Pinecone-hosted embedding model, which makes it easy to interact with the index. To learn more about integrated embedding, including what other models are available, check it out [here](https://docs.pinecone.io/guides/get-started/overview#integrated-embedding).\n", + "\n", + "\n", + "Here, we'll create an index with the [pinecone-sparse-english-v0](https://docs.pinecone.io/models/pinecone-sparse-english-v0) embedding model. We also specify a mapping for what field in our records we will embed with this model." + ] + }, + { + "cell_type": "code", + "execution_count": 77, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "kT8pfoO46Iwg", + "outputId": "cf44d0c1-1e1c-49eb-aa78-bee8fb2dcfff" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'index_fullness': 0.0,\n", + " 'metric': 'dotproduct',\n", + " 'namespaces': {'headlines': {'vector_count': 96}},\n", + " 'total_vector_count': 96,\n", + " 'vector_type': 'sparse'}" + ] + }, + "execution_count": 77, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "\n", + "index_name = \"lexical-search\"\n", + "\n", + "if not pc.has_index(index_name):\n", + " pc.create_index_for_model(\n", + " name=index_name,\n", + " cloud=\"aws\",\n", + " region=\"us-east-1\",\n", + " embed={\n", + " \"model\": \"pinecone-sparse-english-v0\",\n", + " \"field_map\":{\"text\": \"chunk_text\"}\n", + " }\n", + " )\n", + "\n", + "# Initialize index client\n", + "index = pc.Index(name=index_name)\n", + "\n", + "# View index stats\n", + "index.describe_index_stats()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "qDZYe_KaK7dx" + }, + "source": [ + "### Creating our dataset\n", + "\n", + "Sparse indexes excel when you need exact token matching and predictable precision. For example:\n", + "\n", + "- searching over terminology-heavy domains where subword tokenization can pose a problem, such as medical, legal, and financial domains\n", + "- queries that need precise entity matching, such as over proper nouns, names, part numbers, stock tickers, products etc that are difficult to put into metadata\n", + "\n", + "The example data set below contains financial headlines.\n", + "\n", + "Note: This data set has only 96 records, the maximum allowed by the embedding model in one batch. If we had more than 96 records, we'd have to [upsert in batches](https://docs.pinecone.io/guides/index-data/upsert-data#upsert-in-batches).\n" + ] + }, + { + "cell_type": "code", + "execution_count": 78, + "metadata": { + "id": "Kf6QBz2sooqT" + }, + "outputs": [], + "source": [ + "# Financial headlines\n", + "data = [\n", + " { \"_id\": \"vec1\", \"chunk_text\": \"Apple Inc. issued a $10 billion corporate bond in 2023.\" },\n", + " { \"_id\": \"vec2\", \"chunk_text\": \"ETFs tracking the S&P 500 outperformed active funds last year.\" },\n", + " { \"_id\": \"vec3\", \"chunk_text\": \"Tesla's options volume surged after the latest earnings report.\" },\n", + " { \"_id\": \"vec4\", \"chunk_text\": \"Dividend aristocrats are known for consistently raising payouts.\" },\n", + " { \"_id\": \"vec5\", \"chunk_text\": \"The Federal Reserve raised interest rates by 0.25% to curb inflation.\" },\n", + " { \"_id\": \"vec6\", \"chunk_text\": \"Unemployment hit a record low of 3.7% in Q4 of 2024.\" },\n", + " { \"_id\": \"vec7\", \"chunk_text\": \"The CPI index rose by 6% in July 2024, raising concerns about purchasing power.\" },\n", + " { \"_id\": \"vec8\", \"chunk_text\": \"GDP growth in emerging markets outpaced developed economies.\" },\n", + " { \"_id\": \"vec9\", \"chunk_text\": \"Amazon's acquisition of MGM Studios was valued at $8.45 billion.\" },\n", + " { \"_id\": \"vec10\", \"chunk_text\": \"Alphabet reported a 20% increase in advertising revenue.\" },\n", + " { \"_id\": \"vec11\", \"chunk_text\": \"ExxonMobil announced a special dividend after record profits.\" },\n", + " { \"_id\": \"vec12\", \"chunk_text\": \"Tesla plans a 3-for-1 stock split to attract retail investors.\" },\n", + " { \"_id\": \"vec13\", \"chunk_text\": \"Credit card APRs reached an all-time high of 22.8% in 2024.\" },\n", + " { \"_id\": \"vec14\", \"chunk_text\": \"A 529 college savings plan offers tax advantages for education.\" },\n", + " { \"_id\": \"vec15\", \"chunk_text\": \"Emergency savings should ideally cover 6 months of expenses.\" },\n", + " { \"_id\": \"vec16\", \"chunk_text\": \"The average mortgage rate rose to 7.1% in December.\" },\n", + " { \"_id\": \"vec17\", \"chunk_text\": \"The SEC fined a hedge fund $50 million for insider trading.\" },\n", + " { \"_id\": \"vec18\", \"chunk_text\": \"New ESG regulations require companies to disclose climate risks.\" },\n", + " { \"_id\": \"vec19\", \"chunk_text\": \"The IRS introduced a new tax bracket for high earners.\" },\n", + " { \"_id\": \"vec20\", \"chunk_text\": \"Compliance with GDPR is mandatory for companies operating in Europe.\" },\n", + " { \"_id\": \"vec21\", \"chunk_text\": \"What are the best-performing green bonds in a rising rate environment?\" },\n", + " { \"_id\": \"vec22\", \"chunk_text\": \"How does inflation impact the real yield of Treasury bonds?\" },\n", + " { \"_id\": \"vec23\", \"chunk_text\": \"Top SPAC mergers in the technology sector for 2024.\" },\n", + " { \"_id\": \"vec24\", \"chunk_text\": \"Are stablecoins a viable hedge against currency devaluation?\" },\n", + " { \"_id\": \"vec25\", \"chunk_text\": \"Comparison of Roth IRA vs 401(k) for high-income earners.\" },\n", + " { \"_id\": \"vec26\", \"chunk_text\": \"Stock splits and their effect on investor sentiment.\" },\n", + " { \"_id\": \"vec27\", \"chunk_text\": \"Tech IPOs that disappointed in their first year.\" },\n", + " { \"_id\": \"vec28\", \"chunk_text\": \"Impact of interest rate hikes on bank stocks.\" },\n", + " { \"_id\": \"vec29\", \"chunk_text\": \"Growth vs. value investing strategies in 2024.\" },\n", + " { \"_id\": \"vec30\", \"chunk_text\": \"The role of artificial intelligence in quantitative trading.\" },\n", + " { \"_id\": \"vec31\", \"chunk_text\": \"What are the implications of quantitative tightening on equities?\" },\n", + " { \"_id\": \"vec32\", \"chunk_text\": \"How does compounding interest affect long-term investments?\" },\n", + " { \"_id\": \"vec33\", \"chunk_text\": \"What are the best assets to hedge against inflation?\" },\n", + " { \"_id\": \"vec34\", \"chunk_text\": \"Apple will spend more than $500 billion in the U.S. over the next four years\" },\n", + " { \"_id\": \"vec35\", \"chunk_text\": \"Unemployment hit at 2.4% in Q3 of 2024.\" },\n", + " { \"_id\": \"vec36\", \"chunk_text\": \"Unemployment is expected to hit 2.5% in Q3 of 2024.\" },\n", + " { \"_id\": \"vec37\", \"chunk_text\": \"In Q3 2025 unemployment for the prior year was revised to 2.2%\"},\n", + " { \"_id\": \"vec38\", \"chunk_text\": \"Emerging markets witnessed increased foreign direct investment as global interest rates stabilized.\" },\n", + " { \"_id\": \"vec39\", \"chunk_text\": \"The rise in energy prices significantly impacted inflation trends during the first half of 2024.\" },\n", + " { \"_id\": \"vec40\", \"chunk_text\": \"Labor market trends show a declining participation rate despite record low unemployment in 2024.\" },\n", + " { \"_id\": \"vec41\", \"chunk_text\": \"Forecasts of global supply chain disruptions eased in late 2024, but consumer prices remained elevated due to persistent demand.\" },\n", + " { \"_id\": \"vec42\", \"chunk_text\": \"Tech sector layoffs in Q3 2024 have reshaped hiring trends across high-growth industries.\" },\n", + " { \"_id\": \"vec43\", \"chunk_text\": \"The U.S. dollar weakened against a basket of currencies as the global economy adjusted to shifting trade balances.\" },\n", + " { \"_id\": \"vec44\", \"chunk_text\": \"Central banks worldwide increased gold reserves to hedge against geopolitical and economic instability.\" },\n", + " { \"_id\": \"vec45\", \"chunk_text\": \"Corporate earnings in Q4 2024 were largely impacted by rising raw material costs and currency fluctuations.\" },\n", + " { \"_id\": \"vec46\", \"chunk_text\": \"Economic recovery in Q2 2024 relied heavily on government spending in infrastructure and green energy projects.\" },\n", + " { \"_id\": \"vec47\", \"chunk_text\": \"The housing market saw a rebound in late 2024, driven by falling mortgage rates and pent-up demand.\" },\n", + " { \"_id\": \"vec48\", \"chunk_text\": \"Wage growth outpaced inflation for the first time in years, signaling improved purchasing power in 2024.\" },\n", + " { \"_id\": \"vec49\", \"chunk_text\": \"China's economic growth in 2024 slowed to its lowest level in decades due to structural reforms and weak exports.\" },\n", + " { \"_id\": \"vec50\", \"chunk_text\": \"AI-driven automation in the manufacturing sector boosted productivity but raised concerns about job displacement.\" },\n", + " { \"_id\": \"vec51\", \"chunk_text\": \"The European Union introduced new fiscal policies in 2024 aimed at reducing public debt without stifling growth.\" },\n", + " { \"_id\": \"vec52\", \"chunk_text\": \"Record-breaking weather events in early 2024 have highlighted the growing economic impact of climate change.\" },\n", + " { \"_id\": \"vec53\", \"chunk_text\": \"Cryptocurrencies faced regulatory scrutiny in 2024, leading to volatility and reduced market capitalization.\" },\n", + " { \"_id\": \"vec54\", \"chunk_text\": \"The global tourism sector showed signs of recovery in late 2024 after years of pandemic-related setbacks.\" },\n", + " { \"_id\": \"vec55\", \"chunk_text\": \"Trade tensions between the U.S. and China escalated in 2024, impacting global supply chains and investment flows.\" },\n", + " { \"_id\": \"vec56\", \"chunk_text\": \"Consumer confidence indices remained resilient in Q2 2024 despite fears of an impending recession.\" },\n", + " { \"_id\": \"vec57\", \"chunk_text\": \"Startups in 2024 faced tighter funding conditions as venture capitalists focused on profitability over growth.\" },\n", + " { \"_id\": \"vec58\", \"chunk_text\": \"Oil production cuts in Q1 2024 by OPEC nations drove prices higher, influencing global energy policies.\" },\n", + " { \"_id\": \"vec59\", \"chunk_text\": \"The adoption of digital currencies by central banks increased in 2024, reshaping monetary policy frameworks.\" },\n", + " { \"_id\": \"vec60\", \"chunk_text\": \"Healthcare spending in 2024 surged as governments expanded access to preventive care and pandemic preparedness.\" },\n", + " { \"_id\": \"vec61\", \"chunk_text\": \"The World Bank reported declining poverty rates globally, but regional disparities persisted.\" },\n", + " { \"_id\": \"vec62\", \"chunk_text\": \"Private equity activity in 2024 focused on renewable energy and technology sectors amid shifting investor priorities.\" },\n", + " { \"_id\": \"vec63\", \"chunk_text\": \"Population aging emerged as a critical economic issue in 2024, especially in advanced economies.\" },\n", + " { \"_id\": \"vec64\", \"chunk_text\": \"Rising commodity prices in 2024 strained emerging markets dependent on imports of raw materials.\" },\n", + " { \"_id\": \"vec65\", \"chunk_text\": \"The global shipping industry experienced declining freight rates in 2024 due to overcapacity and reduced demand.\" },\n", + " { \"_id\": \"vec66\", \"chunk_text\": \"Bank lending to small and medium-sized enterprises surged in 2024 as governments incentivized entrepreneurship.\" },\n", + " { \"_id\": \"vec67\", \"chunk_text\": \"Renewable energy projects accounted for a record share of global infrastructure investment in 2024.\" },\n", + " { \"_id\": \"vec68\", \"chunk_text\": \"Cybersecurity spending reached new highs in 2024, reflecting the growing threat of digital attacks on infrastructure.\" },\n", + " { \"_id\": \"vec69\", \"chunk_text\": \"The agricultural sector faced challenges in 2024 due to extreme weather and rising input costs.\" },\n", + " { \"_id\": \"vec70\", \"chunk_text\": \"Consumer spending patterns shifted in 2024, with a greater focus on experiences over goods.\" },\n", + " { \"_id\": \"vec71\", \"chunk_text\": \"The economic impact of the 2008 financial crisis was mitigated by quantitative easing policies.\" },\n", + " { \"_id\": \"vec72\", \"chunk_text\": \"In early 2024, global GDP growth slowed, driven by weaker exports in Asia and Europe.\" },\n", + " { \"_id\": \"vec73\", \"chunk_text\": \"The historical relationship between inflation and unemployment is explained by the Phillips Curve.\" },\n", + " { \"_id\": \"vec74\", \"chunk_text\": \"Apple prices first bond offering in 2 years\" },\n", + " { \"_id\": \"vec75\", \"chunk_text\": \"The collapse of Silicon Valley Bank raised questions about regulatory oversight in 2024.\" },\n", + " { \"_id\": \"vec76\", \"chunk_text\": \"The cost of living crisis has been exacerbated by stagnant wage growth and rising inflation.\" },\n", + " { \"_id\": \"vec77\", \"chunk_text\": \"Supply chain resilience became a top priority for multinational corporations in 2024.\" },\n", + " { \"_id\": \"vec78\", \"chunk_text\": \"Consumer sentiment surveys in 2024 reflected optimism despite high interest rates.\" },\n", + " { \"_id\": \"vec79\", \"chunk_text\": \"The resurgence of industrial policy in Q1 2024 focused on decoupling critical supply chains.\" },\n", + " { \"_id\": \"vec80\", \"chunk_text\": \"Technological innovation in the fintech sector disrupted traditional banking in 2024.\" },\n", + " { \"_id\": \"vec81\", \"chunk_text\": \"Apple to pay $95 million to settle lawsuit accusing Siri of eavesdropping.\" },\n", + " { \"_id\": \"vec82\", \"chunk_text\": \"Renewable energy subsidies in 2024 reduced the global reliance on fossil fuels.\" },\n", + " { \"_id\": \"vec83\", \"chunk_text\": \"The economic fallout of geopolitical tensions was evident in rising defense budgets worldwide.\" },\n", + " { \"_id\": \"vec84\", \"chunk_text\": \"The IMF's 2024 global outlook highlighted risks of stagflation in emerging markets.\" },\n", + " { \"_id\": \"vec85\", \"chunk_text\": \"Declining birth rates in advanced economies pose long-term challenges for labor markets.\" },\n", + " { \"_id\": \"vec86\", \"chunk_text\": \"Digital transformation initiatives in 2024 drove productivity gains in the services sector.\" },\n", + " { \"_id\": \"vec87\", \"chunk_text\": \"The U.S. labor market's resilience in 2024 defied predictions of a severe recession.\" },\n", + " { \"_id\": \"vec88\", \"chunk_text\": \"New fiscal measures in the European Union aimed to stabilize debt levels post-pandemic.\" },\n", + " { \"_id\": \"vec89\", \"chunk_text\": \"Venture capital investments in 2024 leaned heavily toward AI and automation startups.\" },\n", + " { \"_id\": \"vec90\", \"chunk_text\": \"The surge in e-commerce in 2024 was facilitated by advancements in logistics technology.\" },\n", + " { \"_id\": \"vec91\", \"chunk_text\": \"The impact of ESG investing on corporate strategies has been a major focus in 2024.\" },\n", + " { \"_id\": \"vec92\", \"chunk_text\": \"Income inequality widened in 2024 despite strong economic growth in developed nations.\" },\n", + " { \"_id\": \"vec93\", \"chunk_text\": \"The collapse of FTX highlighted the volatility and risks associated with cryptocurrencies.\" },\n", + " { \"_id\": \"vec94\", \"chunk_text\": \"Cyberattacks targeting financial institutions in 2024 led to record cybersecurity spending.\" },\n", + " { \"_id\": \"vec95\", \"chunk_text\": \"Latest Data Shows Another Bad News For Apple (AAPL)\" },\n", + " { \"_id\": \"vec96\", \"chunk_text\": \"New trade agreements signed 2021 will make an impact in 2023\"},\n", + "]" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "zzruIsSBK7dy" + }, + "source": [ + "### Upserting data into the Pinecone index" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "piqKTquoK7dz" + }, + "source": [ + "Here, we embed and upsert the data into Pinecone. During the upsert process, a vector embedding is created for each record using the embedding model we specified on index creation. These vector embeddings are then stored in the index with any additional info, or metadata, we specify. Read more about metadata [here](https://docs.pinecone.io/guides/index-data/indexing-overview#metadata).\n", + "\n", + "We specify a namespace called \"headlines\", which is a higher level unit of organization when interacting with Pinecone but has some important benefits. Querying by namespace performs a sort of broad filter to only records that exist in that namespace. This can be used for isolating customer data for multi-tenancy. And when you divide records into namespaces in a logical way, you speed up queries by ensuring only relevant records are scanned. You can learn more about namespaces [here](https://docs.pinecone.io/guides/index-data/indexing-overview#namespaces)." + ] + }, + { + "cell_type": "code", + "execution_count": 102, + "metadata": { + "id": "RhR6WOi1huXZ" + }, + "outputs": [], + "source": [ + "namespace = \"headlines\"\n", + "index.upsert_records(records = data, namespace = namespace)" + ] + }, + { + "cell_type": "code", + "execution_count": 103, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "rJzZk7x06J0g", + "outputId": "5386f60d-9de9-4484-dc53-333ba4f94613" + }, + "outputs": [ + { + "data": { + "text/plain": [ + "{'index_fullness': 0.0,\n", + " 'metric': 'dotproduct',\n", + " 'namespaces': {'headlines': {'vector_count': 96}},\n", + " 'total_vector_count': 96,\n", + " 'vector_type': 'sparse'}" + ] + }, + "execution_count": 103, + "metadata": {}, + "output_type": "execute_result" + } + ], + "source": [ + "index.describe_index_stats()" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "VrK_IN079Vuu" + }, + "source": [ + "## 3. Running queries\n" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "rr4unPAq9alb" + }, + "source": [ + "Now that our index is populated we can begin making queries.\n", + "\n", + "Since we're using Pinecone's integrated embedding, we can query our sparse index with the text we want to search for. The search query is vectorized using the same embedding model we specified prior and then only the most relevant documents (based on dotproduct score) are returned." + ] + }, + { + "cell_type": "code", + "execution_count": 80, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "YB0q0uFtK7dz", + "outputId": "3350cf34-272b-498d-beb4-26a2d6a53d88" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "id: vec1, score: 8.68 text: Apple Inc. issued a $10 billion corporate bond in 2023.\n", + "id: vec95, score: 6.08 text: Latest Data Shows Another Bad News For Apple (AAPL)\n", + "id: vec34, score: 5.26 text: Apple will spend more than $500 billion in the U.S. over the next four years\n", + "id: vec74, score: 4.39 text: Apple prices first bond offering in 2 years\n", + "id: vec81, score: 4.21 text: Apple to pay $95 million to settle lawsuit accusing Siri of eavesdropping.\n", + "id: vec71, score: 3.2 text: The economic impact of the 2008 financial crisis was mitigated by quantitative easing policies.\n", + "id: vec96, score: 3.03 text: New trade agreements signed 2021 will make an impact in 2023\n", + "id: vec94, score: 2.81 text: Cyberattacks targeting financial institutions in 2024 led to record cybersecurity spending.\n", + "id: vec50, score: 0.51 text: AI-driven automation in the manufacturing sector boosted productivity but raised concerns about job displacement.\n", + "id: vec75, score: 0.49 text: The collapse of Silicon Valley Bank raised questions about regulatory oversight in 2024.\n" + ] + } + ], + "source": [ + "search_query = \"2023 financial data about Apple\"\n", + "\n", + "results = index.search(\n", + " namespace=namespace,\n", + " query={\n", + " \"top_k\": 10,\n", + " \"inputs\": {\n", + " 'text': search_query\n", + " }\n", + " }\n", + ")\n", + "\n", + "print_hits(results)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "kBJ5hoNRtEoZ" + }, + "source": [ + "### Rerank the results\n", + "\n", + "We can use a reranking model to further score the results based on their *semantic relevance* to the query and return a new, more accurate ranking. This means results aren't sorted based on their *lexical relevance* anymore, but instead now based on *semantic relevance*. You can see this by inspecting the results and scores from the first query above with the results and scores in this next query." + ] + }, + { + "cell_type": "code", + "execution_count": 99, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "AP5dxEObor2b", + "outputId": "a2a7040f-c73b-46e3-9d23-5b72ca7a3f88" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "id: vec1, score: 0.8 text: Apple Inc. issued a $10 billion corporate bond in 2023.\n", + "id: vec34, score: 0.03 text: Apple will spend more than $500 billion in the U.S. over the next four years\n", + "id: vec95, score: 0.02 text: Latest Data Shows Another Bad News For Apple (AAPL)\n", + "id: vec96, score: 0.0 text: New trade agreements signed 2021 will make an impact in 2023\n", + "id: vec74, score: 0.0 text: Apple prices first bond offering in 2 years\n", + "id: vec81, score: 0.0 text: Apple to pay $95 million to settle lawsuit accusing Siri of eavesdropping.\n", + "id: vec94, score: 0.0 text: Cyberattacks targeting financial institutions in 2024 led to record cybersecurity spending.\n", + "id: vec75, score: 0.0 text: The collapse of Silicon Valley Bank raised questions about regulatory oversight in 2024.\n", + "id: vec71, score: 0.0 text: The economic impact of the 2008 financial crisis was mitigated by quantitative easing policies.\n", + "id: vec50, score: 0.0 text: AI-driven automation in the manufacturing sector boosted productivity but raised concerns about job displacement.\n" + ] + } + ], + "source": [ + "results = index.search(\n", + " namespace=namespace,\n", + " query={\n", + " \"top_k\": 10,\n", + " \"inputs\": {\n", + " 'text': search_query\n", + " }\n", + " },\n", + " rerank={\n", + " \"model\": \"bge-reranker-v2-m3\",\n", + " \"rank_fields\": [\"chunk_text\"]\n", + " }\n", + ")\n", + "\n", + "print_hits(results)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7Ae9FRQKrDqu" + }, + "source": [ + "\n", + "You may get fewer than top_k results if top_k is larger than the number of sparse vectors in your index that match your query. That is, any vectors where the dotproduct score is 0 will be discarded.\n", + "\n", + "Searching our financial headlines for the word \"banana\" will return zero results because our data doesn't include the term \"banana\" at all. For example:" + ] + }, + { + "cell_type": "code", + "execution_count": 84, + "metadata": { + "colab": { + "base_uri": "https://localhost:8080/" + }, + "id": "cEKqiufKmgL7", + "outputId": "b313d8a9-6cca-47c2-ed3d-d67d74ad4097" + }, + "outputs": [ + { + "name": "stdout", + "output_type": "stream", + "text": [ + "No results found\n" + ] + } + ], + "source": [ + "search_query = \"banana\"\n", + "\n", + "results = index.search(\n", + " namespace=namespace,\n", + " query={\n", + " \"top_k\": 10,\n", + " \"inputs\": {\n", + " 'text': search_query\n", + " }\n", + " }\n", + ")\n", + "\n", + "print_hits(results)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "7mWkxCNdK7d0" + }, + "source": [ + "## 4. Demo cleanup\n", + "\n", + "When you're done, delete the index to save resources.\n", + "\n", + "Congrats, you've just implemented lexical search with Pinecone!\n" + ] + }, + { + "cell_type": "code", + "execution_count": null, + "metadata": { + "id": "-cWdeKzhAtww" + }, + "outputs": [], + "source": [ + "pc.delete_index(name=index_name)" + ] + }, + { + "cell_type": "markdown", + "metadata": { + "id": "2B0zxR6hbf5d" + }, + "source": [ + "---" + ] + } + ], + "metadata": { + "colab": { + "provenance": [] + }, + "gpuClass": "standard", + "kernelspec": { + "display_name": "Python 3 (ipykernel)", + "language": "python", + "name": "python3" + }, + "language_info": { + "codemirror_mode": { + "name": "ipython", + "version": 3 + }, + "file_extension": ".py", + "mimetype": "text/x-python", + "name": "python", + "nbconvert_exporter": "python", + "pygments_lexer": "ipython3", + "version": "3.11.9" + } + }, + "nbformat": 4, + "nbformat_minor": 0 +}