Vectors are included for all older indices when they weren't before #136502
Description
Elasticsearch Version
9.1 -> 9.2 (see details)
Installed Plugins
No response
Java Version
bundled
OS Version
Cloud and Serverless
Problem Description
There is a bug with exclude_vectors for older index versions. This was introduced in #131907
We observed this with live data on an index with version 9015000. I was able to duplicate this with Elasticsearch 9.1.1 to start.
For older indices, when we upgrade to newer versions of Elasticsearch, we are returning all chunks.
Steps to Reproduce
I started by creating a cloud deployment running 9.1.1. I then created and populated an index with the following queries:
PUT books
{
"mappings": {
"properties": {
"title": { "type": "semantic_text" },
"author": { "type": "keyword" },
"publication_year": { "type": "integer" },
"rating": { "type": "float" },
"genre": { "type": "keyword" },
"synopsis": { "type": "semantic_text" }
}
}
}
POST books/_bulk
{"index": {}}
{"title": "Dune", "author": "Frank Herbert", "publication_year": 1965, "rating": 4.8, "genre": ["sci-fi", "space", "politics"], "synopsis": "On the desert planet Arrakis, Paul Atreides is thrust into a deadly struggle over spice, the universe\u2019s most valuable resource. As he embraces his destiny, he must navigate betrayal, prophecy, and war. This novel explores environmentalism, power, and religious mysticism across an epic scope. It is one of the most influential sci-fi works of all time."}
{"index": {}}
{"title": "Foundation", "author": "Isaac Asimov", "publication_year": 1951, "rating": 4.7, "genre": ["sci-fi", "galactic", "political"], "synopsis": "Mathematician Hari Seldon predicts the fall of the Galactic Empire and sets in motion a plan to shorten the coming dark age. As the Foundation grows, it faces threats from warlords, politicians, and internal strife. The novel explores the rise and fall of civilizations through science and foresight. It pioneered the concept of psychohistory in fiction."}
{"index": {}}
{"title": "Neuromancer", "author": "William Gibson", "publication_year": 1984, "rating": 4.4, "genre": ["cyberpunk", "sci-fi", "AI"], "synopsis": "Case, a washed-up hacker, is hired for a final mission in a future dominated by corporations and cybernetics. Alongside a street samurai named Molly, he uncovers a plot involving powerful artificial intelligences. The novel coined many cyberpunk tropes and remains a defining work of the genre. It also predicted a globally connected virtual reality."}
{"index": {}}
{"title": "The Left Hand of Darkness", "author": "Ursula K. Le Guin", "publication_year": 1969, "rating": 4.6, "genre": ["sci-fi", "gender", "political"], "synopsis": "An envoy from Earth visits the planet Gethen, where the inhabitants are ambisexual and have no fixed gender. He becomes entangled in political intrigue and forms a deep bond with a native companion. Le Guin\u2019s novel challenges perceptions of gender, trust, and the human condition in alien environments. It is considered a landmark feminist sci-fi work."}
{"index": {}}
{"title": "Snow Crash", "author": "Neal Stephenson", "publication_year": 1992, "rating": 4.3, "genre": ["cyberpunk", "sci-fi", "satire"], "synopsis": "In a future America ruled by corporate enclaves, Hiro Protagonist delivers pizza and hacks the Metaverse. When a new drug called Snow Crash threatens the minds of users, Hiro must uncover its origins. The novel blends satire, linguistics, and fast-paced action in a vividly realized cyberpunk world. It\u2019s one of the early depictions of virtual reality."}
{"index": {}}
{"title": "Dune 6", "author": "Frank Herbert", "publication_year": 1971, "rating": 4.4, "genre": ["sci-fi", "space", "politics"], "synopsis": "On the desert planet Arrakis, Paul Atreides is thrust into a deadly struggle over spice, the universe\u2019s most valuable resource. As he embraces his destiny, he must navigate betrayal, prophecy, and war. This novel explores environmentalism, power, and religious mysticism across an epic scope. It is one of the most influential sci-fi works of all time. This version includes updated content for testing search variations."}
{"index": {}}
{"title": "Snow Crash 7", "author": "Neal Stephenson", "publication_year": 1999, "rating": 4.1, "genre": ["cyberpunk", "sci-fi", "satire"], "synopsis": "In a future America ruled by corporate enclaves, Hiro Protagonist delivers pizza and hacks the Metaverse. When a new drug called Snow Crash threatens the minds of users, Hiro must uncover its origins. The novel blends satire, linguistics, and fast-paced action in a vividly realized cyberpunk world. It\u2019s one of the early depictions of virtual reality. This version includes updated content for testing search variations."}
{"index": {}}
{"title": "Snow Crash 7 8", "author": "Neal Stephenson", "publication_year": 2007, "rating": 4.7, "genre": ["cyberpunk", "sci-fi", "satire"], "synopsis": "In a future America ruled by corporate enclaves, Hiro Protagonist delivers pizza and hacks the Metaverse. When a new drug called Snow Crash threatens the minds of users, Hiro must uncover its origins. The novel blends satire, linguistics, and fast-paced action in a vividly realized cyberpunk world. It\u2019s one of the early depictions of virtual reality. This version includes updated content for testing search variations. This version includes updated content for testing search variations."}
{"index": {}}
{"title": "The Left Hand of Darkness 9", "author": "Ursula K. Le Guin", "publication_year": 1978, "rating": 4.6, "genre": ["sci-fi", "gender", "political"], "synopsis": "An envoy from Earth visits the planet Gethen, where the inhabitants are ambisexual and have no fixed gender. He becomes entangled in political intrigue and forms a deep bond with a native companion. Le Guin\u2019s novel challenges perceptions of gender, trust, and the human condition in alien environments. It is considered a landmark feminist sci-fi work. This version includes updated content for testing search variations."}
{"index": {}}
{"title": "Dune 10", "author": "Frank Herbert", "publication_year": 1965, "rating": 4.4, "genre": ["sci-fi", "space", "politics"], "synopsis": "On the desert planet Arrakis, Paul Atreides is thrust into a deadly struggle over spice, the universe\u2019s most valuable resource. As he embraces his destiny, he must navigate betrayal, prophecy, and war. This novel explores environmentalism, power, and religious mysticism across an epic scope. It is one of the most influential sci-fi works of all time. This version includes updated content for testing search variations."}
{"index": {}}
{"title": "Snow Crash 11", "author": "Neal Stephenson", "publication_year": 1993, "rating": 4.0, "genre": ["cyberpunk", "sci-fi", "satire"], "synopsis": "In a future America ruled by corporate enclaves, Hiro Protagonist delivers pizza and hacks the Metaverse. When a new drug called Snow Crash threatens the minds of users, Hiro must uncover its origins. The novel blends satire, linguistics, and fast-paced action in a vividly realized cyberpunk world. It\u2019s one of the early depictions of virtual reality. This version includes updated content for testing search variations."}
{"index": {}}
{"title": "Snow Crash 11 12", "author": "Neal Stephenson", "publication_year": 1995, "rating": 4.4, "genre": ["cyberpunk", "sci-fi", "satire"], "synopsis": "In a future America ruled by corporate enclaves, Hiro Protagonist delivers pizza and hacks the Metaverse. When a new drug called Snow Crash threatens the minds of users, Hiro must uncover its origins. The novel blends satire, linguistics, and fast-paced action in a vividly realized cyberpunk world. It\u2019s one of the early depictions of virtual reality. This version includes updated content for testing search variations. This version includes updated content for testing search variations."}
{"index": {}}
{"title": "The Left Hand of Darkness 9 13", "author": "Ursula K. Le Guin", "publication_year": 1981, "rating": 4.9, "genre": ["sci-fi", "gender", "political"], "synopsis": "An envoy from Earth visits the planet Gethen, where the inhabitants are ambisexual and have no fixed gender. He becomes entangled in political intrigue and forms a deep bond with a native companion. Le Guin\u2019s novel challenges perceptions of gender, trust, and the human condition in alien environments. It is considered a landmark feminist sci-fi work. This version includes updated content for testing search variations. This version includes updated content for testing search variations."}
{"index": {}}
{"title": "Snow Crash 7 14", "author": "Neal Stephenson", "publication_year": 2003, "rating": 4.7, "genre": ["cyberpunk", "sci-fi", "satire"], "synopsis": "In a future America ruled by corporate enclaves, Hiro Protagonist delivers pizza and hacks the Metaverse. When a new drug called Snow Crash threatens the minds of users, Hiro must uncover its origins. The novel blends satire, linguistics, and fast-paced action in a vividly realized cyberpunk world. It\u2019s one of the early depictions of virtual reality. This version includes updated content for testing search variations. This version includes updated content for testing search variations."}
{"index": {}}
{"title": "The Left Hand of Darkness 9 13 15", "author": "Ursula K. Le Guin", "publication_year": 1986, "rating": 4.1, "genre": ["sci-fi", "gender", "political"], "synopsis": "An envoy from Earth visits the planet Gethen, where the inhabitants are ambisexual and have no fixed gender. He becomes entangled in political intrigue and forms a deep bond with a native companion. Le Guin\u2019s novel challenges perceptions of gender, trust, and the human condition in alien environments. It is considered a landmark feminist sci-fi work. This version includes updated content for testing search variations. This version includes updated content for testing search variations. This version includes updated content for testing search variations."}
{"index": {}}
{"title": "Snow Crash 7 16", "author": "Neal Stephenson", "publication_year": 2005, "rating": 4.5, "genre": ["cyberpunk", "sci-fi", "satire"], "synopsis": "In a future America ruled by corporate enclaves, Hiro Protagonist delivers pizza and hacks the Metaverse. When a new drug called Snow Crash threatens the minds of users, Hiro must uncover its origins. The novel blends satire, linguistics, and fast-paced action in a vividly realized cyberpunk world. It\u2019s one of the early depictions of virtual reality. This version includes updated content for testing search variations. This version includes updated content for testing search variations."}
{"index": {}}
{"title": "Snow Crash 11 12 17", "author": "Neal Stephenson", "publication_year": 2002, "rating": 4.6, "genre": ["cyberpunk", "sci-fi", "satire"], "synopsis": "In a future America ruled by corporate enclaves, Hiro Protagonist delivers pizza and hacks the Metaverse. When a new drug called Snow Crash threatens the minds of users, Hiro must uncover its origins. The novel blends satire, linguistics, and fast-paced action in a vividly realized cyberpunk world. It\u2019s one of the early depictions of virtual reality. This version includes updated content for testing search variations. This version includes updated content for testing search variations. This version includes updated content for testing search variations."}
{"index": {}}
{"title": "The Left Hand of Darkness 9 13 18", "author": "Ursula K. Le Guin", "publication_year": 1989, "rating": 4.9, "genre": ["sci-fi", "gender", "political"], "synopsis": "An envoy from Earth visits the planet Gethen, where the inhabitants are ambisexual and have no fixed gender. He becomes entangled in political intrigue and forms a deep bond with a native companion. Le Guin\u2019s novel challenges perceptions of gender, trust, and the human condition in alien environments. It is considered a landmark feminist sci-fi work. This version includes updated content for testing search variations. This version includes updated content for testing search variations. This version includes updated content for testing search variations."}
{"index": {}}
{"title": "Dune 19", "author": "Frank Herbert", "publication_year": 1974, "rating": 4.2, "genre": ["sci-fi", "space", "politics"], "synopsis": "On the desert planet Arrakis, Paul Atreides is thrust into a deadly struggle over spice, the universe\u2019s most valuable resource. As he embraces his destiny, he must navigate betrayal, prophecy, and war. This novel explores environmentalism, power, and religious mysticism across an epic scope. It is one of the most influential sci-fi works of all time. This version includes updated content for testing search variations."}
{"index": {}}
{"title": "Dune 6 20", "author": "Frank Herbert", "publication_year": 1971, "rating": 4.5, "genre": ["sci-fi", "space", "politics"], "synopsis": "On the desert planet Arrakis, Paul Atreides is thrust into a deadly struggle over spice, the universe\u2019s most valuable resource. As he embraces his destiny, he must navigate betrayal, prophecy, and war. This novel explores environmentalism, power, and religious mysticism across an epic scope. It is one of the most influential sci-fi works of all time. This version includes updated content for testing search variations. This version includes updated content for testing search variations."}
{"index": {}}
{"title": "Dune 10 21", "author": "Frank Herbert", "publication_year": 1966, "rating": 4.7, "genre": ["sci-fi", "space", "politics"], "synopsis": "On the desert planet Arrakis, Paul Atreides is thrust into a deadly struggle over spice, the universe\u2019s most valuable resource. As he embraces his destiny, he must navigate betrayal, prophecy, and war. This novel explores environmentalism, power, and religious mysticism across an epic scope. It is one of the most influential sci-fi works of all time. This version includes updated content for testing search variations. This version includes updated content for testing search variations."}
{"index": {}}
{"title": "Neuromancer 22", "author": "William Gibson", "publication_year": 1986, "rating": 4.1, "genre": ["cyberpunk", "sci-fi", "AI"], "synopsis": "Case, a washed-up hacker, is hired for a final mission in a future dominated by corporations and cybernetics. Alongside a street samurai named Molly, he uncovers a plot involving powerful artificial intelligences. The novel coined many cyberpunk tropes and remains a defining work of the genre. It also predicted a globally connected virtual reality. This version includes updated content for testing search variations."}
{"index": {}}
{"title": "Snow Crash 7 16 23", "author": "Neal Stephenson", "publication_year": 2008, "rating": 4.8, "genre": ["cyberpunk", "sci-fi", "satire"], "synopsis": "In a future America ruled by corporate enclaves, Hiro Protagonist delivers pizza and hacks the Metaverse. When a new drug called Snow Crash threatens the minds of users, Hiro must uncover its origins. The novel blends satire, linguistics, and fast-paced action in a vividly realized cyberpunk world. It\u2019s one of the early depictions of virtual reality. This version includes updated content for testing search variations. This version includes updated content for testing search variations. This version includes updated content for testing search variations."}
{"index": {}}
{"title": "The Left Hand of Darkness 9 13 24", "author": "Ursula K. Le Guin", "publication_year": 1985, "rating": 4.2, "genre": ["sci-fi", "gender", "political"], "synopsis": "An envoy from Earth visits the planet Gethen, where the inhabitants are ambisexual and have no fixed gender. He becomes entangled in political intrigue and forms a deep bond with a native companion. Le Guin\u2019s novel challenges perceptions of gender, trust, and the human condition in alien environments. It is considered a landmark feminist sci-fi work. This version includes updated content for testing search variations. This version includes updated content for testing search variations. This version includes updated content for testing search variations."}
{"index": {}}
{"title": "Dune 10 21 25", "author": "Frank Herbert", "publication_year": 1971, "rating": 4.0, "genre": ["sci-fi", "space", "politics"], "synopsis": "On the desert planet Arrakis, Paul Atreides is thrust into a deadly struggle over spice, the universe\u2019s most valuable resource. As he embraces his destiny, he must navigate betrayal, prophecy, and war. This novel explores environmentalism, power, and religious mysticism across an epic scope. It is one of the most influential sci-fi works of all time. This version includes updated content for testing search variations. This version includes updated content for testing search variations. This version includes updated content for testing search variations."}
Once populated, this RRF query will show that we do not return the embeddings by default:
POST books/_search
{
"retriever": {
"rrf": {
"retrievers": [
{
"standard": {
"query": {
"match": {
"synopsis": "environmental issues"
}
}
}
},
{
"standard": {
"query": {
"match": {
"synopsis": "environmental issues"
}
}
}
},
{
"standard": {
"query": {
"multi_match": {
"query": "environmental issues",
"fields": ["author", "genre"],
"type": "most_fields"
}
}
}
}
]
}
}
}
Next I upgraded this cluster. I confirmed this behavior with both v9.2.0-BC01-9be773b8.
If you re-run the RRF query, you will see that now, chunks are returned automatically even though they weren't before and we didn't ask for them.
Logs (if relevant)
@Mikep86 Assigning this to you to start as you're on SDH this week but please feel free to re-route as needed.