Term vectors for nested fields

[Barabanga]

Elasticsearch Version

7.14.2

Installed Plugins

No response

Java Version

openjdk version "1.8.0-262" OpenJDK Runtime Environment (build 1.8.0-262-b10) OpenJDK 64-Bit Server VM (build 25.71-b10, mixed mode)

OS Version

windows 10

Problem Description

Using the mtermvectors api to get the term vectors (with term_statistics) for an artificial document Elasticsearch will ignore the nested fields.

Steps to Reproduce

To reproduce:
Create the index and upload a doc

PUT /my-index-000001
{
  "mappings": {
    "properties": {
      "user": {
        "type": "nested",
        "properties": {
            "first" :  {
                "type": "text"
            },
            "last" :  {
                "type": "text"
            },
            "description":{
                "type": "text"
            }
        }
      }
    }
  }
}
PUT /my-index-000001/_doc/1
{
  "group" : "fans",
  "user" : [
    {
      "first" : "John",
      "last" :  "Smith",
      "description":"first description"
    },
    {
      "first" : "Alice",
      "last" :  "White",      
      "description":"second description"
    }
  ]
}

Request the term vector for an artificial document

GET my-index-000001/_mtermvectors
{ 
   "docs": [ 
    { 
        "doc" : {
            "group":"test",
            "user" : [
                {
                "first" : "John",
                "last" :  "Smith",
                "description":"artificial description"
                }
            ]
        }, 
        "fields": ["*"], 
        "term_statistics":true,
        "positions":false,
        "offsets":false
    }
   ]
}

Response

{
    "docs": [
        {
            "_index": "my-index-000001",
            "_type": "_doc",
            "_version": 0,
            "found": true,
            "took": 0,
            "term_vectors": {
                "group": {
                    "field_statistics": {
                        "sum_doc_freq": 1,
                        "doc_count": 1,
                        "sum_ttf": 1
                    },
                    "terms": {
                        "test": {
                            "term_freq": 1
                        }
                    }
                }
            }
        }
    ]
}

The response does not include the term vector for the nested field.

Logs (if relevant)

No response

[elasticsearchmachine]

Pinging @elastic/es-search (Team:Search)

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[NikhilAdur]

Looks like this limitation where term vectors are not directly accessible for nested fields using the standard _termvectors endpoint. This limitation exists because nested fields are stored as separate Lucene documents, and the term vector API doesn't traverse these nested structures.

Try this:
GET /my-index-000001/_termvectors/1
{
"fields" : ["user.first", "user.last", "user.description"],
"offsets" : true,
"positions" : true,
"term_statistics" : true,
"field_statistics" : true,
"payloads" : false,
"per_field_analyzer" : {
"user.first": "standard",
"user.last": "standard",
"user.description": "standard"
}
}

After reproducing the issue:
Response:
{
"_index": "my-index-000001",
"_id": "1",
"_version": 1,
"found": true,
"took": 0,
"term_vectors": {
"user.description": {
"field_statistics": {
"sum_doc_freq": 4,
"doc_count": 2,
"sum_ttf": 4
},
"terms": {
"description": {
"doc_freq": 2,
"ttf": 2,
"term_freq": 2,
"tokens": [
{
"position": 1,
"start_offset": 6,
"end_offset": 17
},
{
"position": 103,
"start_offset": 25,
"end_offset": 36
}
]
},
"first": {
"doc_freq": 1,
"ttf": 1,
"term_freq": 1,
"tokens": [
{
"position": 0,
"start_offset": 0,
"end_offset": 5
}
]
},
"second": {
"doc_freq": 1,
"ttf": 1,
"term_freq": 1,
"tokens": [
{
"position": 102,
"start_offset": 18,
"end_offset": 24
}
]
}
}
},
"user.first": {
"field_statistics": {
"sum_doc_freq": 2,
"doc_count": 2,
"sum_ttf": 2
},
"terms": {
"alice": {
"doc_freq": 1,
"ttf": 1,
"term_freq": 1,
"tokens": [
{
"position": 101,
"start_offset": 5,
"end_offset": 10
}
]
},
"john": {
"doc_freq": 1,
"ttf": 1,
"term_freq": 1,
"tokens": [
{
"position": 0,
"start_offset": 0,
"end_offset": 4
}
]
}
}
},
"user.last": {
"field_statistics": {
"sum_doc_freq": 2,
"doc_count": 2,
"sum_ttf": 2
},
"terms": {
"smith": {
"doc_freq": 1,
"ttf": 1,
"term_freq": 1,
"tokens": [
{
"position": 0,
"start_offset": 0,
"end_offset": 5
}
]
},
"white": {
"doc_freq": 1,
"ttf": 1,
"term_freq": 1,
"tokens": [
{
"position": 101,
"start_offset": 6,
"end_offset": 11
}
]
}
}
}
}
}