[BUG] metadata grains failing since 3006.3

Description
metadata grain is not returning successfully since 3006.3. This did work in 3006.2.

Setup
EC2 host in AWS
metadata_server_grains: True in minion config

Steps to Reproduce the behavior

$ sudo salt-call grains.get dynamic:instance-identity --out=json
[CRITICAL] Failed to load grains defined in grain file metadata.metadata in function <LoadedFunc name='metadata.metadata'>, error:
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/__init__.py", line 1150, in grains
    ret = funcs[key](**kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 87, in metadata
    return _search()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 64, in _search
    ret[line] = _search(prefix=os.path.join(prefix, line + "/"))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 53, in _search
    linedata["headers"].get("Content-Type", "text/plain")
KeyError: 'headers'
{
    "local": ""
}

Expected behavior
command should return some valid json, along the lines of:

{
  "accountId" : "xxx",
  "architecture" : "x86_64",
  "availabilityZone" : "us-east-1a",
  "billingProducts" : null,
  "devpayProductCodes" : null,
  "marketplaceProductCodes" : null,
  "imageId" : "ami-xxx",
  "instanceId" : "i-xxx",
  "instanceType" : "t2.large",
  "kernelId" : null,
  "pendingTime" : "2023-05-02T13:54:39Z",
  "privateIp" : "xxx.xxx.xxx.xxx",
  "ramdiskId" : null,
  "region" : "us-east-1",
  "version" : "2017-09-30"
}

Curl commands against the metadata API return expected results.
curl http://169.254.169.254/latest/dynamic/instance-identity/document/

Versions Report

salt --versions-report

(Provided by running salt --versions-report. Please also mention any differences in master/minion versions.)

Salt Version:
          Salt: 3006.3

Python Version:
        Python: 3.10.13 (main, Sep  6 2023, 02:11:27) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: Not Installed
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.13.10
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 22.04.3 jammy
        locale: utf-8
       machine: x86_64
       release: 6.2.0-1009-aws
        system: Linux
       version: Ubuntu 22.04.3 jammy

[welcome]

Hi there! Welcome to the Salt Community! Thank you for making your first contribution. We have a lengthy process for issues and PRs. Someone from the Core Team will follow up as soon as possible. In the meantime, here’s some information that may help as you continue your Salt journey.
Please be sure to review our Code of Conduct. Also, check out some of our community resources including:

• Community Wiki
• Salt’s Contributor Guide
• Join our Community Slack
• IRC on LiberaChat
• Salt Project YouTube channel
• Salt Project Twitch channel

There are lots of ways to get involved in our community. Every month, there are around a dozen opportunities to meet with other contributors and the Salt Core team and collaborate in real time. The best way to keep track is by subscribing to the Salt Community Events Calendar.
If you have additional questions, email us at saltproject@vmware.com. We’re glad you’ve joined our community and look forward to doing awesome things with you!

I have confirmed downgrading to 3006.2 resolved the issue.

Still an issue in 3006.4, though it's no longer throwing a stack trace. It just returns an empty object instead of the correct data.

$ sudo salt-call grains.get dynamic:instance-identity --out=json
{
    "local": ""
}

salt --versions-report

Salt Version:
          Salt: 3006.4

Python Version:
        Python: 3.10.13 (main, Oct  4 2023, 21:54:22) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: unknown
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.13.12
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 22.04.3 jammy
        locale: utf-8
       machine: x86_64
       release: 6.2.0-1009-aws
        system: Linux
       version: Ubuntu 22.04.3 jammy

Just wondering if we are going to see some movement on this.

Some additional info: I verified that metadata is enabled on multiple affected instances, and that IMDSv2 is set to Optional.

$ sudo salt-call grains.get dynamic:instance-identity --out=json
[CRITICAL] Failed to load grains defined in grain file metadata.metadata in function <LoadedFunc name='metadata.metadata'>, error:
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/__init__.py", line 1154, in grains
    ret = funcs[key](**kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 87, in metadata
    return _search()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 64, in _search
    ret[line] = _search(prefix=os.path.join(prefix, line + "/"))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 53, in _search
    linedata["headers"].get("Content-Type", "text/plain")
KeyError: 'headers'
{
    "local": ""
}

salt-call --versions-report

Salt Version:
          Salt: 3006.5

Python Version:
        Python: 3.10.13 (main, Nov 15 2023, 04:34:27) [GCC 11.2.0]

Dependency Versions:
          cffi: 1.14.6
      cherrypy: 18.6.1
      dateutil: 2.8.1
     docker-py: Not Installed
         gitdb: Not Installed
     gitpython: Not Installed
        Jinja2: 3.1.2
       libgit2: Not Installed
  looseversion: 1.0.2
      M2Crypto: Not Installed
          Mako: Not Installed
       msgpack: 1.0.2
  msgpack-pure: Not Installed
  mysql-python: Not Installed
     packaging: 22.0
     pycparser: 2.21
      pycrypto: Not Installed
  pycryptodome: 3.9.8
        pygit2: Not Installed
  python-gnupg: 0.4.8
        PyYAML: 6.0.1
         PyZMQ: 23.2.0
        relenv: 0.14.2
         smmap: Not Installed
       timelib: 0.2.4
       Tornado: 4.5.3
           ZMQ: 4.3.4

System Versions:
          dist: ubuntu 22.04.2 jammy
        locale: utf-8
       machine: x86_64
       release: 6.2.0-1009-aws
        system: Linux
       version: Ubuntu 22.04.2 jammy

[mdschmitt]

So, perhaps @garethgreenaway can weigh in here. Is it possible that the behavior in the most recent commit was bugged..?
f9fa938

[houstonj1]

This seems related to changes salt.utils.http

Nothing about metadata.py has changed from 3006.2 ... 3006.3, but the http util has changes.

Just wondering if we are going to see some movement on this.

Some additional info: I verified that metadata is enabled on multiple affected instances, and that IMDSv2 is set to Optional.

$ sudo salt-call grains.get dynamic:instance-identity --out=json
[CRITICAL] Failed to load grains defined in grain file metadata.metadata in function <LoadedFunc name='metadata.metadata'>, error:
Traceback (most recent call last):
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/__init__.py", line 1154, in grains
    ret = funcs[key](**kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 159, in __call__
    ret = self.loader.run(run_func, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1245, in run
    return self._last_context.run(self._run_as, _func_or_method, *args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/loader/lazy.py", line 1260, in _run_as
    return _func_or_method(*args, **kwargs)
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 87, in metadata
    return _search()
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 64, in _search
    ret[line] = _search(prefix=os.path.join(prefix, line + "/"))
  File "/opt/saltstack/salt/lib/python3.10/site-packages/salt/grains/metadata.py", line 53, in _search
    linedata["headers"].get("Content-Type", "text/plain")
KeyError: 'headers'
{
    "local": ""
}

salt-call --versions-report

linedata is the result of http.query, looking into changes there is next on the list IMO