Allow env vars to be use global alias scoped within nested configs

[inickt]

We have a case where we would like to leverage validation_alias in our configs to be the value of the environment variable it reads from, ignoring any nesting for the configuration object.

Similar request threads:
pydantic/pydantic#8989
pydantic/pydantic#8221

Originally this issue was filed as a regression with 2.12, but this was due to me using 2.11 incorrectly, so keeping the original issue below:

---

Hi! We seem to be hitting a regression with config overrides when updated from 2.11 to 2.12. We have a few layers of config sources that we use to build our config objects.

We set our order to init > env > dotenv > multiple toml files.

We also use validation_alias to make our env variables more explicit (especially when deeply nested). So we expect our env/dotenv sources to use the validation_alias, but all other configs would use the normal field name.

For the regression, if a value is set in a nested object (top level seems to work as expected) in our toml source, we are now unable to override it with environment variables. However, init overrides still work. This leads me to think it is a regression in #688 for fixing #180 (fyi @chbndrhnns @hramezani).

The following is a simple repro for the regression:

from typing import Annotated, ClassVar, override

from pydantic import Field
from pydantic_settings import (
    BaseSettings,
    PydanticBaseSettingsSource,
    SettingsConfigDict,
    TomlConfigSettingsSource,
)

with open(".env", "w") as f:
    _ = f.write("""\
CONFIG_API_KEY=dotenv-api-key
""")

with open("config.toml", "w") as f:
    _ = f.write("""\
[api]
api_key = "api-key-from-toml"
""")


class ApiConfig(BaseSettings):
    model_config: ClassVar[SettingsConfigDict] = SettingsConfigDict(
        env_file=".env",
        populate_by_name=True,
        extra="ignore",
    )

    api_key: Annotated[str, Field(validation_alias="CONFIG_API_KEY")]


class Config(BaseSettings):
    model_config: ClassVar[SettingsConfigDict] = SettingsConfigDict(
        env_file=".env",
        populate_by_name=True,
        extra="ignore",
    )

    api: ApiConfig

    @override
    @classmethod
    def settings_customise_sources(
        cls,
        settings_cls: type[BaseSettings],
        init_settings: PydanticBaseSettingsSource,
        env_settings: PydanticBaseSettingsSource,
        dotenv_settings: PydanticBaseSettingsSource,
        file_secret_settings: PydanticBaseSettingsSource,
    ) -> tuple[PydanticBaseSettingsSource, ...]:
        return (
            init_settings,
            env_settings,
            dotenv_settings,
            TomlConfigSettingsSource(settings_cls, "config.toml"),
        )


print(Config())
# 2.11: api=ApiConfig(api_key='dotenv-api-key') [correct]
# 2.12: api=ApiConfig(api_key='api-key-from-toml') [wrong]

print(Config(api=ApiConfig(api_key="init-api")))
# 2.11: api=ApiConfig(api_key='api-key-from-toml') [wrong]
# 2.12: api=ApiConfig(api_key='init-api') [correct]

Thanks for the help!

[hramezani]

Thanks @inickt for this issue.

print(Config())
2.11: api=ApiConfig(api_key='dotenv-api-key') [correct]
2.12: api=ApiConfig(api_key='api-key-from-toml') [wrong]

Why do you expect dotenv-api-key? You initialize Config() in your code. this is the value collected by pydantic-setting from different sources:

{'api': {'api_key': 'api-key-from-toml'}, 'config_api_key': 'dotenv-api-key'}
{'api_key': 'api-key-from-toml'} 

You will see two lines because your nested model inherits from pydantic_settings.BaseSettings instead of pydantic.BaseModel and this is not the right approach. check the doc

I think the 'api-key-from-toml' is the right value because you are initializing Config model and pydantic-settings looks for collecting the value for api field.

I think you will get the same result in 2.11 if your ApiConfig model inherits from BaseModel.

BTW, if you still want your ApiConfig model inherits from BaseSettings, you need to change your .env file value to API__CONFIG_API_KEY=dotenv-api-key and add env_nested_delimiter="__" to your Config model model_config

Note: I would recommend changing your nested model(ApiConfig) and inherit from BaseSettings.

[inickt]

@hramezani Wow I had completely missed the section on nested models needing to inherit from BaseModel, sorry about that!

Is there any way of having validation_alias/environment variables be "global", instead of the env variable being based on its path within the config? So for the original example, I want the env variable to always be CONFIG_API_KEY without worrying of where it is nested in the original model.

[inickt]

It seems what I am looking for might not be possible?
pydantic/pydantic#8989
pydantic/pydantic#8221
They both suggest inheriting from BaseSettings but that isn't suggested

[hramezani]

@inickt yeah, it is not possible because pydantic-settings loops over the model fields and you can provide values for nested model by the root model field.

The standard approach is to use the env_nested_delimiter config and not inherit from BaseSettings

[inickt]

@hramezani Would you consider a feature request for adding global env variables? Seems like this has come up a few times before.

For context of why we want this behavior:

• We have an external provider that generates a set of environment variables– we load these in one of our nested configs, but ideally we wouldn't have to manipulate those variables/we would like to treat the nesting as an implementation detail so it is hidden at the env variable layer
• We have a few API key secrets we inject through env variables– some of them are used in lists of config objects for the same provider, but in different parts of the config. Ideally we would only read those from one single key.

I was looking to see if I could write my own settings provider, but it mucks around a bit more in pydantic details than I am currently comfortable with. I am not sure if much of EnvSettingsSource is able to be reused or not.

At a high level the way I would imagine this would work is it traverses recursively the entire model, and at each level it could see the fields in the current model, and see if any of them (or their aliases) match any of the env variables.

[hramezani]

@inickt I think it would be hard to implement this as you need it for a nested model. As I said before, pydantic-settings only considers the root model fields and collects values for them.

I am ok to keep the issue open as a feature request. Unfortunately, I don't have time to work on this.

Note: the patch shouldn't be too complicated

[inickt]

@hramezani thanks! I updated the title/description a bit to be slightly clearer.

We are okay pinning to 2.11 in the short term (with our nested BaseSettings hack), but I would like to help contribute a patch if you think there is a straightforward way of doing it.

Couple questions:

1. Where in the source do you think the core of this patch will live? Just clarifying between "I think it would be hard to implement" and "the patch shouldn't be too complicated"– did you just mean hard to implement as a. consumer, but not too bed within the lib itself?
2. Behavior wise, is my current intended approach of only looking at aliases okay, or does it make more sense to match on all field values? I am not sure which would be more expected for Pydantic, I am assuming matching on all field values is probably better which I am fine with
3. This should probably be opt-in, so a new value in SettingsConfigDict? I don't think there is anything that would make sense with overriding env_nested_delimiter or env_nested_max_split, unless you have a thought.

Thanks for the feedback!

[hramezani]

pydantic-settings collects values from different sources and provides them to pydantic. Consider the following example in pydantic:

from pydantic import BaseModel, Field

class NestedModel(BaseModel):
    name: str = Field(validation_alias='NAME_ALIAS')


class MainModel(BaseModel):
    nested: NestedModel


print(MainModel.model_validate({'NAME_ALIAS': 'example'}))

This will raise a validation error because pydantic expects the value to be nested. so, we need to find the value based on the nested field alias and then include it under the root field in the final dict.

Where in the source do you think the core of this patch will live? Just clarifying between "I think it would be hard to implement" and "the patch shouldn't be too complicated"– did you just mean hard to implement as a. consumer, but not too bed within the lib itself?

we have a complex logic in def explode_env_vars and def next_field that tries to extract the nested values. probably you need to take a look there.

Behavior wise, is my current intended approach of only looking at aliases okay, or does it make more sense to match on all field values? I am not sure which would be more expected for Pydantic, I am assuming matching on all field values is probably better which I am fine with

I think the alias is okay. because we have a good support of fetching nested values with env_nested_delimiter.

• This should probably be opt-in, so a new value in SettingsConfigDict? I don't think there is anything that would make sense with overriding env_nested_delimiter or env_nested_max_split, unless you have a thought.

Yes