[Bug]: mpr.materials.summary.search does not work for mp-api-0.45.8

[Jeff-oakley]

Code snippet

What happened?

from mp_api.client.mprester import MPRester

mpr = MPRester('')
all_docs = mpr.materials.summary.search()

The code above can not work for mp_api-0.45.8. I tried to pip install and upgrade through python 3.12 and 3.13. Neither way works. The error messages are like follows:

Retrieving SummaryDoc documents: 
 210579/? [03:03<00:00, 1228.81it/s]
---------------------------------------------------------------------------
ValidationError                           Traceback (most recent call last)
Cell In[4], line 7
      3 from copy import deepcopy
      5 mpr = MPRester('HgxDEuZIOSRIDxNncSMKLVBi5NCk36W3')
----> 7 all_docs = mpr.materials.summary.search()
      9 count = 0
     11 for doc in all_docs:

File ~/miniconda3/envs/py3.12/lib/python3.12/site-packages/mp_api/client/routes/materials/summary.py:367, in SummaryRester.search(self, band_gap, chemsys, crystal_system, density, deprecated, e_electronic, e_ionic, e_total, efermi, elastic_anisotropy, elements, energy_above_hull, equilibrium_reaction_energy, exclude_elements, formation_energy, formula, g_reuss, g_voigt, g_vrh, has_props, has_reconstructed, is_gap_direct, is_metal, is_stable, k_reuss, k_voigt, k_vrh, magnetic_ordering, material_ids, n, num_elements, num_sites, num_magnetic_sites, num_unique_magnetic_sites, piezoelectric_modulus, poisson_ratio, possible_species, shape_factor, spacegroup_number, spacegroup_symbol, surface_energy_anisotropy, theoretical, total_energy, total_magnetization, total_magnetization_normalized_formula_units, total_magnetization_normalized_vol, uncorrected_energy, volume, weighted_surface_energy, weighted_work_function, include_gnome, num_chunks, chunk_size, all_fields, fields, **kwargs)
    359     query_params.update({"batch_id_not_eq": "gnome_r2scan_statics"})
    361 query_params = {
    362     entry: query_params[entry]
    363     for entry in query_params
    364     if query_params[entry] is not None
    365 }
--> 367 return super()._search(
    368     num_chunks=num_chunks,
    369     chunk_size=chunk_size,
    370     all_fields=all_fields,
    371     fields=fields,
    372     **query_params,
    373 )

File ~/miniconda3/envs/py3.12/lib/python3.12/site-packages/mp_api/client/core/client.py:1191, in BaseRester._search(self, num_chunks, chunk_size, all_fields, fields, **kwargs)
   1169 """A generic search method to retrieve documents matching specific parameters.
   1170 
   1171 Arguments:
   (...)   1186     A list of documents.
   1187 """
   1188 # This method should be customized for each end point to give more user friendly,
   1189 # documented kwargs.
-> 1191 return self._get_all_documents(
   1192     kwargs,
   1193     all_fields=all_fields,
   1194     fields=fields,
   1195     chunk_size=chunk_size,
   1196     num_chunks=num_chunks,
   1197 )

File ~/miniconda3/envs/py3.12/lib/python3.12/site-packages/mp_api/client/core/client.py:1264, in BaseRester._get_all_documents(self, query_params, all_fields, fields, chunk_size, num_chunks)
   1250 list_entries = sorted(
   1251     (
   1252         (key, len(entry.split(",")))
   (...)   1259     reverse=True,
   1260 )
   1262 chosen_param = list_entries[0][0] if len(list_entries) > 0 else None
-> 1264 results = self._query_resource(
   1265     query_params,
   1266     fields=fields,
   1267     parallel_param=chosen_param,
   1268     chunk_size=chunk_size,
   1269     num_chunks=num_chunks,
   1270 )
   1272 return results["data"]

File ~/miniconda3/envs/py3.12/lib/python3.12/site-packages/mp_api/client/core/client.py:566, in BaseRester._query_resource(self, criteria, fields, suburl, use_document_model, parallel_param, num_chunks, chunk_size, timeout)
    563     data = {"data": unzipped_data, "meta": {}}
    565     if self.use_document_model:
--> 566         data["data"] = self._convert_to_model(data["data"])
    568     data["meta"]["total_doc"] = len(data["data"])
    569 else:

File ~/miniconda3/envs/py3.12/lib/python3.12/site-packages/mp_api/client/core/client.py:1053, in BaseRester._convert_to_model(self, data)
   1043 def _convert_to_model(self, data: list[dict]):
   1044     """Converts dictionary documents to instantiated MPDataDoc objects.
   1045 
   1046     Args:
   (...)   1051 
   1052     """
-> 1053     raw_doc_list = [self.document_model.model_validate(d) for d in data]  # type: ignore
   1055     if len(raw_doc_list) > 0:
   1056         data_model, set_fields, _ = self._generate_returned_model(raw_doc_list[0])

File ~/miniconda3/envs/py3.12/lib/python3.12/site-packages/pydantic/main.py:705, in BaseModel.model_validate(cls, obj, strict, from_attributes, context, by_alias, by_name)
    699 if by_alias is False and by_name is not True:
    700     raise PydanticUserError(
    701         'At least one of `by_alias` or `by_name` must be set to True.',
    702         code='validate-by-alias-and-name-false',
    703     )
--> 705 return cls.__pydantic_validator__.validate_python(
    706     obj, strict=strict, from_attributes=from_attributes, context=context, by_alias=by_alias, by_name=by_name
    707 )

ValidationError: 8 validation errors for SummaryDoc
deprecation_reasons
  Input should be a valid list [type=list_type, input_value=nan, input_type=float]
    For further information visit https://errors.pydantic.dev/2.11/v/list_type
grain_boundaries
  Input should be a valid list [type=list_type, input_value=nan, input_type=float]
    For further information visit https://errors.pydantic.dev/2.11/v/list_type
es_source_calc_id.function-plain[validate()]
  Value error, Invalid MPID Format [type=value_error, input_value=nan, input_type=float]
    For further information visit https://errors.pydantic.dev/2.11/v/value_error
es_source_calc_id.int
  Input should be a finite number [type=finite_number, input_value=nan, input_type=float]
    For further information visit https://errors.pydantic.dev/2.11/v/finite_number
bandstructure
  Input should be a valid dictionary or instance of BandstructureData [type=model_type, input_value=nan, input_type=float]
    For further information visit https://errors.pydantic.dev/2.11/v/model_type
bulk_modulus
  Input should be a valid dictionary [type=dict_type, input_value=nan, input_type=float]
    For further information visit https://errors.pydantic.dev/2.11/v/dict_type
shear_modulus
  Input should be a valid dictionary [type=dict_type, input_value=nan, input_type=float]
    For further information visit https://errors.pydantic.dev/2.11/v/dict_type
has_reconstructed
  Input should be a valid boolean [type=bool_type, input_value=nan, input_type=float]
    For further information visit https://errors.pydantic.dev/2.11/v/bool_type

Version

mp-api-0.45.8

Which OS?

• MacOS
• Windows
• Linux

Log output

[esoteric-ephemera]

Hi @Jeff-oakley, this should only be a problem with searching without filters, since this bulk downloads all data from AWS OpenData. Also make sure not to post your API key, you should regenerate a new on from your dashboard

@tsmathis looks like the OpenData entries got filled with NaN whenever a value was None.

Right now, a workaround is something like this:

from mp_api.client import MPRester

with MPRester(use_document_model=False) as mpr:
    all_docs = mpr.materials.summary.search()

That will return a list of dictionaries. If you need the document model, then you can post-process them as follows:

from emmet.core.summary import SummaryDoc
import numpy as np

summary_docs = []
for i, doc in enumerate(docs):
    try:
        sd = SummaryDoc(**doc)
        summary_docs += [sd]
    except Exception as exc:
        for k in list(doc):
            if isinstance(doc[k],float) and np.isnan(doc[k]):
                doc[k] = None
        try:
            sd = SummaryDoc(**doc)
            summary_docs += [sd]
        except Exception as exc_2:
            summary_docs += [doc]