Conversation
recipes/mursst/recipe.py
Outdated
| # target_root is injected only into certain transforms in pangeo-forge-recipes | ||
| # this is a hacky way to pull it out of the WriteCombinedReference transform | ||
| hacky_way_to_pull = WriteCombinedReference( | ||
| store_name='mursst', | ||
| concat_dims=pattern.concat_dims, | ||
| identical_dims=identical_dims, | ||
| ) |
sbquinlan
force-pushed
the
mursst-kerchunk
branch
from
8cb7925 to
24ca21a
Compare
sbquinlan
force-pushed
the
mursst-kerchunk
branch
from
24ca21a to
8e77f78
Compare
| storage_options=fsspec_open_kwargs, | ||
| ) | ||
| | FilterVars(keep={*pattern.concat_dims, *IDENTICAL_DIMS, *SELECTED_VARS}) | ||
| | WriteCombinedReference( |
There was a problem hiding this comment.
This depends on the main branch of pangeo-forge-recipes
|
Moving this to draft until we can:
|
recipes/mursst/recipe.py
Outdated
| store_name=SHORT_NAME | ||
| ) | ||
| | ConsolidateMetadata() | ||
| # | ValidateDatasetDimensions(expected_dims={'time': None, 'lat': (-90, 90), 'lon': (-180, 180)}) |
There was a problem hiding this comment.
This does not work with the new output of ConsolidateMetadata so will need to revise or remove it.
There was a problem hiding this comment.
Can you try the local version of ConsolidateMetadata?
@beam.ptransform_fn
def ConsolidateMetadata(pcoll: beam.PCollection) -> beam.PCollection:
"""Consolidate metadata into a single .zmetadata field.
See zarr.consolidate_metadata() for details.
"""
def _consolidate(store: zarr.storage.FSStore) -> zarr.storage.FSStore:
zarr.consolidate_metadata(store, path=None)
return store
return pcoll | beam.Map(_consolidate)
It would be nice if it returned FSStore like WriteCombinedReference @norlandrhagen
@abarciauskas-bgse Why is it preferred to write out the zarr and then consolidate the metadata?
There was a problem hiding this comment.
@sbquinlan thanks for keeping track of this PR. I think we want to use transforms maintained by pangeo-forge-recipes when possible. In general, I think you have to write the zarr store before running a consolidate metadata operation but perhaps I am misunderstanding your question.
As it happens, we may be removing consolidate metadata altogether since we need to investigate if it is actually helpful to consolidate metadata for kerchunk reference files (or if someone can point me to where this has already been determined, that would be helpful).
There was a problem hiding this comment.
I think we want to use transforms maintained by pangeo-forge-recipes when possible
The question is basically should we change the committed implementation of ConsolidateMetadata to return the more useful FSStore instead of the less useful ConsolidatedMetadataStore. I can't see ConsolidatedMetadataStore being useful in pangeo-forge. This is more for @norlandrhagen rather than specific to this PR.
In general, I think you have to write the zarr store before running a consolidate metadata operation
Yeah I don't think this is true. It just needs a MutableMapping so you could do it in memory before writing it out, but that's my question basically: is it more performant to consolidate and then write out the references?
remove validation
2 participants
@sharkinsspatial @abarciauskas-bgse