Accessing LTR/STTB/CMR

[joe-pitt]

Lots of important information is stored in the variables LTR, STTB and CMR. This data is retrieved by dobj for the "Obspack" .nc files, e.g.:

from icoscp.dobj import Dobj
dobj = Dobj("https://meta.icos-cp.eu/objects/ILHK0NhgL5-yyk57BxgSzuxf")
dobj.get()

However, for most other files this info isn't retrieved, e.g.:

dobj = Dobj("https://meta.icos-cp.eu/objects/tpXGpHAV7VHxrypNTmp__rMN")
dobj.get()

If you click on the url in the second example and download the ASCII file, the LTR, STTB and CMR data is there. Could the option to retrieve this data be added?

[ZogopZ]

Thank you for bringing this to our attention, @joe-pitt! We’re currently looking into it and will update you as soon as we have more information.

[joe-pitt]

On a similar note, for the FastTrack and EYE-AVE-PAR files (e.g. https://meta.icos-cp.eu/objects/hI04Z_fR5-XvEr5ib-3c_RT9 and https://meta.icos-cp.eu/objects/ASsR0CVUspZZviADW6fH8bU_) it seems like Stdev is missing too.

[ukarst]

Thanks for your feedback. Indeed, it seems that for some data types the Stdev is not part of the "previewable vaiables" and therefore not available using our icoscp python library. We will follow up on this right after the Christmas break.

[ukarst]

I have now traced back the issue with Stdev and other uncertainty measures not being accessible using the icoscp or icoscp_core python libraries.
The reason for this is somewhat complex and specific for our data/metadata system:

• Only a selection of variables in a dataset is previewable and this can differ for different data type definitions. The list of previewable variables can be found on the landing page of a dataset.
• Only previewable variables can be accessed using the icoscp and icoscp_core libraries.
• Which variables are previewable is defined in the dataset specifications.
• The L2 and NRT data share the same (tracer-specific) dataset specification, e.g. https://meta.icos-cp.eu/resources/cpmeta/atcN2oMoleFracTimeSer. This definition was created before the different uncertainty columns were added in the csv files. Therefore only Stdev is previewable, although the csv files nowadays contain more uncertainty measures.
• The FastTrack and EYE-AVE-PAR data use a dataset specification that allows to have timeseries of several tracers in one file or to use the same dataset specification for different tracers, e.g. https://meta.icos-cp.eu/resources/cpmeta/atmGhgTimeSeries. In this case the stdev columns are tracer specific.
• The ObsPack data in netcdf format has the "newest" dataset specification https://meta.icos-cp.eu/resources/cpmeta/cpObspackN2oDataset, which includes SMR, STTB, LTR and standard deviation.

Updating all existing L2 and NRT datsets to include SMR, STTB and LTR would require quite some effort. Therefore, the preferred option to get all uncertainty components would be to access the ICOS data from the ObsPack collection.

@joe-pitt, thanks again for bringing this to our attention. We appreciate your feedback from the perspective of a user of our data and tools.

[joe-pitt]

Thanks for looking into this! I can see that this wouldn't be worth the effort for the L2 and NRT datasets - I will use the Obspack collection in that case. But is there any chance the specification for the FastTrack and EYE-AVE-PAR files could be changed? Without the stdev and uncertainty these data really can't be used in any inversions, and the only way to get at these currently would appear to be manually downloading the files and using some separate code to read them in.

[ukarst]

Good point!
As this would require an executive decision, I will bring this up with my colleagues.

[joe-pitt]

That would be great thanks!

[atverm]

You can use the most recent netcdf Obspack time series that is a composition of ICOS L2 , Fasttrack L1.5 and NRT L1 data, it includes the three uncertainty columns for SMR, STTB, LTR as available for L2 and L1.5. And Stdev but that I would not use for data selection. The data is listed here:

[(https://data.icos-cp.eu/portal/#{"filterCategories":{"theme":["atmosphere"],"format":["netcdfTimeSeries"]}})]

[joe-pitt]

Oh yeah I didn't spot that the Fasttrack data was in there. In that case I guess I can live with the situation as it currently is (I will just download the EYE-AVE-PAR files and read them in), but for future datasets I think it would be nice to include as many columns in the spec as possible.

@atverm - am I right in thinking that there will soon be some new netcdf Obspack time series files on the CP running up to January 2025 (with Fasttrack data up to September 2024)? I can see the ASCII files for this Fasttrack release went up on Monday, but I can't see the netcdf files yet

[atverm]

Yes, we will update the Fasttrack product somewhere next week maybe. We are a bit pressed with time to do the processing at the moment.

Alex

[brendan-m-murphy]

Just a note to anyone else with this issue: you can get the full text file using icoscp_core using:

import io
import re
from zipfille import ZipFile

import pandas as pd
from icoscp_core import data

# save to disk
data.save_to_folder(dobj_uri, folder_path)

# or download into memory
# this method only works if if it a zipped text file
_, resp = data.get_file_stream(dobj_uri)

with ZipFile(io.BytesIO(resp.read()), "r") as z:
    with z.open(z.infolist()[0].filename) as f:
        icos_text = f.read().decode("utf-8")

# read with pandas, assuming ascii ATC time series format
# first need to get length of header
pat = re.compile(r"^# HEADER LINES: (\d+)")
m = pat.search(icos_text)
if m:
    header_lines = int(m.group(1))
else:
    header_lines = len([line for line in icos_text.split("\n") if line.startswith("#")])

df = pd.read_csv(io.StringIO(icos_text), skiprows=header_lines-1, sep=";")
df.columns = [col.removeprefix("#") for col in df.columns]  # first col has '#' from comments

You can also download the netCDF time series in a similar way, but you don't need to unzip the file, and you can read the response like xr.open_dataset(io.BytesIO(resp.read())).

As an aside: the obspack label data variable is quite big, but it looks like a fixed string followed by integers. Can ICOS store the fixed string separately and just have a data variable with the varying integers?