HATS-style Euclid Q1 MER Catalogs now available in S3 for testing

[troyraen]

Euclid Q1 MER Catalogs in HATS format are now available from the Fornax Science Console (FSC). This version contains the MER, MER Morphology, and MER Cutouts catalogs combined into a single Apache Parquet dataset, partitioned following the Hierarchical Adaptive Tiling Scheme (HATS). IRSA is preparing to publicly release this data product and has put the files in an S3 bucket for testing purposes, with access restricted to FSC. The path information is:

bucket_name = "irsa-fornax-testdata"                # S3 bucket accessible from FSC for testing prior to release.
mer_hats_prefix = "EUCLID/q1/mer_catalogue/hats"    # Folder containing the hats 'dataset/' dir, a README, etc.

IRSA's tutorial notebook for this dataset is being prepared in Caltech-IPAC/irsa-tutorials#73 (comments welcome). It contains more information about the schema and how to access with lsdb and hats. The basic read with lsdb is:

import lsdb
# Need UPath for the testing bucket. Otherwise hats will ignore the credentials that Fornax
# provides under the hood. Will be unnecessary after the dataset is released in a public bucket.
from upath import UPath

# Read with lsdb.
euclid_lsdb = lsdb.read_hats(UPath(f"s3://{bucket_name}/{mer_hats_prefix}"))

If you have trouble with the dataset/files or want to suggest changes, please comment below.

This issue will remain open until the dataset is released publicly.

[troyraen]

This dataset is superseded by the dataset in #416, which includes twelve Euclid Q1 tables joined on OBJECT_ID.