Add mursst lithops example (#475)

* Add files

* Refactoring some things

* Fix some refactoring things

* Update PR number

* Use configure_zarr in helpers

* Remove some env specific configuration in lithops.yaml

Author: abarciauskas-bgse

docs/examples.md

@@ -5,3 +5,4 @@ The following examples demonstrate the use of VirtualiZarr to create virtual dat
 1. [Appending new daily NOAA SST data to Icechunk](https://github.com/zarr-developers/VirtualiZarr/blob/main/examples/append/noaa-cdr-sst.ipynb)
 2. [Parallel reference generation using Coiled Functions](https://github.com/zarr-developers/VirtualiZarr/blob/main/examples/coiled/terraclimate.ipynb)
 3. [Serverless parallel reference generation using Lithops](https://github.com/zarr-developers/VirtualiZarr/tree/main/examples/virtualizarr-with-lithops)
+4. [MUR SST Virtual and Zarr Icechunk Store Generation using Lithops](https://github.com/zarr-developers/VirtualiZarr/tree/main/examples/mursst-icechunk-with-lithops)

docs/releases.rst

@@ -21,6 +21,9 @@ Bug fixes
 Documentation
 ~~~~~~~~~~~~~
 
+- Added MUR SST virtual and zarr icechunk store generation using lithops example.
+  (:pull:`475`) by `Aimee Barciauskas <https://github.com/abarciauskas-bgse>`_.
+
 Internal Changes
 ~~~~~~~~~~~~~~~~
 

examples/mursst-icechunk-with-lithops/Dockerfile

@@ -0,0 +1,55 @@
+# Use AWS Lambda base image for Python 3.11
+FROM public.ecr.aws/lambda/python:3.11
+
+ARG FUNCTION_DIR
+
+# Set working directory
+WORKDIR /var/task
+
+# Update system libraries and install necessary utilities
+RUN yum update -y && \
+    yum install -y wget unzip tar gzip git && \
+    yum clean all
+
+# Install uv package manager and move it to /usr/local/bin
+RUN curl -LsSf https://astral.sh/uv/install.sh | sh && \
+    mv ~/.local/bin/uv /usr/local/bin/uv && \
+    chmod +x /usr/local/bin/uv
+
+# Verify uv installation
+RUN uv --version
+
+RUN uv pip install --upgrade pip wheel six setuptools --system \
+    && uv pip install --upgrade --no-cache-dir --system \
+    awslambdaric \
+    boto3 \
+    redis \
+    httplib2 \
+    requests \
+    numpy \
+    scipy \
+    pandas \
+    pika \
+    kafka-python \
+    cloudpickle \
+    ps-mem \
+    tblib \
+    psutil
+
+# Set environment variables for Lambda
+ENV PYTHONPATH="/var/lang/lib/python3.11/site-packages:${FUNCTION_DIR}"
+
+# Copy and install dependencies from requirements.txt using uv
+COPY requirements.txt /tmp/requirements.txt
+RUN uv pip install --no-cache-dir -r /tmp/requirements.txt --system
+
+# Copy application code
+COPY lithops_lambda.zip ${FUNCTION_DIR}
+RUN unzip lithops_lambda.zip \
+    && rm lithops_lambda.zip \
+    && mkdir handler \
+    && touch handler/__init__.py \
+    && mv entry_point.py handler/
+
+# Set Lambda entry point
+CMD [ "handler.entry_point.lambda_handler" ]

examples/mursst-icechunk-with-lithops/README.md

@@ -0,0 +1,121 @@
+# Lithops Package for MUR SST Data Processing
+
+This package provides functionality for processing MUR SST (Multi-scale Ultra-high Resolution Sea Surface Temperature) data using [Lithops](https://lithops-cloud.github.io/), a framework for serverless computing.
+
+## Environment + Lithops Setup
+
+1. Set up a Python environment. The below example uses [`uv`](https://docs.astral.sh/uv/), but other environment mangers should work as well:
+
+```sh
+uv venv virtualizarr-lithops --python 3.11
+source virtualizarr-lithops/bin/activate
+uv pip install -r requirements.txt
+```
+
+2. Follow the [AWS Lambda Configuration](https://lithops-cloud.github.io/docs/source/compute_config/aws_lambda.html#configuration) instructions, unless you already have an appropriate AWS IAM role to use.
+
+3. Follow the [AWS Credential setup](https://lithops-cloud.github.io/docs/source/compute_config/aws_lambda.html#aws-credential-setup) instructions.
+
+4. Check and modify as necessary compute and storage backends for [lithops](https://lithops-cloud.github.io/docs/source/configuration.html) in `lithops.yaml`.
+
+
+5. Build the lithops lambda runtime if it does not exist in your target AWS environemnt.
+```bash
+export LITHOPS_CONFIG_FILE=$(pwd)/lithops.yaml
+lithops runtime build -b aws_lambda -f Dockerfile vz-runtime
+```
+
+For various reasons, you may want to build the lambda runtime on EC2 (docker can be a resource hog and pushing to ECR is faster, for example). If you wish to use EC2, please see the scripts in `ec2_for_lithops_runtime/` in this directory.
+
+> [!IMPORTANT]
+> If the runtime was created with a different IAM identity, an appropriate `user_id` will need to be included in the lithops configuration under `aws_lamda`.
+
+> [!TIP]
+> You can configure the AWS Lambda architecture via the `architecture` key under `aws_lambda` in the lithops configuration file.
+
+
+6. (Optional) To rebuild the Lithops Lambda runtime image, delete the existing one:
+
+```bash
+lithops runtime delete -b aws_lambda -d virtualizarr-runtime
+```
+
+## Package Structure
+
+The package is organized into the following modules:
+
+- `__init__.py`: Package initialization and exports
+- `config.py`: Configuration settings and constants
+- `models.py`: Data models and structures
+- `url_utils.py`: URL generation and file listing
+- `repo.py`: Icechunk repository management
+- `virtual_datasets.py`: Virtual dataset operations
+- `zarr_operations.py`: Zarr array operations
+- `helpers.py`: Data helpers
+- `lithops_functions.py`: Lithops execution wrappers
+- `cli.py`: Command-line interface
+
+## Usage
+
+### Command-line Interface
+
+The package provides a command-line interface for running various functions:
+
+```bash
+python main.py <function> [options]
+```
+
+Available functions:
+
+- `write_to_icechunk`: Write data to Icechunk
+- `check_data_store_access`: Check access to the data store
+- `calc_icechunk_store_mean`: Calculate the mean of the Icechunk store
+- `calc_original_files_mean`: Calculate the mean of the original files
+- `list_installed_packages`: List installed packages
+
+Options:
+
+- `--start_date`: Start date for data processing (YYYY-MM-DD)
+- `--end_date`: End date for data processing (YYYY-MM-DD)
+- `--append_dim`: Append dimension for writing to Icechunk
+
+### Examples
+
+#### Writing Data to Icechunk
+
+```bash
+python main.py write_to_icechunk --start_date 2022-01-01 --end_date 2022-01-02
+```
+
+#### Calculating the Mean of the Icechunk Store
+
+```bash
+python main.py calc_icechunk_store_mean --start_date 2022-01-01 --end_date 2022-01-31
+```
+
+#### Checking Data Store Access
+
+```bash
+python main.py check_data_store_access
+```
+
+## Programmatic Usage
+
+You can also use the package programmatically:
+
+```python
+from lithops_functions import write_to_icechunk
+
+# Write data to Icechunk
+write_to_icechunk(start_date="2022-01-01", end_date="2022-01-31")
+```
+
+## Testing
+
+To test the package, you can use the provided test functions:
+
+```bash
+python main.py check_data_store_access
+```
+
+This will verify that the package can access the data store.

examples/mursst-icechunk-with-lithops/__init__.py

@@ -0,0 +1,26 @@
+"""
+Lithops package for MUR SST data processing import
+
+This package provides functionality for processing MUR SST data using Lithops,
+a framework for serverless computing import
+"""
+
+from . import (
+    config,
+    data_processing,
+    lithops_functions,
+    models,
+    repo,
+    url_utils,
+    virtual_datasets,
+)
+
+__all__ = [
+    "config",
+    "data_processing",
+    "lithops_functions",
+    "models",
+    "repo",
+    "url_utils",
+    "virtual_datasets",
+]

examples/mursst-icechunk-with-lithops/cli.py

@@ -0,0 +1,79 @@
+"""
+Command-line interface.
+
+This module provides a command-line interface for the package.
+"""
+
+import argparse
+
+from lithops_functions import (
+    lithops_calc_icechunk_store_mean,
+    lithops_calc_original_files_mean,
+    lithops_check_data_store_access,
+    lithops_list_installed_packages,
+    write_to_icechunk,
+)
+
+
+def parse_args():
+    """
+    Parse command-line arguments.
+
+    Returns:
+        The parsed arguments
+    """
+    parser = argparse.ArgumentParser(description="Run lithops functions.")
+    parser.add_argument(
+        "function",
+        choices=[
+            "write_to_icechunk",
+            "check_data_store_access",
+            "calc_icechunk_store_mean",
+            "calc_original_files_mean",
+            "list_installed_packages",
+        ],
+        help="The function to run.",
+    )
+    parser.add_argument(
+        "--start_date",
+        type=str,
+        help="Start date for data processing (YYYY-MM-DD).",
+    )
+    parser.add_argument(
+        "--end_date",
+        type=str,
+        help="End date for data processing (YYYY-MM-DD).",
+    )
+    parser.add_argument(
+        "--append_dim",
+        type=str,
+        help="Append dimension for writing to icechunk.",
+    )
+    return parser.parse_args()
+
+
+def main():
+    """
+    Main entry point for the command-line interface.
+    """
+    args = parse_args()
+    start_date = args.start_date
+    end_date = args.end_date
+    append_dim = args.append_dim
+
+    if args.function == "write_to_icechunk":
+        write_to_icechunk(
+            start_date=start_date, end_date=end_date, append_dim=append_dim
+        )
+    elif args.function == "check_data_store_access":
+        lithops_check_data_store_access()
+    elif args.function == "calc_icechunk_store_mean":
+        lithops_calc_icechunk_store_mean(start_date=start_date, end_date=end_date)
+    elif args.function == "calc_original_files_mean":
+        lithops_calc_original_files_mean(start_date=start_date, end_date=end_date)
+    elif args.function == "list_installed_packages":
+        lithops_list_installed_packages()
+
+
+if __name__ == "__main__":
+    main()

examples/mursst-icechunk-with-lithops/config.py

@@ -0,0 +1,51 @@
+"""
+Configuration settings for MUR SST data processing.
+
+This module contains all the configuration settings and constants used
+throughout the package.
+"""
+
+import fsspec
+
+# S3 filesystem for reading data
+fs_read = fsspec.filesystem("s3", anon=False, skip_instance_cache=True)
+
+# Data source configuration
+base_url = "s3://podaac-ops-cumulus-protected/MUR-JPL-L4-GLOB-v4.1"
+data_vars = ["analysed_sst", "analysis_error", "mask", "sea_ice_fraction"]
+drop_vars = ["dt_1km_data", "sst_anomaly"]
+
+# Storage configuration
+bucket = "nasa-eodc-scratch"
+store_name = "MUR-JPL-L4-GLOB-v4.1-virtual-v1"
+directory = "test"
+
+# Spatial subset configuration
+lat_slice = slice(48.5, 48.7)
+lon_slice = slice(-124.7, -124.5)
+
+# Date range processing dictionary
+date_process_dict = {
+    ("2002-06-30", "2003-09-10"): "virtual_dataset",
+    ("2003-09-11", "2003-09-11"): "zarr",
+    ("2003-09-12", "2021-02-19"): "virtual_dataset",
+    ("2021-02-20", "2021-02-21"): "zarr",
+    ("2021-02-22", "2021-12-23"): "virtual_dataset",
+    ("2021-12-24", "2022-01-26"): "zarr",
+    ("2022-01-27", "2022-11-08"): "virtual_dataset",
+    ("2022-11-09", "2022-11-09"): "zarr",
+    ("2022-11-10", "2023-02-23"): "virtual_dataset",
+    ("2023-02-24", "2023-02-28"): "zarr",
+    ("2023-03-01", "2023-04-21"): "virtual_dataset",
+    ("2023-04-22", "2023-04-22"): "zarr",
+    ("2023-04-23", "2023-09-03"): "virtual_dataset",
+}
+
+zarr_concurrency = 4
+
+mursst_var_chunks = {
+    "analysed_sst": {"time": 1, "lat": 1023, "lon": 2047},
+    "analysis_error": {"time": 1, "lat": 1023, "lon": 2047},
+    "mask": {"time": 1, "lat": 1447, "lon": 2895},
+    "sea_ice_fraction": {"time": 1, "lat": 1447, "lon": 2895},
+}

examples/mursst-icechunk-with-lithops/ec2_for_lithops_runtime/00-create-security-group.sh

@@ -0,0 +1,3 @@
+export SECURITY_GROUP_NAME=XXX
+export VPC_ID=XXX
+aws ec2 create-security-group --group-name $SG_GROUP_NAME --description "security group for ithops runtime builder ec2" --vpc-id $VPC_ID

examples/mursst-icechunk-with-lithops/ec2_for_lithops_runtime/01-launch-ec2.sh

@@ -0,0 +1,14 @@
+# look up the group id created
+export SECURITY_GROUP_ID=XXX
+export YOUR_IP=$(curl -s https://checkip.amazonaws.com)
+export AMI_ID=ami-027951e78de46a00e
+export SSH_KEY_NAME=XXX
+aws ec2 authorize-security-group-ingress --group-id $SECURITY_GROUP_ID --ip-permissions '{"IpProtocol":"tcp","FromPort":22,"ToPort":22,"IpRanges":[{"CidrIp":"'$YOUR_IP'/32"}]}'
+aws ec2 run-instances --image-id $AMI_ID \
+  --instance-type "t3.medium" --key-name $SSH_KEY_NAME \
+    --block-device-mappings '{"DeviceName":"/dev/xvda","Ebs":{"Encrypted":false,"DeleteOnTermination":true,"Iops":3000,"SnapshotId":"snap-01783d80c688baa0f","VolumeSize":30,"VolumeType":"gp3","Throughput":125}}' \
+    --network-interfaces '{"AssociatePublicIpAddress":true,"DeviceIndex":0,"Groups":["'$SECURITY_GROUP_ID'"]}' \
+    --credit-specification '{"CpuCredits":"unlimited"}' \
+    --metadata-options '{"HttpEndpoint":"enabled","HttpPutResponseHopLimit":2,"HttpTokens":"required"}' \
+    --private-dns-name-options '{"HostnameType":"ip-name","EnableResourceNameDnsARecord":true,"EnableResourceNameDnsAAAARecord":false}' \
+    --count "1"