VER: Release 0.6.0

See release notes.

Author: cjdsellers

.github/workflows/release.yml

@@ -6,7 +6,7 @@ on:
   workflow_run:
     workflows:
       - test
-    branches: [ main ]
+    branches: [main]
     types:
       - completed
 
@@ -25,7 +25,7 @@ jobs:
 
       # Python setup
       - name: Set up Python environment
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v4
         with:
           python-version: "3.10"
 
@@ -64,7 +64,7 @@ jobs:
           prerelease: false
 
   publish-sdist:
-    needs: [ tag-release ]
+    needs: [tag-release]
     name: publish-sdist - Python 3.10 (ubuntu-latest)
     runs-on: ubuntu-latest
 
@@ -73,7 +73,7 @@ jobs:
 
       # Python setup
       - name: Set up Python environment
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v4
         with:
           python-version: "3.10"
 
@@ -119,7 +119,7 @@ jobs:
         run: twine upload dist/*
 
   publish-wheel:
-    needs: [ tag-release ]
+    needs: [tag-release]
     name: publish-wheel - Python 3.10 (ubuntu-latest)
     runs-on: ubuntu-latest
 
@@ -128,7 +128,7 @@ jobs:
 
       # Python setup
       - name: Set up Python environment
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v4
         with:
           python-version: "3.10"
 

.github/workflows/test.yml

@@ -4,15 +4,15 @@ name: test
 
 on:
   push:
-    branches: [ main, dev ]
+    branches: [main, dev]
 
 jobs:
   test:
     strategy:
       fail-fast: false
       matrix:
-        os: [ ubuntu-latest, macos-latest, windows-latest ]
-        python-version: [ "3.7", "3.8", "3.9", "3.10" ]
+        os: [ubuntu-latest, macos-latest, windows-latest]
+        python-version: ["3.7", "3.8", "3.9", "3.10"]
     name: build - Python ${{ matrix.python-version }} (${{ matrix.os }})
     runs-on: ${{ matrix.os }}
 
@@ -21,7 +21,7 @@ jobs:
 
       # Python setup
       - name: Set up Python environment
-        uses: actions/setup-python@v2
+        uses: actions/setup-python@v4
         with:
           python-version: ${{ matrix.python-version }}
 

CHANGELOG.md

@@ -1,5 +1,9 @@
 # Changelog
 
+## 0.6.0 - 2022-12-02
+- Added `metadata.get_dataset_condition` method to `Historical` client
+- Upgraded `dbz-python` to `0.2.0`
+
 ## 0.5.0 - 2022-11-07
  - Fixed dataframe columns for derived data schemas (dropped `channel_id`)
  - Fixed `batch.submit_job` requests for `dbz` encoding

README.md

@@ -9,29 +9,29 @@
 The official Python client library for [Databento](https://databento.com).
 
 Key features include:
-- Fast, lightweight access to both live and historical data from [multiple markets](https://docs0.databento.com/knowledge-base/new-users/venues-and-publishers?historical=python&live=python).
-- [Multiple schemas](https://docs0.databento.com/knowledge-base/new-users/list-of-supported-market-data-schemas?historical=python&live=python) such as MBO, MBP, top of book, OHLCV, last sale, and more.
-- [Fully normalized](https://docs0.databento.com/knowledge-base/new-users/normalization?historical=python&live=python), i.e. identical message schemas for both live and historical data, across multiple asset classes.
-- Provides mappings between different symbology systems, including [smart symbology](https://docs0.databento.com/reference-historical/basics/symbology?historical=python&live=python) for futures rollovers.
+- Fast, lightweight access to both live and historical data from [multiple markets](https://docs.databento.com/knowledge-base/new-users/venues-and-publishers?historical=python&live=python).
+- [Multiple schemas](https://docs.databento.com/knowledge-base/new-users/list-of-supported-market-data-schemas?historical=python&live=python) such as MBO, MBP, top of book, OHLCV, last sale, and more.
+- [Fully normalized](https://docs.databento.com/knowledge-base/new-users/normalization?historical=python&live=python), i.e. identical message schemas for both live and historical data, across multiple asset classes.
+- Provides mappings between different symbology systems, including [smart symbology](https://docs.databento.com/reference-historical/basics/symbology?historical=python&live=python) for futures rollovers.
 - [Point-in-time]() instrument definitions, free of look-ahead bias and retroactive adjustments.
-- Reads and stores market data in an extremely efficient file format using [Databento Binary Encoding](https://docs0.databento.com/knowledge-base/new-users/dbz-format?historical=python&live=python).
-- Event-driven [market replay](https://docs0.databento.com/reference-historical/helpers/bento-replay?historical=python&live=python), including at high-frequency order book granularity.
-- Support for [batch download](https://docs0.databento.com/knowledge-base/new-users/historical-data-streaming-vs-batch-download?historical=python&live=python) of flat files.
+- Reads and stores market data in an extremely efficient file format using [Databento Binary Encoding](https://docs.databento.com/knowledge-base/new-users/dbz-format?historical=python&live=python).
+- Event-driven [market replay](https://docs.databento.com/reference-historical/helpers/bento-replay?historical=python&live=python), including at high-frequency order book granularity.
+- Support for [batch download](https://docs.databento.com/knowledge-base/new-users/historical-data-streaming-vs-batch-download?historical=python&live=python) of flat files.
 - Support for [pandas](https://pandas.pydata.org/docs/), CSV, and JSON.
 
 ## Documentation
 The best place to begin is with our [Getting started](https://docs.databento.com/getting-started?historical=python&live=python) guide.
 
 You can find our full client API reference on the [Historical Reference](https://docs.databento.com/reference-historical?historical=python&live=python) and
 [Live Reference](https://docs.databento.com/reference-live?historical=python&live=python) sections of our documentation. See also the
-[Examples](https://docs0.databento.com/examples?historical=python&live=python) section for various tutorials and code samples.
+[Examples](https://docs.databento.com/examples?historical=python&live=python) section for various tutorials and code samples.
 
 ## Requirements
 The library is fully compatible with the latest distribution of Anaconda 3.7 and above.
 The minimum dependencies as found in the `requirements.txt` are also listed below:
 - Python (>=3.7)
 - aiohttp (>=3.7.2)
-- dbz-python (>=0.1.5)
+- dbz-python (>=0.2.0)
 - numpy (>=1.17.0)
 - pandas (>=1.1.3)
 - requests (>=2.24.0)

databento/common/bento.py

@@ -662,7 +662,7 @@ def _build_product_id_index(self) -> Dict[dt.date, Dict[int, str]]:
                         end_date=row["end_date"],
                         native=native,
                         product_id=int(row["symbol"]),
-                    )
+                    ),
                 )
 
         product_id_index: Dict[dt.date, Dict[int, str]] = {}

databento/common/parsing.py

@@ -11,7 +11,7 @@ def enum_or_str_lowercase(
     param: str,
 ) -> str:
     """
-    Return the given value parsed to a lowercase string if possible.
+    Return the given value parsed to a lowercase string.
 
     Parameters
     ----------
@@ -44,7 +44,7 @@ def maybe_enum_or_str_lowercase(
     param: str,
 ) -> Optional[str]:
     """
-    Return the given value parsed to a lowercase string if possible.
+    Return the given value parsed to a lowercase string (if not `None`).
 
     Parameters
     ----------
@@ -68,6 +68,68 @@ def maybe_enum_or_str_lowercase(
     return enum_or_str_lowercase(value, param)
 
 
+def enum_or_str_uppercase(
+    value: Union[Enum, str],
+    param: str,
+) -> str:
+    """
+    Return the given value parsed to an uppercase string.
+
+    Parameters
+    ----------
+    value : Enum or str
+        The value to parse.
+    param : str
+        The name of the parameter being validated (for any error message).
+
+    Returns
+    -------
+    str
+
+    Raises
+    ------
+    TypeError
+        If value is not of type NoneType, Enum or str.
+
+    """
+    if isinstance(value, Enum):
+        return value.value.upper()
+    elif isinstance(value, str):
+        if not value.isspace():
+            return value.upper()
+
+    raise TypeError(f"invalid `{param}` type, was {type(value)}.")
+
+
+def maybe_enum_or_str_uppercase(
+    value: Optional[Union[Enum, str]],
+    param: str,
+) -> Optional[str]:
+    """
+    Return the given value parsed to an uppercase string (if not `None`).
+
+    Parameters
+    ----------
+    value : Enum or str, optional
+        The value to parse.
+    param : str
+        The name of the parameter being validated (for any error message).
+
+    Returns
+    -------
+    str or ``None``
+
+    Raises
+    ------
+    TypeError
+        If value is not of type NoneType, Enum or str.
+
+    """
+    if value is None:
+        return value
+    return enum_or_str_uppercase(value, param)
+
+
 def values_list_to_string(
     values: Union[Iterable[str], str],
 ) -> str:
@@ -116,6 +178,7 @@ def maybe_values_list_to_string(
 
 def maybe_symbols_list_to_string(
     symbols: Optional[Union[Iterable[str], str]],
+    stype_in: SType,
 ) -> Optional[str]:
     """
     Concatenate a symbols string or iterable of symbol strings (if not None).
@@ -124,21 +187,31 @@ def maybe_symbols_list_to_string(
     ----------
     symbols : iterable of str or str, optional
         The symbols to concatenate.
+    stype_in : SType
+        The input symbology type for the request.
 
     Returns
     -------
     str or ``None``
 
     """
     if symbols is None:
-        return None  # All symbols
-
-    if isinstance(symbols, str):
-        return symbols.strip().rstrip(",").upper()
-    elif isinstance(symbols, Iterable):
-        return ",".join(symbols).strip().upper()
-    else:
-        raise TypeError(f"invalid symbols type, was {type(symbols)}")
+        return None  # Full universe
+
+    symbols_list = symbols.split(",") if isinstance(symbols, str) else list(symbols)
+    cleaned_symbols: List[str] = []
+    for symbol in symbols_list:
+        if not symbol:
+            continue
+        symbol = symbol.strip().upper()
+        if stype_in == SType.SMART:
+            pieces: List[str] = symbol.split(".")
+            if len(pieces) == 3:
+                symbol = f"{pieces[0]}.{pieces[1].lower()}.{pieces[2]}"
+
+        cleaned_symbols.append(symbol)
+
+    return ",".join(cleaned_symbols)
 
 
 def maybe_date_to_string(value: Optional[Union[date, str]]) -> Optional[str]:

databento/historical/api/metadata.py

@@ -4,7 +4,7 @@
 import pandas as pd
 from databento.common.enums import Dataset, Encoding, FeedMode, Schema, SType
 from databento.common.parsing import (
-    enum_or_str_lowercase,
+    enum_or_str_uppercase,
     maybe_date_to_string,
     maybe_datetime_to_string,
     maybe_enum_or_str_lowercase,
@@ -107,7 +107,7 @@ def list_schemas(
 
         """
         params: List[Tuple[str, Optional[str]]] = [
-            ("dataset", enum_or_str_lowercase(dataset, "dataset")),
+            ("dataset", enum_or_str_uppercase(dataset, "dataset")),
             ("start_date", maybe_date_to_string(start_date)),
             ("end_date", maybe_date_to_string(end_date)),
         ]
@@ -152,7 +152,7 @@ def list_fields(
         validate_maybe_enum(encoding, Encoding, "encoding")
 
         params: List[Tuple[str, str]] = [
-            ("dataset", enum_or_str_lowercase(dataset, "dataset")),
+            ("dataset", enum_or_str_uppercase(dataset, "dataset")),
             ("schema", maybe_enum_or_str_lowercase(schema, "schema")),
             ("encoding", maybe_enum_or_str_lowercase(encoding, "encoding")),
         ]
@@ -229,7 +229,7 @@ def list_unit_prices(
         validate_maybe_enum(mode, FeedMode, "mode")
 
         params: List[Tuple[str, Optional[str]]] = [
-            ("dataset", enum_or_str_lowercase(dataset, "dataset")),
+            ("dataset", enum_or_str_uppercase(dataset, "dataset")),
             ("mode", maybe_enum_or_str_lowercase(mode, "mode")),
             ("schema", maybe_enum_or_str_lowercase(schema, "schema")),
         ]
@@ -241,6 +241,46 @@ def list_unit_prices(
         )
         return response.json()
 
+    def get_dataset_condition(
+        self,
+        dataset: Union[Dataset, str],
+        start_date: Union[date, str],
+        end_date: Union[date, str],
+    ) -> Dict[str, Any]:
+        """
+        Get the dataset condition from Databento.
+
+        Makes a `GET /metadata.get_dataset_condition` HTTP request.
+
+        Use this method to discover data availability and qualility.
+
+        Parameters
+        ----------
+        dataset : Dataset or str
+            The dataset code (string identifier) for the request.
+        start_date : date or str
+            The start date (UTC) for the request range.
+        end_date : date or str
+            The end date (UTC) for the request range.
+
+        Returns
+        -------
+        Dict[str, Any]
+
+        """
+        params: List[Tuple[str, Optional[str]]] = [
+            ("dataset", enum_or_str_uppercase(dataset, "dataset")),
+            ("start_date", str(pd.to_datetime(start_date).date())),
+            ("end_date", str(pd.to_datetime(end_date).date())),
+        ]
+
+        response: Response = self._get(
+            url=self._base_url + ".get_dataset_condition",
+            params=params,
+            basic_auth=True,
+        )
+        return response.json()
+
     def get_record_count(
         self,
         dataset: Union[Dataset, str],
@@ -290,8 +330,8 @@ def get_record_count(
         validate_enum(stype_in, SType, "stype_in")
 
         params: List[Tuple[str, Optional[str]]] = [
-            ("dataset", enum_or_str_lowercase(dataset, "dataset")),
-            ("symbols", maybe_symbols_list_to_string(symbols)),
+            ("dataset", enum_or_str_uppercase(dataset, "dataset")),
+            ("symbols", maybe_symbols_list_to_string(symbols, SType(stype_in))),
             ("schema", Schema(schema).value),
             ("start", maybe_datetime_to_string(start)),
             ("end", maybe_datetime_to_string(end)),

databento/historical/api/symbology.py

@@ -3,7 +3,11 @@
 
 import pandas as pd
 from databento.common.enums import SType
-from databento.common.parsing import enum_or_str_lowercase, maybe_symbols_list_to_string
+from databento.common.parsing import (
+    enum_or_str_lowercase,
+    enum_or_str_uppercase,
+    maybe_symbols_list_to_string,
+)
 from databento.historical.api import API_VERSION
 from databento.historical.http import BentoHttpAPI
 from requests import Response
@@ -58,8 +62,8 @@ def resolve(
 
         """
         params: List[Tuple[str, Optional[str]]] = [
-            ("dataset", enum_or_str_lowercase(dataset, "dataset")),
-            ("symbols", maybe_symbols_list_to_string(symbols)),
+            ("dataset", enum_or_str_uppercase(dataset, "dataset")),
+            ("symbols", maybe_symbols_list_to_string(symbols, SType(stype_in))),
             ("stype_in", enum_or_str_lowercase(stype_in, "stype_in")),
             ("stype_out", enum_or_str_lowercase(stype_out, "stype_out")),
             ("start_date", str(pd.to_datetime(start_date).date())),