VER: Release 0.32.0

Author: nmacholl

CHANGELOG.md

@@ -1,5 +1,22 @@
 # Changelog
 
+## 0.32.0 - 2024-04-04
+
+#### Enhancements
+- Improved exception messages when multiple `ErrorMsg` are received by the `Live` client
+- Upgraded `databento-dbn` to 0.17.1
+
+#### Bug fixes
+- Removed live session ID parsing to `int`, that could cause a session to fail when
+  nothing was wrong
+
+#### Breaking changes
+- Renamed publishers from deprecated datasets to their respective sources (`XNAS.NLS`
+and `XNYS.TRADES` respectively)
+
+#### Deprecations
+- Deprecated dataset values `FINN.NLS` and `FINY.TRADES`
+
 ## 0.31.1 - 2024-03-20
 
 #### Enhancements

README.md

@@ -10,29 +10,29 @@
 The official Python client library for [Databento](https://databento.com).
 
 Key features include:
-- Fast, lightweight access to both live and historical data from [multiple markets](https://docs.databento.com/knowledge-base/new-users/venues-and-publishers?historical=python&live=python).
-- [Multiple schemas](https://docs.databento.com/knowledge-base/new-users/market-data-schemas?historical=python&live=python) such as MBO, MBP, top of book, OHLCV, last sale, and more.
-- [Fully normalized](https://docs.databento.com/knowledge-base/new-users/normalization?historical=python&live=python), i.e. identical message schemas for both live and historical data, across multiple asset classes.
-- Provides mappings between different symbology systems, including [smart symbology](https://docs.databento.com/api-reference-historical/basics/symbology?historical=python&live=python) for futures rollovers.
+- Fast, lightweight access to both live and historical data from [multiple markets](https://databento.com/docs/knowledge-base/new-users/venues-and-publishers?historical=python&live=python).
+- [Multiple schemas](https://databento.com/docs/knowledge-base/new-users/market-data-schemas?historical=python&live=python) such as MBO, MBP, top of book, OHLCV, last sale, and more.
+- [Fully normalized](https://databento.com/docs/knowledge-base/new-users/normalization?historical=python&live=python), i.e. identical message schemas for both live and historical data, across multiple asset classes.
+- Provides mappings between different symbology systems, including [smart symbology](https://databento.com/docs/api-reference-historical/basics/symbology?historical=python&live=python) for futures rollovers.
 - [Point-in-time]() instrument definitions, free of look-ahead bias and retroactive adjustments.
-- Reads and stores market data in an extremely efficient file format using [Databento Binary Encoding](https://docs.databento.com/knowledge-base/new-users/dbn-encoding?historical=python&live=python).
-- Event-driven [market replay](https://docs.databento.com/api-reference-historical/helpers/bento-replay?historical=python&live=python), including at high-frequency order book granularity.
-- Support for [batch download](https://docs.databento.com/knowledge-base/new-users/stream-vs-batch?historical=python&live=python) of flat files.
+- Reads and stores market data in an extremely efficient file format using [Databento Binary Encoding](https://databento.com/docs/knowledge-base/new-users/dbn-encoding?historical=python&live=python).
+- Event-driven [market replay](https://databento.com/docs/api-reference-historical/helpers/bento-replay?historical=python&live=python), including at high-frequency order book granularity.
+- Support for [batch download](https://databento.com/docs/knowledge-base/new-users/stream-vs-batch?historical=python&live=python) of flat files.
 - Support for [pandas](https://pandas.pydata.org/docs/), CSV, and JSON.
 
 ## Documentation
-The best place to begin is with our [Getting started](https://docs.databento.com/getting-started?historical=python&live=python) guide.
+The best place to begin is with our [Getting started](https://databento.com/docs/getting-started?historical=python&live=python) guide.
 
-You can find our full client API reference on the [Historical Reference](https://docs.databento.com/api-reference-historical?historical=python&live=python) and
-[Live Reference](https://docs.databento.com/reference-live?historical=python&live=python) sections of our documentation. See also the
-[Examples](https://docs.databento.com/examples?historical=python&live=python) section for various tutorials and code samples.
+You can find our full client API reference on the [Historical Reference](https://databento.com/docs/api-reference-historical?historical=python&live=python) and
+[Live Reference](https://databento.com/docs/reference-live?historical=python&live=python) sections of our documentation. See also the
+[Examples](https://databento.com/docs/examples?historical=python&live=python) section for various tutorials and code samples.
 
 ## Requirements
 The library is fully compatible with the latest distribution of Anaconda 3.8 and above.
 The minimum dependencies as found in the `pyproject.toml` are also listed below:
 - python = "^3.8"
 - aiohttp = "^3.8.3"
-- databento-dbn = "0.16.0"
+- databento-dbn = "0.17.1"
 - numpy= ">=1.23.5"
 - pandas = ">=1.5.3"
 - pyarrow = ">=13.0.0"
@@ -79,7 +79,7 @@ array = data.to_ndarray()  # to ndarray
 ```
 
 Note that the API key was also passed as a parameter, which is
-[not recommended for production applications](https://docs.databento.com/knowledge-base/new-users/security-managing-api-keys?historical=python&live=python).
+[not recommended for production applications](https://databento.com/docs/knowledge-base/new-users/security-managing-api-keys?historical=python&live=python).
 Instead, you can leave out this parameter to pass your API key via the `DATABENTO_API_KEY` environment variable:
 
 ```python

databento/common/constants.py

@@ -3,6 +3,7 @@
 from typing import Final
 
 import numpy as np
+from databento_dbn import CbboMsg
 from databento_dbn import ImbalanceMsg
 from databento_dbn import InstrumentDefMsg
 from databento_dbn import InstrumentDefMsgV1
@@ -36,9 +37,16 @@
     Schema.OHLCV_1M: OHLCVMsg,
     Schema.OHLCV_1H: OHLCVMsg,
     Schema.OHLCV_1D: OHLCVMsg,
+    Schema.OHLCV_EOD: OHLCVMsg,
     Schema.STATISTICS: StatMsg,
     Schema.TBBO: MBP1Msg,
     Schema.TRADES: TradeMsg,
+    Schema.CBBO: CbboMsg,
+    Schema.CBBO_1S: CbboMsg,
+    Schema.CBBO_1M: CbboMsg,
+    Schema.TCBBO: CbboMsg,
+    Schema.BBO_1S: MBP1Msg,
+    Schema.BBO_1M: MBP1Msg,
 }
 
 SCHEMA_STRUCT_MAP_V1: Final[dict[Schema, type[DBNRecord]]] = {

databento/common/dbnstore.py

@@ -330,7 +330,7 @@ class DBNStore:
 
     See Also
     --------
-    https://docs.databento.com/knowledge-base/new-users/dbn-encoding
+    https://databento.com/docs/knowledge-base/new-users/dbn-encoding
 
     """
 

databento/common/publishers.py

@@ -458,9 +458,9 @@ class Dataset(StringyMixin, str, Enum):
     EPRL_DOM
         MIAX Pearl Depth.
     FINN_NLS
-        FINRA/Nasdaq TRF.
+        FINRA/Nasdaq TRF (DEPRECATED).
     FINY_TRADES
-        FINRA/NYSE TRF.
+        FINRA/NYSE TRF (DEPRECATED).
     OPRA_PILLAR
         OPRA Binary.
     DBEQ_BASIC
@@ -690,9 +690,9 @@ def description(self) -> str:
         if self == Dataset.EPRL_DOM:
             return "MIAX Pearl Depth"
         if self == Dataset.FINN_NLS:
-            return "FINRA/Nasdaq TRF"
+            return "FINRA/Nasdaq TRF (DEPRECATED)"
         if self == Dataset.FINY_TRADES:
-            return "FINRA/NYSE TRF"
+            return "FINRA/NYSE TRF (DEPRECATED)"
         if self == Dataset.OPRA_PILLAR:
             return "OPRA Binary"
         if self == Dataset.DBEQ_BASIC:
@@ -758,11 +758,11 @@ class Publisher(StringyMixin, str, Enum):
         MEMX Memoir Depth.
     EPRL_DOM_EPRL
         MIAX Pearl Depth.
-    FINN_NLS_FINN
+    XNAS_NLS_FINN
         FINRA/Nasdaq TRF Carteret.
-    FINN_NLS_FINC
+    XNAS_NLS_FINC
         FINRA/Nasdaq TRF Chicago.
-    FINY_TRADES_FINY
+    XNYS_TRADES_FINY
         FINRA/NYSE TRF.
     OPRA_PILLAR_AMXO
         OPRA - NYSE American.
@@ -905,9 +905,9 @@ class Publisher(StringyMixin, str, Enum):
     XCIS_TRADES_XCIS = "XCIS.TRADES.XCIS"
     MEMX_MEMOIR_MEMX = "MEMX.MEMOIR.MEMX"
     EPRL_DOM_EPRL = "EPRL.DOM.EPRL"
-    FINN_NLS_FINN = "FINN.NLS.FINN"
-    FINN_NLS_FINC = "FINN.NLS.FINC"
-    FINY_TRADES_FINY = "FINY.TRADES.FINY"
+    XNAS_NLS_FINN = "XNAS.NLS.FINN"
+    XNAS_NLS_FINC = "XNAS.NLS.FINC"
+    XNYS_TRADES_FINY = "XNYS.TRADES.FINY"
     OPRA_PILLAR_AMXO = "OPRA.PILLAR.AMXO"
     OPRA_PILLAR_XBOX = "OPRA.PILLAR.XBOX"
     OPRA_PILLAR_XCBO = "OPRA.PILLAR.XCBO"
@@ -1008,11 +1008,11 @@ def from_int(cls, value: int) -> Publisher:
         if value == 16:
             return Publisher.EPRL_DOM_EPRL
         if value == 17:
-            return Publisher.FINN_NLS_FINN
+            return Publisher.XNAS_NLS_FINN
         if value == 18:
-            return Publisher.FINN_NLS_FINC
+            return Publisher.XNAS_NLS_FINC
         if value == 19:
-            return Publisher.FINY_TRADES_FINY
+            return Publisher.XNYS_TRADES_FINY
         if value == 20:
             return Publisher.OPRA_PILLAR_AMXO
         if value == 21:
@@ -1173,11 +1173,11 @@ def to_int(self) -> int:
             return 15
         if self == Publisher.EPRL_DOM_EPRL:
             return 16
-        if self == Publisher.FINN_NLS_FINN:
+        if self == Publisher.XNAS_NLS_FINN:
             return 17
-        if self == Publisher.FINN_NLS_FINC:
+        if self == Publisher.XNAS_NLS_FINC:
             return 18
-        if self == Publisher.FINY_TRADES_FINY:
+        if self == Publisher.XNYS_TRADES_FINY:
             return 19
         if self == Publisher.OPRA_PILLAR_AMXO:
             return 20
@@ -1340,11 +1340,11 @@ def venue(self) -> Venue:
             return Venue.MEMX
         if self == Publisher.EPRL_DOM_EPRL:
             return Venue.EPRL
-        if self == Publisher.FINN_NLS_FINN:
+        if self == Publisher.XNAS_NLS_FINN:
             return Venue.FINN
-        if self == Publisher.FINN_NLS_FINC:
+        if self == Publisher.XNAS_NLS_FINC:
             return Venue.FINC
-        if self == Publisher.FINY_TRADES_FINY:
+        if self == Publisher.XNYS_TRADES_FINY:
             return Venue.FINY
         if self == Publisher.OPRA_PILLAR_AMXO:
             return Venue.AMXO
@@ -1507,12 +1507,12 @@ def dataset(self) -> Dataset:
             return Dataset.MEMX_MEMOIR
         if self == Publisher.EPRL_DOM_EPRL:
             return Dataset.EPRL_DOM
-        if self == Publisher.FINN_NLS_FINN:
-            return Dataset.FINN_NLS
-        if self == Publisher.FINN_NLS_FINC:
-            return Dataset.FINN_NLS
-        if self == Publisher.FINY_TRADES_FINY:
-            return Dataset.FINY_TRADES
+        if self == Publisher.XNAS_NLS_FINN:
+            return Dataset.XNAS_NLS
+        if self == Publisher.XNAS_NLS_FINC:
+            return Dataset.XNAS_NLS
+        if self == Publisher.XNYS_TRADES_FINY:
+            return Dataset.XNYS_TRADES
         if self == Publisher.OPRA_PILLAR_AMXO:
             return Dataset.OPRA_PILLAR
         if self == Publisher.OPRA_PILLAR_XBOX:
@@ -1674,11 +1674,11 @@ def description(self) -> str:
             return "MEMX Memoir Depth"
         if self == Publisher.EPRL_DOM_EPRL:
             return "MIAX Pearl Depth"
-        if self == Publisher.FINN_NLS_FINN:
+        if self == Publisher.XNAS_NLS_FINN:
             return "FINRA/Nasdaq TRF Carteret"
-        if self == Publisher.FINN_NLS_FINC:
+        if self == Publisher.XNAS_NLS_FINC:
             return "FINRA/Nasdaq TRF Chicago"
-        if self == Publisher.FINY_TRADES_FINY:
+        if self == Publisher.XNYS_TRADES_FINY:
             return "FINRA/NYSE TRF"
         if self == Publisher.OPRA_PILLAR_AMXO:
             return "OPRA - NYSE American"

databento/common/types.py

@@ -9,6 +9,7 @@
 
 
 DBNRecord = Union[
+    databento_dbn.CbboMsg,
     databento_dbn.MBOMsg,
     databento_dbn.MBP1Msg,
     databento_dbn.MBP10Msg,
@@ -18,6 +19,7 @@
     databento_dbn.InstrumentDefMsg,
     databento_dbn.InstrumentDefMsgV1,
     databento_dbn.StatMsg,
+    databento_dbn.StatusMsg,
     databento_dbn.SymbolMappingMsg,
     databento_dbn.SymbolMappingMsgV1,
     databento_dbn.SystemMsg,

databento/live/protocol.py

@@ -80,23 +80,24 @@ def __init__(
         )
         self._gateway_decoder = GatewayDecoder()
 
-        self._authenticated: asyncio.Future[int] = asyncio.Future()
+        self._authenticated: asyncio.Future[str | None] = asyncio.Future()
         self._disconnected: asyncio.Future[None] = asyncio.Future()
+        self._error_msgs: list[str] = []
         self._started: bool = False
 
     @property
-    def authenticated(self) -> asyncio.Future[int]:
+    def authenticated(self) -> asyncio.Future[str | None]:
         """
         Future that completes when authentication with the gateway is
         completed.
 
-        The result will contain the session id if successful.
+        The result will contain the session ID if successful.
         The exception will contain a BentoError if authentication
         fails for any reason.
 
         Returns
         -------
-        asyncio.Future[int]
+        asyncio.Future[str | None]
 
         """
         return self._authenticated
@@ -323,15 +324,22 @@ def _process_dbn(self, data: bytes) -> None:
                 if isinstance(record, databento_dbn.Metadata):
                     self.received_metadata(record)
                     continue
-
                 if isinstance(record, databento_dbn.ErrorMsg):
                     logger.error(
                         "gateway error: %s",
                         record.err,
                     )
-                    self.disconnected.set_exception(
-                        BentoError(record.err),
-                    )
+                    self._error_msgs.append(record.err)
+                    if record.is_last:
+                        if len(self._error_msgs) > 1:
+                            errors = ", ".join(self._error_msgs)
+                            error_msg = f"The following errors occurred: {errors}"
+                        else:
+                            error_msg = self._error_msgs[-1]
+                        self._error_msgs.clear()
+                        self.disconnected.set_exception(
+                            BentoError(error_msg),
+                        )
                 if isinstance(record, databento_dbn.SystemMsg):
                     if record.is_heartbeat:
                         logger.debug("gateway heartbeat")
@@ -391,10 +399,7 @@ def _(self, message: AuthenticationResponse) -> None:
             )
             self.transport.close()
         else:
-            if message.session_id is None:
-                session_id = 0
-            else:
-                session_id = int(message.session_id)
+            session_id = message.session_id
 
             logger.debug(
                 "CRAM authenticated session id assigned `%s`",

databento/live/session.py

@@ -281,17 +281,17 @@ def __init__(
 
         self._user_gateway: str | None = user_gateway
         self._port = port
-        self._session_id: int = 0
+        self._session_id: str | None = None
 
     @property
-    def session_id(self) -> int:
+    def session_id(self) -> str | None:
         """
-        Return the authenticated session ID. A zero value indicates no session
+        Return the authenticated session ID. A None value indicates no session
         has started.
 
         Returns
         -------
-        int
+        str | None
 
         """
         return self._session_id

databento/version.py

@@ -1 +1 @@
-__version__ = "0.31.1"
+__version__ = "0.32.0"

notebooks/quickstart.ipynb

@@ -27,9 +27,9 @@
     "\n",
     "**Note:**\n",
     "\n",
-    "For information on our symbology, refer to https://docs.databento.com/api-reference-historical/basics/symbology. \n",
+    "For information on our symbology, refer to https://databento.com/docs/api-reference-historical/basics/symbology. \n",
     "\n",
-    "For a more detailed API reference, refer to https://docs.databento.com/api-reference-historical.\n",
+    "For a more detailed API reference, refer to https://databento.com/docs/api-reference-historical.\n",
     "\n",
     "This tutorial covers the following:\n",
     "- Using the historical client to request metadata\n",