ARROW-151 Update documentation for extension types (#154)

Author: blink1073

bindings/python/docs/source/supported_types.rst renamed to bindings/python/docs/source/data_types.rst

@@ -1,12 +1,11 @@
 .. _type support:
 
-Supported Types
-===============
+Data Types
+==========
 
-PyMongoArrow currently supports a small subset of all BSON types.
+PyMongoArrow supports a majority of the BSON types.
 Support for additional types will be added in subsequent releases.
 
-
 .. note:: For more information about BSON types, see the
    `BSON specification <http://bsonspec.org/spec.html>`_.
 
@@ -24,7 +23,7 @@ Support for additional types will be added in subsequent releases.
    * - Embedded document
      - :class:`py.dict`, and instance of :class:`pyarrow.struct`
    * - Embedded array
-     - :class:`py.list`, an instance of :class:`pyarrow.list_`,
+     - An instance of :class:`pyarrow.list_`,
    * - ObjectId
      - :class:`py.bytes`, :class:`bson.ObjectId`, an instance of :class:`pymongoarrow.types.ObjectIdType`, an instance of :class:`pymongoarrow.pandas_types.PandasObjectId`
    * - Decimal128
@@ -39,6 +38,10 @@ Support for additional types will be added in subsequent releases.
      - :class:`~py.int`, :class:`bson.int64.Int64`, an instance of :meth:`pyarrow.int64`
    * - UTC datetime
      - an instance of :class:`~pyarrow.timestamp` with ``ms`` resolution, :class:`py.datetime.datetime`
+   * - Binary data
+     - :class:`bson.Binary`, an instance of :class:`pymongoarrow.types.BinaryType`, an instance of :class:`pymongoarrow.pandas_types.PandasBinary`.
+   * - JavaScript code
+     - :class:`bson.Code`, an instance of :class:`pymongoarrow.types.CodeType`, an instance of :class:`pymongoarrow.pandas_types.PandasCode`.
 
 Type identifiers can be used to specify that a field is of a certain type
 during :class:`pymongoarrow.api.Schema` declaration. For example, if your data
@@ -54,6 +57,81 @@ respectively, and '_id' that is an `ObjectId`, your schema can be defined as::
 Unsupported data types in a schema cause a ``ValueError`` identifying the
 field and its data type.
 
+
+Embedded Array Considerations
+-----------------------------
+
+The schema used for an Embedded Array must use the `pyarrow.list_()` type,
+so that the type of the array elements can be specified.  For example,
+
+.. code-block: python
+
+  from pyarrow import list_, float64
+  schema = Schema({'_id': ObjectId,
+    'location': {'coordinates': list_(float64())}
+  })
+
+
+Extension Types
+---------------
+
+The ``ObjectId``, ``Decimal128``, ``Binary data`` and ``JavaScript code``
+are implemented as extension types for PyArrow and Pandas.
+For arrow tables, values of these types will have the appropriate
+``pymongoarrow`` extension type (e.g. :class:`pymongoarrow.types.ObjectIdType`).  The appropriate ``bson`` Python object can be obtained using the ``.as_py()`` method,
+or by calling ``.to_pylist()`` on the table.
+
+.. code-block:: pycon
+
+  >>> from pymongo import MongoClient
+  >>> from bson import ObjectId
+  >>> from pymongoarrow.api import find_arrow_all
+  >>> client = MongoClient()
+  >>> coll = client.test.test
+  >>> coll.insert_many([{"_id": ObjectId(), "foo": 100}, {"_id": ObjectId(), "foo": 200}])
+  <pymongo.results.InsertManyResult at 0x1080a72b0>
+  >>> table = find_arrow_all(coll, {})
+  >>> table
+  pyarrow.Table
+  _id: extension<arrow.py_extension_type<ObjectIdType>>
+  foo: int32
+  ----
+  _id: [[64408B0D5AC9E208AF220142,64408B0D5AC9E208AF220143]]
+  foo: [[100,200]]
+  >>> table["_id"][0]
+  <pyarrow.ObjectIdScalar: ObjectId('64408b0d5ac9e208af220142')>
+  >>> table["_id"][0].as_py()
+  ObjectId('64408b0d5ac9e208af220142')
+  >>> table.to_pylist()
+  [{'_id': ObjectId('64408b0d5ac9e208af220142'), 'foo': 100},
+   {'_id': ObjectId('64408b0d5ac9e208af220143'), 'foo': 200}]
+
+When converting to pandas, the extension type columns will have an appropriate
+``pymongoarrow`` extension type (e.g. :class:`pymongoarrow.pandas_types.PandasDecimal128`).  The value of the element in the
+dataframe will be the appropriate ``bson`` type.
+
+.. code-block:: pycon
+
+  >>> from pymongo import MongoClient
+  >>> from bson import Decimal128
+  >>> from pymongoarrow.api import find_pandas_all
+  >>> client = MongoClient()
+  >>> coll = client.test.test
+  >>> coll.insert_many([{"foo": Decimal128("0.1")}, {"foo": Decimal128("0.1")}])
+  <pymongo.results.InsertManyResult at 0x1080a72b0>
+  >>> df = find_pandas_all(coll, {})
+  >>> df
+                          _id  foo
+  0  64408bf65ac9e208af220144  0.1
+  1  64408bf65ac9e208af220145  0.1
+  >>> df["foo"].dtype
+  <pymongoarrow.pandas_types.PandasDecimal128 at 0x11fe0ae90>
+  >>> df["foo"][0]
+  Decimal128('0.1')
+  >>> df["_id"][0]
+  ObjectId('64408bf65ac9e208af220144')
+
+
 Null Values and Conversion to Pandas DataFrames
 -----------------------------------------------
 

bindings/python/docs/source/extension_types.rst

@@ -18,11 +18,8 @@ know to use **PyMongoArrow**.
 :doc:`quickstart`
   Start here for a quick overview.
 
-:doc:`supported_types`
-  A list of BSON types that are supported by PyMongoArrow.
-
-:doc:`extension_types`
-  A more in-depth explanation of the support for extension types such as ObjectId.
+:doc:`data_types`
+  Data type support with PyMongoArrow.
 
 :doc:`faq`
   Frequently asked questions.
@@ -88,8 +85,7 @@ Indices and tables
 
    installation
    quickstart
-   supported_types
-   extension_types
+   data_types
    faq
    api/index
    changelog

bindings/python/docs/source/index.rst

@@ -68,8 +68,8 @@ to type-specifiers, e.g.::
   schema = Schema({'_id': int, 'amount': float, 'last_updated': datetime})
 
 There are multiple permissible type-identifiers for each supported BSON type.
-For a full-list of supported types and associated type-identifiers see
-:doc:`supported_types`.
+For a full-list of data types and associated type-identifiers see
+:doc:`data_types`.
 
 Nested data (embedded documents) are also supported::