diff --git a/source/connect/connection-targets.txt b/source/connect/connection-targets.txt index 9ee5b932..6bf70fab 100644 --- a/source/connect/connection-targets.txt +++ b/source/connect/connection-targets.txt @@ -156,6 +156,38 @@ Troubleshooting .. include:: /includes/troubleshooting/connection-targets.rst +Timeout When Accessing MongoDB from {+driver-short+} with Tunneling +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +If you try to connect to a MongoDB replica set over an SSH tunnel, you +receive the following error: + +.. code-block:: python + + File "/Library/Python/2.7/site-packages/pymongo/collection.py", line 1560, in count + return self._count(cmd, collation, session) + File "/Library/Python/2.7/site-packages/pymongo/collection.py", line 1504, in _count + with self._socket_for_reads() as (connection, slave_ok): + File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__ + return self.gen.next() + File "/Library/Python/2.7/site-packages/pymongo/mongo_client.py", line 982, in _socket_for_reads + server = topology.select_server(read_preference) + File "/Library/Python/2.7/site-packages/pymongo/topology.py", line 224, in select_server + address)) + File "/Library/Python/2.7/site-packages/pymongo/topology.py", line 183, in select_servers + selector, server_timeout, address) + File "/Library/Python/2.7/site-packages/pymongo/topology.py", line 199, in _select_servers_loop + self._error_message(selector)) + pymongo.errors.ServerSelectionTimeoutError: localhost:27017: timed out + +This occurs because {+driver-short+} discovers replica set members by using the response +from the ``isMaster`` command, which contains the addresses and ports of the other +replica set members. However, you can't access these addresses and ports through the SSH +tunnel. + +Instead, you can connect directly to a single MongoDB node by using the +``directConnection=True`` option with SSH tunneling. + API Documentation ----------------- diff --git a/source/connect/mongoclient.txt b/source/connect/mongoclient.txt index 683afc1d..82d4f671 100644 --- a/source/connect/mongoclient.txt +++ b/source/connect/mongoclient.txt @@ -199,6 +199,9 @@ child process. a high probability of deadlock among ``MongoClient`` instances in the child process. {+driver-short+} tries to issue a warning if this deadlock might occur. + For more information about deadlock in forked processes, see + :ref:`pymongo-fork-deadlock`. + Multiprocessing ~~~~~~~~~~~~~~~ @@ -245,6 +248,70 @@ create the ``MongoClient`` object, as shown in the following example: from typing import Any, Dict client: MongoClient[Dict[str, Any]] = MongoClient() +Troubleshooting +--------------- + +MongoClient Fails ConfigurationError +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +Providing invalid keyword argument names causes the driver to raise this error. + +Ensure that the keyword arguments that you specify exist and are +spelled correctly. + +.. _pymongo-fork-deadlock: + +Forking a Process Causes a Deadlock +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +A ``MongoClient`` instance spawns multiple threads to run background tasks, such as +monitoring connected servers. These threads share state that is protected by instances +of the ``threading.Lock`` class, which are themselves +`not fork-safe `__. +{+driver-short+} is subject to the same limitations as any other multithreaded +code that uses the ``threading.Lock`` class, or any mutexes. + +One of these limitations is that the locks become useless after calling the +``fork()`` method. When ``fork()`` executes, the driver copies all the parent process's locks to +the child process in the same state as they were in the parent. If they are +locked in the parent process, they are also locked in the child process. The child process +created by ``fork()`` has only one thread, so any locks created by +other threads in the parent process are never released in the child process. +The next time the child process attempts to acquire one of these locks, deadlock occurs. + +Starting in {+driver-short+} version 4.3, after you call the ``os.fork()`` method, the +driver uses the ``os.register_at_fork()`` method to reset its locks and other shared state +in the child process. Although this reduces the likelihood of a deadlock, +{+driver-short+} depends +on libraries that aren't fork-safe in multithreaded applications, including +`OpenSSL `__ and +`getaddrinfo(3). `__ +Therefore, a deadlock can still occur. + +The Linux manual page for `fork(2) `__ +also imposes the following restriction: + +.. blockquote:: + + After a ``fork()`` in a multithreaded program, the child can + safely call only async-signal-safe functions (see + `signal-safety(7) `__) + until such time as it calls + `execve(2) `__. + +Because {+driver-short+} relies on functions that are *not* +async-signal-safe, it can cause deadlocks or crashes when running in a child +process. + +.. tip:: + + For an example of a deadlock in a child process, see + `PYTHON-3406 `__ in Jira. + + For more information about the problems caused by Python locks in + multithreaded contexts with ``fork()``, see `Issue 6721 `__ + in the Python Issue Tracker. + API Documentation ----------------- diff --git a/source/connect/tls.txt b/source/connect/tls.txt index 083609a3..51cc6b9e 100644 --- a/source/connect/tls.txt +++ b/source/connect/tls.txt @@ -231,8 +231,10 @@ To disable only hostname verification, set the ``tlsAllowInvalidHostnames`` opti vulnerable to expired certificates and to foreign processes posing as valid client instances. -Troubleshoot TLS ----------------- +.. _pymongo-troubleshoot-tls: + +Troubleshooting +--------------- .. include:: /includes/troubleshooting/tls.rst diff --git a/source/data-formats/dates-and-times.txt b/source/data-formats/dates-and-times.txt index d81157d6..01d18700 100644 --- a/source/data-formats/dates-and-times.txt +++ b/source/data-formats/dates-and-times.txt @@ -440,6 +440,76 @@ at the beginning and end of the allowed range. datetime before the range: {"date": datetime.datetime(1, 1, 1, 0, 0)} datetime after the range: {"date": datetime.datetime(9999, 12, 31, 23, 59, 59, 999000)} +Troubleshooting +--------------- + +OverflowError When Decoding Dates Stored by Another Language's Driver +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +{+driver-short+} decodes BSON ``datetime`` values to instances of Python's +``datetime.datetime`` class. Instances of ``datetime.datetime`` are +limited to years between ``datetime.MINYEAR`` (1) and +``datetime.MAXYEAR`` (9999). Some MongoDB drivers +can store BSON datetimes with year values far outside those supported +by ``datetime.datetime``. + +There are a few ways to work around this issue. Starting with {+driver-short+} 4.3, +``bson.decode`` can decode BSON ``datetime`` values in one of four ways. You can specify +the conversion method by using ``datetime_conversion`` parameter of +``~bson.codec_options.CodecOptions``. + +The default conversion option is +``~bson.codec_options.DatetimeConversion.DATETIME``, which will +attempt to decode the value as a ``datetime.datetime``, allowing +``~builtin.OverflowError`` to occur for out-of-range dates. +``~bson.codec_options.DatetimeConversion.DATETIME_AUTO`` alters +this behavior to instead return ``~bson.datetime_ms.DatetimeMS`` when +representations are out-of-range, while returning ``~datetime.datetime`` +objects as before: + +.. io-code-block:: + + .. input:: + :language: python + + from datetime import datetime + from bson.datetime_ms import DatetimeMS + from bson.codec_options import DatetimeConversion + from pymongo import MongoClient + + client = MongoClient(datetime_conversion=DatetimeConversion.DATETIME_AUTO) + client.db.collection.insert_one({"x": datetime(1970, 1, 1)}) + + client.db.collection.insert_one({"x": DatetimeMS(2**62)}) + + for x in client.db.collection.find(): + print(x) + + .. output:: + + {'_id': ObjectId('...'), 'x': datetime.datetime(1970, 1, 1, 0, 0)} + {'_id': ObjectId('...'), 'x': DatetimeMS(4611686018427387904)} + +For other options, see the API documentation for the +`DatetimeConversion <{+api-root+}bson/codec_options.html#bson.codec_options.DatetimeConversion>`__ +class. + +Another option that does not involve setting ``datetime_conversion`` is to +filter out document values outside of the range supported by +``~datetime.datetime``: + +.. code-block:: python + + from datetime import datetime + coll = client.test.dates + cur = coll.find({'dt': {'$gte': datetime.min, '$lte': datetime.max}}) + +If you don't need the value of ``datetime``, you can filter out just that field: + +.. code-block:: python + + cur = coll.find({}, projection={'dt': False}) + API Documentation ----------------- diff --git a/source/read/specify-a-query.txt b/source/read/specify-a-query.txt index 3b0a1887..f61c69c4 100644 --- a/source/read/specify-a-query.txt +++ b/source/read/specify-a-query.txt @@ -238,6 +238,48 @@ all documents with a ``name`` field value that has at least two consecutive {'_id': 1, 'name': 'apples', 'qty': 5, 'rating': 3, 'color': 'red', 'type': ['fuji', 'honeycrisp']} {'_id': 4, 'name': 'pineapple', 'qty': 3, 'rating': 5, 'color': 'yellow'} +Troubleshooting +--------------- + +.. _web-application-querying-by-objectid: + +No Results When Querying for a Document by ObjectId in Web Applications +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +It's common in web applications to encode documents' ObjectIds in URLs, as shown +in the following code example: + +.. code-block:: python + + "/posts/50b3bda58a02fb9a84d8991e" + +Your web framework passes the ObjectId part of the URL to your request +handler as a string. You must convert the string to an ``ObjectId`` instance +before passing it to the ``find_one()`` method. + +The following code example shows how to perform this conversion in a +`Flask `__ application. The process is similar for other web +frameworks. + +.. code-block:: python + + from pymongo import MongoClient + from bson.objectid import ObjectId + + from flask import Flask, render_template + + client = MongoClient() + app = Flask(__name__) + + @app.route("/posts/<_id>") + def show_post(_id): + # NOTE!: converting _id from string to ObjectId before passing to find_one + post = client.db.posts.find_one({'_id': ObjectId(_id)}) + return render_template('post.html', post=post) + + if __name__ == "__main__": + app.run() + Additional Information ---------------------- diff --git a/source/troubleshooting.txt b/source/troubleshooting.txt deleted file mode 100644 index c20b02c6..00000000 --- a/source/troubleshooting.txt +++ /dev/null @@ -1,266 +0,0 @@ -.. _pymongo-troubleshooting: - -Troubleshooting -=============== - -.. contents:: On this page - :local: - :backlinks: none - :depth: 1 - :class: singlecol - -.. facet:: - :name: genre - :values: reference - -.. meta:: - :keywords: error, help - -On this page, you can find solutions to common issues encountered while using -{+driver-short+} with MongoDB. - -Connection ----------- - -.. include:: /includes/troubleshooting/connection-targets.rst - -Timeout When Accessing MongoDB from {+driver-short+} with Tunneling -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -If you try to connect to a MongoDB replica set over an SSH tunnel, you -receive the following error: - -.. code-block:: python - - File "/Library/Python/2.7/site-packages/pymongo/collection.py", line 1560, in count - return self._count(cmd, collation, session) - File "/Library/Python/2.7/site-packages/pymongo/collection.py", line 1504, in _count - with self._socket_for_reads() as (connection, slave_ok): - File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/contextlib.py", line 17, in __enter__ - return self.gen.next() - File "/Library/Python/2.7/site-packages/pymongo/mongo_client.py", line 982, in _socket_for_reads - server = topology.select_server(read_preference) - File "/Library/Python/2.7/site-packages/pymongo/topology.py", line 224, in select_server - address)) - File "/Library/Python/2.7/site-packages/pymongo/topology.py", line 183, in select_servers - selector, server_timeout, address) - File "/Library/Python/2.7/site-packages/pymongo/topology.py", line 199, in _select_servers_loop - self._error_message(selector)) - pymongo.errors.ServerSelectionTimeoutError: localhost:27017: timed out - -This occurs because {+driver-short+} discovers replica set members by using the response -from the ``isMaster`` command, which contains the addresses and ports of the other -replica set members. However, you can't access these addresses and ports through the SSH -tunnel. - -Instead, you can connect directly to a single MongoDB node by using the -``directConnection=True`` option with SSH tunneling. - -Read and Write Operations -------------------------- - -.. include:: /includes/troubleshooting/read-write-options.rst - -.. include:: /includes/troubleshooting/count.rst - -MongoClient Fails ConfigurationError -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -Providing invalid keyword argument names causes the driver to raise this error. - -Ensure that the keyword arguments you specify exist and are -spelled correctly. - -.. _web-application-querying-by-objectid: - -No Results When Querying for a Document by ObjectId in Web Applications -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -It's common in web applications to encode documents' ObjectIds in URLs, as shown -in the following code example: - -.. code-block:: python - - "/posts/50b3bda58a02fb9a84d8991e" - -Your web framework passes the ObjectId part of the URL to your request -handler as a string. You must convert the string to an ``ObjectId`` instance -before passing it to the ``find_one()`` method. - -The following code example shows how to perform this conversion in a -`Flask `__ application. The process is similar for other web -frameworks. - -.. code-block:: python - - from pymongo import MongoClient - from bson.objectid import ObjectId - - from flask import Flask, render_template - - client = MongoClient() - app = Flask(__name__) - - @app.route("/posts/<_id>") - def show_post(_id): - # NOTE!: converting _id from string to ObjectId before passing to find_one - post = client.db.posts.find_one({'_id': ObjectId(_id)}) - return render_template('post.html', post=post) - - if __name__ == "__main__": - app.run() - -Cursors -------- - -.. include:: /includes/troubleshooting/cursors.rst - -.. _pymongo-fork-safe-details: - -Projections ------------ - -.. include:: /includes/troubleshooting/projections.rst - -Indexes -------- - -.. include:: /includes/troubleshooting/unique-index.rst - -Data Formats ------------- - -.. include:: /includes/troubleshooting/uuid.rst - -OverflowError When Decoding Dates Stored by Another Language's Driver -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -{+driver-short+} decodes BSON ``datetime`` values to instances of Python's -``datetime.datetime`` class. Instances of ``datetime.datetime`` are -limited to years between ``datetime.MINYEAR`` (1) and -``datetime.MAXYEAR`` (9999). Some MongoDB drivers -can store BSON datetimes with year values far outside those supported -by ``datetime.datetime``. - -There are a few ways to work around this issue. Starting with {+driver-short+} 4.3, -``bson.decode`` can decode BSON ``datetime`` values in one of four ways. You can specify -the conversion method by using ``datetime_conversion`` parameter of -``~bson.codec_options.CodecOptions``. - -The default conversion option is -``~bson.codec_options.DatetimeConversion.DATETIME``, which will -attempt to decode the value as a ``datetime.datetime``, allowing -``~builtin.OverflowError`` to occur for out-of-range dates. -``~bson.codec_options.DatetimeConversion.DATETIME_AUTO`` alters -this behavior to instead return ``~bson.datetime_ms.DatetimeMS`` when -representations are out-of-range, while returning ``~datetime.datetime`` -objects as before: - -.. io-code-block:: - - .. input:: - :language: python - - from datetime import datetime - from bson.datetime_ms import DatetimeMS - from bson.codec_options import DatetimeConversion - from pymongo import MongoClient - - client = MongoClient(datetime_conversion=DatetimeConversion.DATETIME_AUTO) - client.db.collection.insert_one({"x": datetime(1970, 1, 1)}) - - client.db.collection.insert_one({"x": DatetimeMS(2**62)}) - - for x in client.db.collection.find(): - print(x) - - .. output:: - - {'_id': ObjectId('...'), 'x': datetime.datetime(1970, 1, 1, 0, 0)} - {'_id': ObjectId('...'), 'x': DatetimeMS(4611686018427387904)} - -For other options, see the API documentation for the -`DatetimeConversion <{+api-root+}bson/codec_options.html#bson.codec_options.DatetimeConversion>`__ -class. - -Another option that does not involve setting ``datetime_conversion`` is to -filter out document values outside of the range supported by -``~datetime.datetime``: - -.. code-block:: python - - from datetime import datetime - coll = client.test.dates - cur = coll.find({'dt': {'$gte': datetime.min, '$lte': datetime.max}}) - -If you don't need the value of ``datetime``, you can filter out just that field: - -.. code-block:: python - - cur = coll.find({}, projection={'dt': False}) - -.. _pymongo-troubleshoot-tls: - -TLS ---- - -.. include:: /includes/troubleshooting/tls.rst - -Client-Side Operation Timeouts ------------------------------- - -.. include:: /includes/troubleshooting/csot.rst - -Forking Processes ------------------ - -Forking a Process Causes a Deadlock -~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ - -A ``MongoClient`` instance spawns multiple threads to run background tasks, such as -monitoring connected servers. These threads share state that is protected by instances -of the ``threading.Lock`` class, which are themselves -`not fork-safe `__. -{+driver-short+} is subject to the same limitations as any other multithreaded -code that uses the ``threading.Lock`` class, or any mutexes. - -One of these limitations is that the locks become useless after calling the -``fork()`` method. When ``fork()`` executes, the driver copies all the parent process's locks to -the child process in the same state as they were in the parent. If they are -locked in the parent process, they are also locked in the child process. The child process -created by ``fork()`` has only one thread, so any locks created by -other threads in the parent process are never released in the child process. -The next time the child process attempts to acquire one of these locks, deadlock occurs. - -Starting in {+driver-short+} version 4.3, after you call the ``os.fork()`` method, the -driver uses the ``os.register_at_fork()`` method to reset its locks and other shared state -in the child process. Although this reduces the likelihood of a deadlock, -{+driver-short+} depends -on libraries that aren't fork-safe in multithreaded applications, including -`OpenSSL `__ and -`getaddrinfo(3). `__ -Therefore, a deadlock can still occur. - -The Linux manual page for `fork(2) `__ -also imposes the following restriction: - -.. blockquote:: - - After a ``fork()`` in a multithreaded program, the child can - safely call only async-signal-safe functions (see - `signal-safety(7) `__) - until such time as it calls - `execve(2) `__. - -Because {+driver-short+} relies on functions that are *not* -async-signal-safe, it can cause deadlocks or crashes when running in a child -process. - -.. tip:: - - For an example of a deadlock in a child process, see - `PYTHON-3406 `__ in Jira. - - For more information about the problems caused by Python locks in - multithreaded contexts with ``fork()``, see `Issue 6721 `__ - in the Python Issue Tracker. \ No newline at end of file