fix(data): update /api/v2/data/ limits DEV-1278

jsapp/js/api/react-query/survey-data.ts

@@ -351,7 +351,7 @@ curl -X GET https://kf.kobotoolbox.org/api/v2/assets/{uid}/data/
 Two parameters can be used to control pagination.
 
 * `start`: Index (zero-based) from which the results start
-* `limit`: Number of results per page <span class='label label-warning'>Maximum results per page is **30000**</span>
+* `limit`: Number of results per page <span class='label label-warning'>Maximum results per page is **1000**</span>
 
 ```shell
 curl -X GET https://kf.kobotoolbox.org/api/v2/assets/{uid}/data/?start=0&limit=10

kobo/settings/base.py

@@ -916,7 +916,7 @@ def __init__(self, *args, **kwargs):
 
 # Impose a limit on the number of records returned by the submission list
 # endpoint. This overrides any `?limit=` query parameter sent by a client
-SUBMISSION_LIST_LIMIT = 30000
+SUBMISSION_LIST_LIMIT = 1000
 
 # uWSGI, NGINX, etc. allow only a limited amount of time to process a request.
 # Set this value to match their limits

kpi/docs/api/v2/data/list.md

@@ -10,7 +10,7 @@ curl -X GET https://kf.kobotoolbox.org/api/v2/assets/{uid}/data/
 Two parameters can be used to control pagination.
 
 * `start`: Index (zero-based) from which the results start
-* `limit`: Number of results per page <span class='label label-warning'>Maximum results per page is **30000**</span>
+* `limit`: Number of results per page <span class='label label-warning'>Maximum results per page is **1000**</span>
 
 ```shell
 curl -X GET https://kf.kobotoolbox.org/api/v2/assets/{uid}/data/?start=0&limit=10

kpi/paginators.py

@@ -123,7 +123,7 @@ class DataPagination(LimitOffsetPagination):
     """
     Pagination class for submissions.
     """
-    default_limit = settings.SUBMISSION_LIST_LIMIT
+    default_limit = 100
     offset_query_param = 'start'
     max_limit = settings.SUBMISSION_LIST_LIMIT
 

kpi/views/v2/data.py

@@ -667,17 +667,22 @@ def _filter_mongo_query(self, request):
 
         # Remove `format` from filters. No need to use it
         filters.pop('format', None)
-        # Do not allow requests to retrieve more than `SUBMISSION_LIST_LIMIT`
-        # submissions at one time
-        limit = filters.get('limit', settings.SUBMISSION_LIST_LIMIT)
-        try:
-            filters['limit'] = positive_int(
-                limit, strict=True, cutoff=settings.SUBMISSION_LIST_LIMIT
-            )
-        except ValueError:
-            raise serializers.ValidationError(
-                {'limit': t('A positive integer is required')}
-            )
+        # Do not allow requests to retrieve more than `max_limit`
+        # submissions at one time only if a limit is explicitly defined.
+        if 'limit' in filters:
+            try:
+                filters['limit'] = positive_int(
+                    filters['limit'],
+                    strict=True,
+                    cutoff=self.pagination_class.max_limit,
+                )
+            except ValueError:
+                raise serializers.ValidationError(
+                    {'limit': t('A positive integer is required')}
+                )
+        else:
+            # If no limit is specified, use the default limit (100)
+            filters['limit'] = self.pagination_class.default_limit
 
         return filters
 

static/openapi/schema_v2.json

@@ -1883,7 +1883,7 @@
         "/api/v2/assets/{uid_asset}/data/": {
             "get": {
                 "operationId": "api_v2_assets_data_list",
-                "description": "## List of submissions for a specific asset\n\nBy default, JSON format is used, but XML and GeoJSON are also available:\n\n```shell\ncurl -X GET https://kf.kobotoolbox.org/api/v2/assets/{uid}/data/\n```\n\n### Pagination\nTwo parameters can be used to control pagination.\n\n* `start`: Index (zero-based) from which the results start\n* `limit`: Number of results per page <span class='label label-warning'>Maximum results per page is **30000**</span>\n\n```shell\ncurl -X GET https://kf.kobotoolbox.org/api/v2/assets/{uid}/data/?start=0&limit=10\n```\n\n### Query submitted data\nProvides a list of submitted data for a specific form. Use `query`\nparameter to apply form data specific, see\n<a href=\"http://docs.mongodb.org/manual/reference/operator/query/\">http://docs.mongodb.org/manual/reference/operator/query/</a>.\n\nFor more details see\n<a href=\"https://github.com/SEL-Columbia/formhub/wiki/Formhub-Access-Points-(API)#api-parameters\">API Parameters</a>.\n\n```shell\ncurl -X GET https://kf.kobotoolbox.org/api/v2/assets/{uid}/data/?query={\"__version__\": \"vWvkKzNE8xCtfApJvabfjG\"}\ncurl https://kf.kobotoolbox.org/api/v2/assets/{uid}/data/?query={\"_submission_time\": {\"$gt\": \"2019-09-01T01:02:03\"}}\n```\n\n### About the GeoJSON format\nRequesting the `geojson` format returns a `FeatureCollection` where each\nsubmission is a `Feature`. If your form has multiple geographic questions,\nuse the `geo_question_name` query parameter to determine which question's\nresponses populate the `geometry` for each `Feature`; otherwise, the first\ngeographic question is used.  All question/response pairs are included in\nthe `properties` of each `Feature`, but _repeating groups are omitted_.\n\nQuestion types are mapped to GeoJSON geometry types as follows:\n\n* `geopoint` to `Point`;\n* `geotrace` to `LineString`;\n* `geoshape` to `Polygon`.\n\n\n\n### ⚠️ Note: DRF-Spectacular Limitation\n\nDue to limitations in **DRF-Spectacular**, the `ACCEPT` headers do not sync properly with the request. As a result, all responses will default to `application/json`, regardless of the specified format.\n\nThis means that while alternative formats (like XML) are technically supported and will work via command-line tools (e.g., `curl`), **they will not work** when trying out the endpoint directly from the documentation page.\n\nWe’ve still included the header to show supported formats, but keep in mind:\n**Only `application/json` will be used in the docs UI.**\n\n",
+                "description": "## List of submissions for a specific asset\n\nBy default, JSON format is used, but XML and GeoJSON are also available:\n\n```shell\ncurl -X GET https://kf.kobotoolbox.org/api/v2/assets/{uid}/data/\n```\n\n### Pagination\nTwo parameters can be used to control pagination.\n\n* `start`: Index (zero-based) from which the results start\n* `limit`: Number of results per page <span class='label label-warning'>Maximum results per page is **1000**</span>\n\n```shell\ncurl -X GET https://kf.kobotoolbox.org/api/v2/assets/{uid}/data/?start=0&limit=10\n```\n\n### Query submitted data\nProvides a list of submitted data for a specific form. Use `query`\nparameter to apply form data specific, see\n<a href=\"http://docs.mongodb.org/manual/reference/operator/query/\">http://docs.mongodb.org/manual/reference/operator/query/</a>.\n\nFor more details see\n<a href=\"https://github.com/SEL-Columbia/formhub/wiki/Formhub-Access-Points-(API)#api-parameters\">API Parameters</a>.\n\n```shell\ncurl -X GET https://kf.kobotoolbox.org/api/v2/assets/{uid}/data/?query={\"__version__\": \"vWvkKzNE8xCtfApJvabfjG\"}\ncurl https://kf.kobotoolbox.org/api/v2/assets/{uid}/data/?query={\"_submission_time\": {\"$gt\": \"2019-09-01T01:02:03\"}}\n```\n\n### About the GeoJSON format\nRequesting the `geojson` format returns a `FeatureCollection` where each\nsubmission is a `Feature`. If your form has multiple geographic questions,\nuse the `geo_question_name` query parameter to determine which question's\nresponses populate the `geometry` for each `Feature`; otherwise, the first\ngeographic question is used.  All question/response pairs are included in\nthe `properties` of each `Feature`, but _repeating groups are omitted_.\n\nQuestion types are mapped to GeoJSON geometry types as follows:\n\n* `geopoint` to `Point`;\n* `geotrace` to `LineString`;\n* `geoshape` to `Polygon`.\n\n\n\n### ⚠️ Note: DRF-Spectacular Limitation\n\nDue to limitations in **DRF-Spectacular**, the `ACCEPT` headers do not sync properly with the request. As a result, all responses will default to `application/json`, regardless of the specified format.\n\nThis means that while alternative formats (like XML) are technically supported and will work via command-line tools (e.g., `curl`), **they will not work** when trying out the endpoint directly from the documentation page.\n\nWe’ve still included the header to show supported formats, but keep in mind:\n**Only `application/json` will be used in the docs UI.**\n\n",
                 "parameters": [
                     {
                         "in": "query",

static/openapi/schema_v2.yaml

@@ -1341,7 +1341,7 @@ paths:
         Two parameters can be used to control pagination.
 
         * `start`: Index (zero-based) from which the results start
-        * `limit`: Number of results per page <span class='label label-warning'>Maximum results per page is **30000**</span>
+        * `limit`: Number of results per page <span class='label label-warning'>Maximum results per page is **1000**</span>
 
         ```shell
         curl -X GET https://kf.kobotoolbox.org/api/v2/assets/{uid}/data/?start=0&limit=10