Merge branch 'master' into feat/kick-off-code-samples

Author: m-murasovs

apify-api/openapi/components/schemas/datasets/DatasetSchemaValidationError.yaml

@@ -0,0 +1,53 @@
+type: object
+properties:
+  error:
+    type: object
+    properties:
+      type:
+        type: string
+        description: The type of the error.
+        example: "schema-validation-error"
+      message:
+        type: string
+        description: A human-readable message describing the error.
+        example: "Schema validation failed"
+      data:
+        type: object
+        properties:
+          invalidItems:
+            type: array
+            description: A list of invalid items in the received array of items.
+            items:
+              type: object
+              properties:
+                itemPosition:
+                  type: number
+                  description: The position of the invalid item in the array.
+                  example: 2
+                validationErrors:
+                  type: array
+                  description: A complete list of AJV validation error objects for the invalid item.
+                  items:
+                    type: object
+                    properties:
+                      instancePath:
+                        type: string
+                        description: The path to the instance being validated.
+                      schemaPath:
+                        type: string
+                        description: The path to the schema that failed the validation.
+                      keyword:
+                        type: string
+                        description: The validation keyword that caused the error.
+                      message:
+                        type: string
+                        description: A message describing the validation error.
+                      params:
+                        type: object
+                        description: Additional parameters specific to the validation error.
+        required:
+          - invalidItems
+    required:
+      - type
+      - message
+      - data

apify-api/openapi/components/schemas/datasets/PutItemResponseError.yaml

@@ -0,0 +1,9 @@
+title: PutItemResponseError
+required:
+  - error
+type: object
+properties:
+  error:
+    allOf:
+      - $ref: ./DatasetSchemaValidationError.yaml
+      - {}

apify-api/openapi/paths/datasets/datasets.yaml

@@ -106,6 +106,7 @@ post:
     Keep in mind that data stored under unnamed dataset follows [data retention period](https://docs.apify.com/platform/storage#data-retention).
     It creates a dataset with the given name if the parameter name is used.
     If a dataset with the given name already exists then returns its object.
+
   operationId: datasets_post
   parameters:
     - name: name

apify-api/openapi/paths/datasets/datasets@{datasetId}@items.yaml

@@ -478,6 +478,10 @@ post:
     The POST payload is a JSON object or a JSON array of objects to save into the dataset.
 
 
+    If the data you attempt to store in the dataset is invalid (meaning any of the items received by the API fails the validation), the whole request is discarded and the API will return a response with status code 400.
+    For more information about dataset schema validation, see [Dataset schema](https://docs.apify.com/platform/actors/development/actor-definition/dataset-schema/validation).
+
+
     **IMPORTANT:** The limit of request payload size for the dataset is 5 MB. If the array exceeds the size, you'll need to split it into a number of smaller arrays.
   operationId: dataset_items_post
   parameters:
@@ -523,6 +527,33 @@ post:
             type: object
             example: {}
           example: {}
+    '400':
+      description: ''
+      headers: {}
+      content:
+        application/json:
+            schema:
+                allOf:
+                - $ref: >-
+                    ../../components/schemas/datasets/PutItemResponseError.yaml
+                - example:
+                    error:
+                        type: schema-validation-error
+                        message: Schema validation failed
+            example:
+                error:
+                    type: schema-validation-error
+                    message: Schema validation failed
+                    data:
+                        invalidItems:
+                            - itemPosition: 2
+                              validationErrors:
+                                - instancePath: /1/stringField
+                                  schemaPath: /items/properties/stringField/type
+                                  keyword: type
+                                  params:
+                                    type: string
+                                  message: 'must be string'
   deprecated: false
   x-legacy-doc-urls:
     - https://docs.apify.com/api/v2#/reference/datasets/item-collection/put-items

package-lock.json

@@ -111,5 +111,5 @@
     "engines": {
         "node": ">=18.0.0"
     },
-    "packageManager": "[email protected].1"
+    "packageManager": "[email protected].2"
 }

package.json

@@ -118,7 +118,7 @@ The template above defines the configuration for the default dataset output view
 
 The default behavior of the Output tab UI table is to display all fields from `transformation.fields` in the specified order. You can customize the display properties for specific formats or column labels if needed.
 
-![Output tab UI](./images/output-schema-example.png)
+![Output tab UI](../images/output-schema-example.png)
 
 ## Structure
 

sources/platform/actors/development/actor_definition/output_schema.md renamed to sources/platform/actors/development/actor_definition/dataset_schema/index.md

@@ -0,0 +1,222 @@
+---
+title: Dataset validation
+description:  Specify the dataset schema within the Actors so you can add monitoring and validation at the field level.
+slug: /actors/development/actor-definition/dataset-schema/validation
+---
+
+**Specify the dataset schema within the Actors so you can add monitoring and validation at the field level.**
+
+---
+
+To define a schema for a default dataset of an Actor run, you need to set `fields` property in the dataset schema.
+
+:::info
+
+The schema defines a single item in the dataset. Be careful not to define the schema as an array, it always needs to be a schema of an object.
+
+Schema configuration is not available for named datasets or dataset views.
+
+:::
+
+You can either do that directly through `actor.json`:
+
+```json title=".actor.json"
+{
+    "actorSpecification": 1,
+    "storages": {
+        "dataset": {
+            "actorSpecification": 1,
+            "fields": {
+                "$schema": "http://json-schema.org/draft-07/schema#",
+                "type": "object",
+                "properties": {
+                    "name": {
+                        "type": "string"
+                    }
+                },
+                "required": ["name"]
+            },
+            "views": {}
+        }
+    }
+}
+```
+
+Or in a separate file linked from the `.actor.json`:
+
+```json title=".actor.json"
+{
+    "actorSpecification": 1,
+    "storages": {
+        "dataset": "./dataset_schema.json"
+    }
+}
+```
+
+```json title="dataset_schema.json"
+{
+    "actorSpecification": 1,
+    "fields": {
+        "$schema": "http://json-schema.org/draft-07/schema#",
+        "type": "object",
+        "properties": {
+            "name": {
+                "type": "string"
+            }
+        },
+        "required": ["name"]
+    },
+    "views": {}
+}
+```
+
+:::important
+
+Dataset schema needs to be a valid JSON schema draft-07, so the `$schema` line is important and must be exactly this value or it must be omitted:
+
+`"$schema": "http://json-schema.org/draft-07/schema#"`
+
+:::
+
+## Dataset validation
+
+When you define a schema of your default dataset, the schema is then always used when you insert data into the dataset to perform validation (we use [AJV](https://ajv.js.org/)).
+
+If the validation succeeds, nothing changes from the current behavior, data is stored and an empty response with status code `201` is returned.
+
+If the data you attempt to store in the dataset is _invalid_ (meaning any of the items received by the API fails validation), _the entire request will be discarded_, The API will return a response with status code `400` and the following JSON response:
+
+```json
+{
+    "error": {
+        "type": "schema-validation-error",
+        "message": "Schema validation failed",
+        "data": {
+            "invalidItems": [{
+                "itemPosition": "<array index in the received array of items>",
+                "validationErrors": "<Complete list of AJV validation error objects>"
+            }]
+        }
+    }
+}
+```
+
+The type of the AJV validation error object is [here](https://github.com/ajv-validator/ajv/blob/master/lib/types/index.ts#L86).
+
+If you use the Apify JS client or Apify SDK and call `pushData` function you can access the validation errors in a `try catch` block like this:
+
+```javascript
+try {
+    const response = await Actor.pushData(items);
+} catch (error) {
+    if (!error.data?.invalidItems) throw error;
+    error.data.invalidItems.forEach((item) => {
+        const { itemPosition, validationErrors } = item;
+    });
+}
+```
+
+## Examples of common types of validation
+
+Optional field (price is optional in this case):
+
+```json
+{
+    "$schema": "http://json-schema.org/draft-07/schema#",
+    "type": "object",
+    "properties": {
+        "name": {
+            "type": "string"
+        },
+        "price": {
+            "type": "number"
+        }
+    },
+    "required": ["name"]
+}
+```
+
+Field with multiple types:
+
+```json
+{
+    "price": {
+        "type": ["string", "number"]
+    }
+}
+```
+
+Field with type `any`:
+
+```json
+{
+    "price": {
+        "type": ["string", "number", "object", "array", "boolean"]
+    }
+}
+```
+
+Enabling fields to be `null` :
+
+```json
+{
+    "name": {
+        "type": "string",
+        "nullable": true
+    }
+}
+```
+
+Define type of objects in array:
+
+```json
+{
+    "comments": {
+        "type": "array",
+        "items": {
+            "type": "object",
+            "properties": {
+                "author_name": {
+                    "type": "string"
+                }
+            }
+        }
+    }
+}
+```
+
+Define specific fields, but allow anything else to be added to the item:
+
+```json
+{
+    "$schema": "http://json-schema.org/draft-07/schema#",
+    "type": "object",
+    "properties": {
+        "name": {
+            "type": "string"
+        }
+    },
+    "additionalProperties": true
+}
+```
+
+See [json schema reference](https://json-schema.org/understanding-json-schema/reference) for additional options.
+
+You can also use [conversion tools](https://www.liquid-technologies.com/online-json-to-schema-converter) to convert an existing JSON document into it's JSON schema.
+
+## Dataset field statistics
+
+When you configure the dataset fields schema, we generate a field list and measure the following statistics:
+
+- **Null count:** how many items in the dataset have the field set to null
+- **Empty count:** how many items in the dataset are `undefined` , meaning that for example empty string is not considered empty
+- **Minimum and maximum**
+  - For numbers, this is calculated directly
+  - For strings, this field tracks string length
+  - For arrays, this field tracks the number of items in the array
+  - For objects, this tracks the number of keys
+  - For booleans, this tracks whether the boolean was set to true. Minimum is always 0, but maximum can be either 1 or 0 based on whether at least one item in the dataset has the boolean field set to true.
+
+
+You can use them in [monitoring](../../../../monitoring#alert-configuration).
+