parallel -> concurrent, document on fail in concurrency

Author: CalebCourier

docs/concepts/parallelization.md renamed to docs/concepts/concurrency.md

@@ -1,11 +1,11 @@
-# Parallelization
+# Concurrency
 ## And the Orchestration of Guard Executions
 
 This document is a description of the current implementation of the Guardrails' validation loop.  It attempts to explain the current patterns used with some notes on why those patterns were accepted at the time of implementation and potential future optimizations.  It is _not_ meant to be prescriptive as there can, and will, be improvements made in future versions.
 
 In general you will find that our approach to performance is two fold:
 1. Complete computationally cheaper, static checks first and exit early to avoid spending time and resources on more expensive checks that are unlikely to pass when the former fail.
-2. Parallelize processing where possible.
+2. Run processes concurrently where possible.
 
 ## Background: The Validation Loop
 When a Guard is executed, that is called via `guard()`, `guard.parse()`, `guard.validate()`, etc., it goes through an internal process that has the following steps:
@@ -60,7 +60,7 @@ Besides handling asynchronous calls to the LLM, using an `AsyncGuard` also ensur
 * An asyncio event loop is available.
 * The asyncio event loop is not taken/already running.
 
-## Validation Orchestration and Parallelization
+## Validation Orchestration and Concurrency
 
 ### Structured Data Validation
 We perform validation with a "deep-first" approach.  This has no meaning for unstructured text output since there is only one value, but for structured output it means that the objects are validated from the inside out.
@@ -79,7 +79,7 @@ Take the below structure as an example:
 }
 ```
 
-As of versions v0.4.x and v0.5.x of Guardrails, the above object would validated as follows:
+As of versions v0.4.x and v0.5.x of Guardrails, the above object would be validated as follows:
 
 1. foo.baz
 2. foo.bez
@@ -88,11 +88,134 @@ As of versions v0.4.x and v0.5.x of Guardrails, the above object would validated
 5. bar.buz
 6. bar
 
-    > NOTE: The approach currently used, and outlined above, was predicated on the assumption that if child properties fail validation, it is unlikely that the parent property would pass.  With the current atomic state of validation, it can be argued that this assumption is false.  That is, the types of validations applied to parent properties typically take the form of checking the appropriate format of the container like a length check on a list. These types of checks are generally independent of any requirements the child properties have.  This opens up the possibility of running all six paths listed above in parallel at once instead of performing them in steps based on key path.
+    > NOTE: The approach currently used, and outlined above, was predicated on the assumption that if child properties fail validation, it is unlikely that the parent property would pass.  With the current atomic state of validation, it can be argued that this assumption is false.  That is, the types of validations applied to parent properties typically take the form of checking the appropriate format of the container like a length check on a list. These types of checks are generally independent of any requirements the child properties have.  This opens up the possibility of running all six paths listed above concurrently instead of performing them in steps based on key path.
 
 When synchronous validation occurs as defined in [Benefits of AsyncGuard](#benefits-of-async-guard), the validators for each property would be run in the order they are defined on the schema.  That also means that any on fail actions are applied in that same order.
 
-When asynchronous validation occurs, there are multiple levels of parallelization possible.  First, running validation on the child properties (e.g. `foo.baz` and `foo.bez`) will happen in parallel via the asyncio event loop.  Second, within the validation for each property, if the validators have `run_in_separate_process` set to `True`, they are run in parallel via multiprocessing.  This multiprocessing is capped to the process count specified by the `GUARDRAILS_PROCESS_COUNT` environment variable which defaults to 10.  Note that some environments, like AWS Lambda, may not support multiprocessing in which case you would need to set this environment variable to 1.
+When asynchronous validation occurs, there are multiple levels of concurrency possible.  First, running validation on the child properties (e.g. `foo.baz` and `foo.bez`) will happen concurrently via the asyncio event loop.  Second, the validators on any given property are also run concurrently via the event loop.  For validators that only define a synchronous `validate` method, calls to this method are run in the event loops default executore.  Note that some environments, like AWS Lambda, may not support multiprocessing in which case you would need to either set the executor to a thread processor instead or limit validation to running synchronously by setting `GUARDRAILS_PROCESS_COUNT=1` or `GUARDRAILS_RUN_SYNC=true`.
 
 ### Unstructured Data Validation
-When validating unstructured data, i.e. text, the LLM output is treated the same as if it were a property on an object.  This means that the validators applied to is have the ability to run in parallel utilizing multiprocessing when `run_in_separate_process` is set to `True` on the validators.
+When validating unstructured data, i.e. text, the LLM output is treated the same as if it were a property on an object.  This means that the validators applied to is have the ability to run concurrently utilizing the event loop.
+
+### Handling Failures During Async Concurrency
+The Guardrails validation loop is opinionated about how it handles failures when running validators concurrently so that it spends the least amount of time processing an output that would result in a failure.  It's behaviour comes down to when and what it returns based on the [corrective action](/how_to_guides/custom_validators#on-fail) specified on a validator.  Corrective actions are processed concurrently since they are specific to a given validator on a given property.  This means that interuptive corrective actions, namely `EXCEPTION`, will be the first corrective action enforced because the exception is raised as soon as the failure is evaluated.  The remaining actions are handled in the following order after all futures are collected from the validation of a specific property:
+1. `FILTER` and `REFRAIN`
+2. `REASK`
+3. `FIX`
+
+    \*_NOTE:_ `NOOP` Does not require any special handling because it does not alter the value.
+    
+    \*_NOTE:_ `FIX_REASK` Will fall into either the `REASK` or `FIX` bucket based on if the fixed value passes the second round of validation.
+
+This means that if any valdiator with `on_fail=OnFailAction.EXCEPTION` returns a `FailResult`, then Guardrails will raise a `ValidationError` interrupting the process.
+
+If any validator on a specific property which has `on_fail=OnFailAction.FILTER` or `on_fail=OnFailAction.REFRAIN` returns a `FailResult`, whichever of these is the first to finish will the returned early as the value for that property,
+
+If any validator on a specific property which has `on_fail=OnFailAction.REASK` returns a `FailResult`, all reasks for that property will be merged and a `FieldReAsk` will be returned early as the value for that property.
+
+If any validator on a specific property which has `on_fail=OnFailAction.FIX` returns a `FailResult`, all fix values for that property will be merged and the result of that merge will be returned as the value for that property.
+
+Custom on_fail handlers will fall into one of the above actions based on what it returns; i.e. if it returns an updated value it's considered a `FIX`, if it returns an instance of `Filter` then `FILTER`, etc..
+
+Let's look at an example.  We'll keep the validation logic simple and write out some assertions to demonstrate the evaluation order discussed above.
+
+```py
+import asyncio
+from random import randint
+from typing import Optional
+from guardrails import AsyncGuard, ValidationOutcome
+from guardrails.errors import ValidationError
+from guardrails.validators import (
+    Validator,
+    register_validator,
+    ValidationResult,
+    PassResult,
+    FailResult
+)
+
+@register_validator(name='custom/contains', data_type='string')
+class Contains(Validator):
+    def __init__(self, match_value: str, **kwargs):
+        super().__init__(
+            match_value=match_value,
+            **kwargs
+        )
+        self.match_value = match_value
+
+    def validate(self, value, metadata = {}) -> ValidationResult:
+        if self.match_value in value:
+            return PassResult()
+        
+        fix_value = None
+        if self.on_fail_descriptor == 'fix':
+            # Insert the match_value into the value at a random index
+            insertion = randint(0, len(value))
+            fix_value = f"{value[:insertion]}{self.match_value}{value[insertion:]}"
+        
+        return FailResult(
+            error_message=f'Value must contain {self.match_value}',
+            fix_value=fix_value
+        )
+    
+exception_validator = Contains("a", on_fail='exception')
+filter_validator = Contains("b", on_fail='filter')
+refrain_validator = Contains("c", on_fail='refrain')
+reask_validator_1 = Contains("d", on_fail='reask')
+reask_validator_2 = Contains("e", on_fail='reask')
+fix_validator_1 = Contains("f", on_fail='fix')
+fix_validator_2 = Contains("g", on_fail='fix')
+
+guard = AsyncGuard().use_many(
+    exception_validator,
+    filter_validator,
+    refrain_validator,
+    reask_validator_1,
+    reask_validator_2,
+    fix_validator_1,
+    fix_validator_2
+)
+
+### Trigger the exception validator ###
+error = None
+result: Optional[ValidationOutcome] = None
+try:
+    result = asyncio.run(guard.validate("z", metadata={}))
+
+except ValidationError as e:
+    error = e
+
+assert result is None
+assert error is not None
+assert str(error) == "Validation failed for field with errors: Value must contain a"
+
+
+
+### Trigger the Filter and Refrain validators ###
+result = asyncio.run(guard.validate("a", metadata={}))
+
+assert result.validation_passed is False
+# The output was filtered or refrained
+assert result.validated_output is None
+assert result.reask is None
+
+
+
+### Trigger the Reask validator ###
+result = asyncio.run(guard.validate("abc", metadata={}))
+
+assert result.validation_passed is False
+# If allowed, a ReAsk would have occured
+assert result.reask is not None
+error_messages = [f.error_message for f in result.reask.fail_results]
+assert error_messages == ["Value must contain d", "Value must contain e"]
+
+
+### Trigger the Fix validator ###
+result = asyncio.run(guard.validate("abcde", metadata={}))
+
+assert result.validation_passed is True
+# The fix values have been merged
+assert "f" in result.validated_output
+assert "g" in result.validated_output
+print(result.validated_output)
+```

docusaurus/sidebars.js

@@ -66,7 +66,7 @@ const sidebars = {
         "concepts/streaming_fixes",
       ],
     },
-    "concepts/parallelization",
+    "concepts/concurrency",
     "concepts/logs",
     "concepts/telemetry",
     "concepts/error_remediation",