Multiple-create: Request Validation (#2057)

## Why make this change?
With support for multiple-create, we introduce additional flexibility in
the GraphQL schema where we make the referencing fields in an entity
optional because their value can be derived from insertions in the
referenced entity. So to say, _the values of referencing fields in a
referencing entity can have 3 sources of truth_. The value for a
referencing field can be derived via:
1. User input data
2. Insertion in the parent referenced entity (if current entity is a
referencing entity for a relationship with its parent entity)
3. Insertion in the child referenced entity (if the current entity is a
referencing entity for a relationship with its child entity)

**_They can all exist at the same time, or none of them can. Presence of
more than one sources of truth leads to possible conflicting values for
the referencing column. Thus, at a time, we want exactly one of them to
be present - so that we exactly one source of truth_**. This needs us to
do request validation during request execution, so that only valid GQL
mutations reach the query generation/query execution stage. This PR adds
the logic for the same.


## What is this change?

- Added class `MultipleMutationInputValidator.cs` to perform the
validations for a multiple-create mutation.
- Added class `MultipleMutationEntityInputValidationContext` to store
relevant information while performing validations on a particular
entity.
- A call to the method
`MultipleMutationInputValidator.ValidateGraphQLValueNode()` will be made
_just after Authorization_ - request validation only occurs when the
user is authorized to execute the mutation.
- The method `MultipleMutationInputValidator.ValidateGraphQLValueNode()`
makes recursive call to itself to parse out the entire create mutation
request body based on an entity or relationship basis. Meaning, this
method will be called on the top-level entity, followed by all the
relationships for that entity, followed by all the relationships for the
entity's relationships and so on....
- The actual validation for the input data for an entity (top-level
entity or a nested entity) which includes column fields and relationship
fields is done by the method
`GraphQLRequestValidator.ValidateObjectFieldNodes()` which is called
from the method `ValidateGraphQLValueNode()`.
- The method `ValidateObjectFieldNodes()` contains logic to ensure that
when the current entity acts as the referencing entity for the
relationships with its target entity, then the current entity should not
specify a value for the referencing fields - as the values for the
referencing fields will be derived from the insertion in the referenced
target entity. Note: The determination of referencing entity is done via
a call to `MultipleCreateOrderHelper.GetReferencingEntityName()` method
added via: #2056.
- The method `ValidateObjectFieldNodes()` makes calls to
1. `ValidateAbsenceOfReferencingColumnsInTargetEntity()`: To validate
that the current entity does not specify value for referencing columns
to its parent entity (when parent entity is the referenced entity).
2. `ProcessRelationshipField()`: To process a relationship field and
does several things. Please refer to method summary.
3. `ValidateAbsenceOfRepeatedReferencingColumn()`: This method is called
for all the relationships with the current entity which are included in
the request body. This ensures that referencing columns in the
referencing entity (whichever entity, either source or target turns out
as the referencing entity) does not hold references to multiple columns
in the referenced entity - to avoid multiple sources of truth for the
referencing column's value. (Issue:
#2019)
4. `ValidatePresenceOfRequiredColumnsForInsertion()`: To ensure that we
have one source of truth for values for referencing fields.
5. `ValidateRelationshipFields()`: Recursively perform the same
validations for all the relationships with the current entity included
in the mutation request body.
`
## How was this tested?
- [x] Integration Tests - Added tests to
`MultipleMutationIntegrationTests.cs` to validate:
1. Throwing exception for absence of source of truth for referencing
column for create one mutations
2. Throwing exception for absence of source of truth for referencing
column for create multiple mutations.
3. Throwing exception for presence of multiple sources of truth for
referencing column for create one mutations.
4. Throwing exception for presence of multiple sources of truth for
referencing column for create one mutations.

## Sample Request(s)
1. Request:

![image](https://github.com/Azure/data-api-builder/assets/34566234/b6ef03bf-016d-4a45-9adf-dbe422683f73)
    Response:
    

![image](https://github.com/Azure/data-api-builder/assets/34566234/cbec3e22-3a9e-4bfd-abc8-0fb4fd45cfdd)

2. Request:

![image](https://github.com/Azure/data-api-builder/assets/34566234/6c58f6b9-54b8-42b3-8945-39272023c2bf)

Response:

![image](https://github.com/Azure/data-api-builder/assets/34566234/b0d5f689-d029-4ba9-9aeb-40fca7eb1b85)

3. Request and response:

![image](https://github.com/Azure/data-api-builder/assets/34566234/c39ad294-412c-45d9-9fdd-5efb69908836)

---------

Co-authored-by: Shyam Sundar J <[email protected]>
Co-authored-by: Sean Leonard <[email protected]>

Author: 3 people

config-generators/mssql-commands.txt

@@ -1,4 +1,4 @@
-init --config "dab-config.MsSql.json" --database-type mssql --set-session-context true --connection-string "Server=tcp:127.0.0.1,1433;Persist Security Info=False;User ID=sa;Password=REPLACEME;MultipleActiveResultSets=False;Connection Timeout=5;" --host-mode Development --cors-origin "http://localhost:5000"
+init --config "dab-config.MsSql.json" --database-type mssql --set-session-context true --connection-string "Server=tcp:127.0.0.1,1433;Persist Security Info=False;User ID=sa;Password=REPLACEME;MultipleActiveResultSets=False;Connection Timeout=5;" --host-mode Development --cors-origin "http://localhost:5000" --graphql.multiple-create.enabled true
 add Publisher --config "dab-config.MsSql.json" --source publishers --permissions "anonymous:read"
 add Stock --config "dab-config.MsSql.json" --source stocks --permissions "anonymous:create,read,update,delete"
 add Book --config "dab-config.MsSql.json" --source books --permissions "anonymous:create,read,update,delete" --graphql "book:books"
@@ -33,6 +33,8 @@ add User_NonAutogenRelationshipColumn --config "dab-config.MsSql.json" --source
 add UserProfile --config "dab-config.MsSql.json" --source "user_profiles" --permissions "anonymous:*" --rest true --graphql true
 add User_AutogenRelationshipColumn --config "dab-config.MsSql.json" --source "users" --permissions "anonymous:*" --rest true --graphql true
 add User_AutogenToNonAutogenRelationshipColumn --config "dab-config.MsSql.json" --source "users" --permissions "anonymous:*" --rest true --graphql true
+add User_RepeatedReferencingColumnToOneEntity --config "dab-config.MsSql.json" --source "users" --permissions "anonymous:*" --rest true --graphql true
+add UserProfile_RepeatedReferencingColumnToTwoEntities --config "dab-config.MsSql.json" --source "user_profiles" --permissions "anonymous:*" --rest true --graphql true
 add GetBooks --config "dab-config.MsSql.json" --source "get_books" --source.type "stored-procedure" --permissions "anonymous:execute" --rest true --graphql true
 add GetBook --config "dab-config.MsSql.json" --source "get_book_by_id" --source.type "stored-procedure" --permissions "anonymous:execute" --rest true --graphql false
 add GetPublisher --config "dab-config.MsSql.json" --source "get_publisher_by_id" --source.type "stored-procedure" --permissions "anonymous:execute" --rest true --graphql true --graphql.operation "query"
@@ -145,6 +147,9 @@ update ArtOfWar --config "dab-config.MsSql.json" --permissions "authenticated:*"
 update User_NonAutogenRelationshipColumn --config "dab-config.MsSql.json" --relationship UserProfile_NonAutogenRelationshipColumn --target.entity UserProfile --cardinality one --relationship.fields "username:username"
 update User_AutogenRelationshipColumn --config "dab-config.MsSql.json" --relationship UserProfile_AutogenRelationshipColumn --target.entity UserProfile --cardinality one --relationship.fields "userid:profileid"
 update User_AutogenToNonAutogenRelationshipColumn --config "dab-config.MsSql.json" --relationship UserProfile_AutogenToNonAutogenRelationshipColumn --target.entity UserProfile --cardinality one --relationship.fields "userid,username:userid,username"
+update User_RepeatedReferencingColumnToOneEntity --config "dab-config.MsSql.json" --relationship UserProfile --target.entity UserProfile --cardinality one --relationship.fields "username,username:profilepictureurl,username"
+update UserProfile_RepeatedReferencingColumnToTwoEntities --config "dab-config.MsSql.json" --relationship book --target.entity Book --cardinality one --relationship.fields "userid:id"
+update UserProfile_RepeatedReferencingColumnToTwoEntities --config "dab-config.MsSql.json" --relationship publisher --target.entity Publisher --cardinality one --relationship.fields "userid:id"
 update GetBook --config "dab-config.MsSql.json" --permissions "authenticated:execute" --rest.methods "Get"
 update GetPublisher --config "dab-config.MsSql.json" --permissions "authenticated:execute"
 update GetBooks --config "dab-config.MsSql.json" --permissions "authenticated:execute" --graphql.operation "Query" --rest.methods "Get"

src/Config/DataApiBuilderException.cs

@@ -31,6 +31,10 @@ public enum SubStatusCodes
         /// </summary>
         EntityNotFound,
         /// <summary>
+        /// The relationship for a pair of source/target entities does not exist.
+        /// </summary>
+        RelationshipNotFound,
+        /// <summary>
         /// Request failed authentication. i.e. No/Invalid JWT token
         /// </summary>
         AuthenticationChallenge,

src/Core/Models/MultipleMutationEntityInputValidationContext.cs

@@ -0,0 +1,48 @@
+// Copyright (c) Microsoft Corporation.
+// Licensed under the MIT License.
+
+namespace Azure.DataApiBuilder.Core.Models
+{
+    /// <summary>
+    /// Class to represent input for an entity in a multiple-create request.
+    /// </summary>
+    public class MultipleMutationEntityInputValidationContext
+    {
+        /// <summary>
+        /// Current entity name.
+        /// </summary>
+        public string EntityName { get; }
+
+        /// <summary>
+        /// Parent entity name. For the topmost entity, this will be set as an empty string.
+        /// </summary>
+        public string ParentEntityName { get; }
+
+        /// <summary>
+        /// Set of columns in the current entity whose values are derived from insertion in the parent entity
+        /// (i.e. parent entity would have been the referenced entity).
+        /// For the topmost entity, this will be an empty set.</param>
+        /// </summary>
+        public HashSet<string> ColumnsDerivedFromParentEntity { get; }
+
+        /// <summary>
+        /// Set of columns in the current entity whose values are to be derived from insertion in the entity or its subsequent
+        /// referenced entities and returned to the parent entity so as to provide values for the corresponding referencing fields
+        /// (i.e. parent entity would have been the referencing entity)
+        ///  For the topmost entity, this will be an empty set.
+        /// </summary>
+        public HashSet<string> ColumnsToBeDerivedFromEntity { get; }
+
+        public MultipleMutationEntityInputValidationContext(
+            string entityName,
+            string parentEntityName,
+            HashSet<string> columnsDerivedFromParentEntity,
+            HashSet<string> columnsToBeDerivedFromEntity)
+        {
+            EntityName = entityName;
+            ParentEntityName = parentEntityName;
+            ColumnsDerivedFromParentEntity = columnsDerivedFromParentEntity;
+            ColumnsToBeDerivedFromEntity = columnsToBeDerivedFromEntity;
+        }
+    }
+}

src/Core/Resolvers/SqlMutationEngine.cs

@@ -96,6 +96,25 @@ public SqlMutationEngine(
             // If authorization fails, an exception will be thrown and request execution halts.
             AuthorizeMutation(context, parameters, entityName, mutationOperation);
 
+            // Multiple create mutation request is validated to ensure that the request is valid semantically.
+            string inputArgumentName = IsPointMutation(context) ? MutationBuilder.ITEM_INPUT_ARGUMENT_NAME : MutationBuilder.ARRAY_INPUT_ARGUMENT_NAME;
+            if (parameters.TryGetValue(inputArgumentName, out object? param) && mutationOperation is EntityActionOperation.Create)
+            {
+                IInputField schemaForArgument = context.Selection.Field.Arguments[inputArgumentName];
+                MultipleMutationEntityInputValidationContext multipleMutationEntityInputValidationContext = new(
+                    entityName: entityName,
+                    parentEntityName: string.Empty,
+                    columnsDerivedFromParentEntity: new(),
+                    columnsToBeDerivedFromEntity: new());
+                MultipleMutationInputValidator multipleMutationInputValidator = new(sqlMetadataProviderFactory: _sqlMetadataProviderFactory, runtimeConfigProvider: _runtimeConfigProvider);
+                multipleMutationInputValidator.ValidateGraphQLValueNode(
+                    schema: schemaForArgument,
+                    context: context,
+                    parameters: param,
+                    nestingLevel: 0,
+                    multipleMutationEntityInputValidationContext: multipleMutationEntityInputValidationContext);
+            }
+
             // The presence of READ permission is checked in the current role (with which the request is executed) as well as Anonymous role. This is because, for GraphQL requests,
             // READ permission is inherited by other roles from Anonymous role when present.
             bool isReadPermissionConfigured = _authorizationResolver.AreRoleAndOperationDefinedForEntity(entityName, roleName, EntityActionOperation.Read)
@@ -1225,7 +1244,7 @@ private void AuthorizeEntityAndFieldsForMutation(
         ///                         title: "book #1",
         ///                         reviews: [{ content: "Good book." }, { content: "Great book." }],
         ///                         publishers: { name: "Macmillan publishers" },
-        ///                         authors: [{ birthdate: "1997-09-03", name: "Red house authors", author_name: "Dan Brown" }]
+        ///                         authors: [{ birthdate: "1997-09-03", name: "Red house authors", royal_percentage: 4.6 }]
         ///                     })
         ///                 {
         ///                     id
@@ -1236,13 +1255,13 @@ private void AuthorizeEntityAndFieldsForMutation(
         ///                         title: "book #1",
         ///                         reviews: [{ content: "Good book." }, { content: "Great book." }],
         ///                         publishers: { name: "Macmillan publishers" },
-        ///                         authors: [{ birthdate: "1997-09-03", name: "Red house authors", author_name: "Dan Brown" }]
+        ///                         authors: [{ birthdate: "1997-09-03", name: "Red house authors", royal_percentage: 4.9 }]
         ///                     },
         ///                     {
         ///                         title: "book #2",
         ///                         reviews: [{ content: "Awesome book." }, { content: "Average book." }],
         ///                         publishers: { name: "Pearson Education" },
-        ///                         authors: [{ birthdate: "1990-11-04", name: "Penguin Random House", author_name: "William Shakespeare" }]
+        ///                         authors: [{ birthdate: "1990-11-04", name: "Penguin Random House", royal_percentage: 8.2  }]
         ///                     }])
         ///                 {
         ///                     items{

src/Core/Services/MetadataProviders/ISqlMetadataProvider.cs

@@ -208,5 +208,30 @@ public DatabaseObject GetDatabaseObjectForGraphQLType(string graphqlType)
         void InitializeAsync(
             Dictionary<string, DatabaseObject> entityToDatabaseObject,
             Dictionary<string, string> graphQLStoredProcedureExposedNameToEntityNameMap);
+
+        /// <summary>
+        /// Helper method to get the Foreign Key definition in the object definition of the source entity which relates it
+        /// with the target entity. In the Foreign key definition, the table backing the referencing entity acts as the referencing table
+        /// and the table backing the referenced entity acts as the referenced table.
+        /// </summary>
+        /// <param name="sourceEntityName">Source entity name.</param>
+        /// <param name="targetEntityName">Target entity name.</param>
+        /// <param name="referencedEntityName">Referenced entity name.</param>
+        /// <param name="referencingEntityName">Referencing entity name.</param>
+        /// <param name="foreignKeyDefinition">Stores the required foreign key definition from the referencing to referenced entity.</param>
+        /// <returns>true when the foreign key definition is successfully determined.</returns>
+        /// <example>
+        /// For a 1:N relationship between Publisher: Book entity defined in Publisher entity's config:
+        /// sourceEntityName: Publisher (The entity in whose config the relationship is defined)
+        /// targetEntityName: Book (The target.entity in the relationship config)
+        /// referencingEntityName: Book (Entity holding foreign key reference)
+        /// referencedEntityName: Publisher (Entity being referenced by foreign key).
+        /// </example>
+        public bool TryGetFKDefinition(
+            string sourceEntityName,
+            string targetEntityName,
+            string referencingEntityName,
+            string referencedEntityName,
+            [NotNullWhen(true)] out ForeignKeyDefinition? foreignKeyDefinition) => throw new NotImplementedException();
     }
 }

src/Core/Services/MetadataProviders/SqlMetadataProvider.cs

@@ -1911,6 +1911,38 @@ public bool IsDevelopmentMode()
         {
             return _runtimeConfigProvider.GetConfig().IsDevelopmentMode();
         }
+
+        /// <inheritdoc/>
+        public bool TryGetFKDefinition(
+            string sourceEntityName,
+            string targetEntityName,
+            string referencingEntityName,
+            string referencedEntityName,
+            [NotNullWhen(true)] out ForeignKeyDefinition? foreignKeyDefinition)
+        {
+            if (GetEntityNamesAndDbObjects().TryGetValue(sourceEntityName, out DatabaseObject? sourceDbObject) &&
+                GetEntityNamesAndDbObjects().TryGetValue(referencingEntityName, out DatabaseObject? referencingDbObject) &&
+                GetEntityNamesAndDbObjects().TryGetValue(referencedEntityName, out DatabaseObject? referencedDbObject))
+            {
+                DatabaseTable referencingDbTable = (DatabaseTable)referencingDbObject;
+                DatabaseTable referencedDbTable = (DatabaseTable)referencedDbObject;
+                SourceDefinition sourceDefinition = sourceDbObject.SourceDefinition;
+                RelationShipPair referencingReferencedPair = new(referencingDbTable, referencedDbTable);
+                List<ForeignKeyDefinition> fKDefinitions = sourceDefinition.SourceEntityRelationshipMap[sourceEntityName].TargetEntityToFkDefinitionMap[targetEntityName];
+
+                // At this point, DAB guarantees that a valid foreign key definition exists between the the referencing entity
+                // and the referenced entity. That's because DAB validates that all foreign key metadata
+                // was inferred for each relationship during startup.
+                foreignKeyDefinition = fKDefinitions.FirstOrDefault(
+                    fk => fk.Pair.Equals(referencingReferencedPair) &&
+                    fk.ReferencingColumns.Count > 0
+                    && fk.ReferencedColumns.Count > 0)!;
+                return true;
+            }
+
+            foreignKeyDefinition = null;
+            return false;
+        }
     }
 }