[Security Solution][Rules Management] Separate actions import logic from rules import (elastic#216380)

## Summary

Redo of elastic#193471
Closes elastic/security-team#8644

> Fixes a bug where importing a rule fails with a connector into a space
where (1) the connector already exists, and (2) the existing connector
was exported and re-imported from another space. The import logic in
this scenario effectively tries to convert the action ID on the rule
import twice. The second conversion attempt tries to use the old action
ID to look up the correct new action ID in a map, however, in this test
scenario the action ID has already been updated by legacy SO ID
migration logic and there is no map entry with the new ID as a key. The
result is that the second attempt sets the action ID to undefined,
resulting in an import failure.

The root cause of the bug is that we have two different places in the
rule import logic where action IDs are migrated. The first ID migration
was done by `migrateLegacyActionsIds` prior to importing rule actions,
and the second migration was done by `importRuleActionConnectors` after
importing the actions. `importRuleActionConnectors` used a lookup table
to convert old IDs to new IDs, but if the connector already existed and
had an `originId` then the rule action would already be migrated by
`migrateLegacyActionsIds`. The lookup table used by
`importRuleActionConnectors` does not have entries for migrated IDs,
only the original IDs, so in that case the result of the lookup is
`undefined` which we assign to the action ID.

This PR reworks the logic to create a clean separation between action
and rule import. We now import the connectors first, ignoring the rules,
then migrate action IDs on the rules afterwards. This handles connectors
changing IDs in any way, either through the 7.x->8.0 migration long ago
or IDs changing on import if there are ID conflicts. Only after the
connectors are imported and rule actions are migrated do we then verify
if each rule action references a connector ID that actually exists with
the new `checkRuleActions` function, replacing
`checkIfActionsHaveMissingConnectors` and related functions that were
also buggy.

Finally, as a nice side effect this rework removes "rule action
connector missing" errors out of the `action_connector_errors` part of
the response. `action_connector_errors` is reserved for errors importing
connectors specifically. If a rule action is missing a connector and
therefore we don't import the rule, that's a rule error and it's
represented in the `errors` part of the response. Since the shape of the
response is not changing, I don't consider this a breaking change but
rather a bug fix.

## Repro Steps

Repro Steps
1. Download the export file below and change the extension back to
.ndjson from .json (github does not allow .ndjson files

[rules_export.json](https://github.com/user-attachments/files/17065272/rules_export.json)
2. Import the rule and connector into a space (default is fine)
3. Create a new space
4. Import the rule and connector into the new space
5. Import the rule and connector into the new space again, but check the
`Overwrite existing connectors with conflicting action "id"` box.
Observe the failure.

---------

Co-authored-by: kibanamachine <[email protected]>

Author: 2 people

x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/routes/utils.ts

@@ -72,7 +72,6 @@ export const createBulkErrorObject = ({
     };
   } else {
     return {
-      rule_id: '(unknown id)',
       error: {
         status_code: statusCode,
         message,

x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/rule_management/api/rules/import_rules/route.test.ts

@@ -202,7 +202,6 @@ describe.skip('Import rules route', () => {
               message: `Unexpected token 'h', "this is not"... is not valid JSON`,
               status_code: 400,
             },
-            rule_id: '(unknown id)',
           },
         ],
         success: false,
@@ -350,14 +349,12 @@ describe.skip('Import rules route', () => {
               message: 'rule_id: Required',
               status_code: 400,
             },
-            rule_id: '(unknown id)',
           },
           {
             error: {
               message: 'rule_id: Required',
               status_code: 400,
             },
-            rule_id: '(unknown id)',
           },
         ],
         success: false,

x-pack/solutions/security/plugins/security_solution/server/lib/detection_engine/rule_management/api/rules/import_rules/route.ts

@@ -26,6 +26,7 @@ import {
 } from '../../../../routes/utils';
 import { createPrebuiltRuleAssetsClient } from '../../../../prebuilt_rules/logic/rule_assets/prebuilt_rule_assets_client';
 import { importRuleActionConnectors } from '../../../logic/import/action_connectors/import_rule_action_connectors';
+import { validateRuleActions } from '../../../logic/import/action_connectors/validate_rule_actions';
 import { createRuleSourceImporter } from '../../../logic/import/rule_source_importer';
 import { importRules } from '../../../logic/import/import_rules';
 
@@ -123,13 +124,9 @@ export const importRulesRoute = (router: SecuritySolutionPluginRouter, config: C
             maxExceptionsImportSize: objectLimit,
           });
           // report on duplicate rules
-          const [duplicateIdErrors, parsedObjectsWithoutDuplicateErrors] =
-            getTupleDuplicateErrorsAndUniqueRules(rules, request.query.overwrite);
-
-          const migratedParsedObjectsWithoutDuplicateErrors = await migrateLegacyActionsIds(
-            parsedObjectsWithoutDuplicateErrors,
-            actionSOClient,
-            actionsClient
+          const [duplicateIdErrors, rulesToImportOrErrors] = getTupleDuplicateErrorsAndUniqueRules(
+            rules,
+            request.query.overwrite
           );
 
           // import actions-connectors
@@ -138,20 +135,17 @@ export const importRulesRoute = (router: SecuritySolutionPluginRouter, config: C
             success: actionConnectorSuccess,
             warnings: actionConnectorWarnings,
             errors: actionConnectorErrors,
-            rulesWithMigratedActions,
           } = await importRuleActionConnectors({
             actionConnectors,
-            actionsClient,
             actionsImporter,
-            rules: migratedParsedObjectsWithoutDuplicateErrors,
             overwrite: request.query.overwrite_action_connectors,
           });
 
-          // rulesWithMigratedActions: Is returned only in case connectors were exported from different namespace and the
-          // original rules actions' ids were replaced with new destinationIds
-          const parsedRuleStream = actionConnectorErrors.length
-            ? []
-            : rulesWithMigratedActions || migratedParsedObjectsWithoutDuplicateErrors;
+          const migratedRulesToImportOrErrors = await migrateLegacyActionsIds(
+            rulesToImportOrErrors,
+            actionSOClient,
+            actionsClient
+          );
 
           const ruleSourceImporter = createRuleSourceImporter({
             config,
@@ -160,8 +154,20 @@ export const importRulesRoute = (router: SecuritySolutionPluginRouter, config: C
             prebuiltRuleObjectsClient: createPrebuiltRuleObjectsClient(rulesClient),
           });
 
-          const [parsedRules, parsedRuleErrors] = partition(isRuleToImport, parsedRuleStream);
-          const ruleChunks = chunk(CHUNK_PARSED_OBJECT_SIZE, parsedRules);
+          const [parsedRules, parsedRuleErrors] = partition(
+            isRuleToImport,
+            migratedRulesToImportOrErrors
+          );
+
+          // After importing the actions and migrating action IDs on rules to import,
+          // validate that all actions referenced by rules exist
+          // Filter out rules that reference non-existent actions
+          const { validatedActionRules, missingActionErrors } = await validateRuleActions({
+            actionsClient,
+            rules: parsedRules,
+          });
+
+          const ruleChunks = chunk(CHUNK_PARSED_OBJECT_SIZE, validatedActionRules);
 
           const importRuleResponse = await importRules({
             ruleChunks,
@@ -180,9 +186,9 @@ export const importRulesRoute = (router: SecuritySolutionPluginRouter, config: C
           const importErrors = importRuleResponse.filter(isBulkError);
           const errors = [
             ...parseErrors,
-            ...actionConnectorErrors,
             ...duplicateIdErrors,
             ...importErrors,
+            ...missingActionErrors,
           ];
 
           const successes = importRuleResponse.filter((resp) => {