feat(importer-rework): enhance table rule_metadata (#3863) #3885
+177
−1
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
- add mgm_id column to rule_metadata - delete and add fk(not ready yet) - delete and add Unique from rule_uid to (rule_uid, mgm,id) - get data from rule table for mgm_ids in rule_metadata
…porter-rework-rule_metadataEnhance
tpurschke
reviewed
Nov 5, 2025
Contributor
tpurschke
left a comment
tpurschke
left a comment
There was a problem hiding this comment.
Looking good but I need some more time for testing this. Die you try both a clean install and an upgrade?
Contributor
Author
|
I performed a clean installation and tested the upgrade script several times. |
…//github.com/Imat00/firewall-orchestrator into feature/importer-rework-rule_metadataEnhance
tpurschke
requested changes
Nov 10, 2025
Contributor
tpurschke
left a comment
tpurschke
left a comment
There was a problem hiding this comment.
The constraint does not seem correct yet.
multiple versions of a rule with the same uid is not an issue as long as all of these occur on the same management.
…//github.com/Imat00/firewall-orchestrator into feature/importer-rework-rule_metadataEnhance
tpurschke
reviewed
Nov 10, 2025
Contributor
There was a problem hiding this comment.
found an error in import caused by the change here, so I guess we have to adapt the importer at the same time:
2025-11-10T17:28:07+0100 [INFO ] [fwo_api.py :_call_chunked : 217] Processing chunked API call (upsertRuleMetadata)...
2025-11-10T17:28:07+0100 [DEBUG] [fwo_api.py :_call_chunked : 257] Chunk 1: 1000/9295 processed elements.
2025-11-10T17:28:07+0100 [ERROR] [fwo_api.py :_handle_chunked_calls_res: 289] encountered error while handling chunked call: [{'message': 'Not-NULL violation. null value in column "mgm_id" of relation "rule_metadata" violates not-null constraint', 'extensions': {'path': '$.selectionSet.insert_rule_metadata.args.objects', 'code': 'constraint-violation'}}]
2025-11-10T17:28:07+0100 [ERROR] [fwo_api.py :call : 84] FwoImporterError during API call: encountered error while handling chunked call: [{'message': 'Not-NULL violation. null value in column "mgm_id" of relation "rule_metadata" violates not-null constraint', 'extensions': {'path': '$.selectionSet.insert_rule_metadata.args.objects', 'code': 'constraint-violation'}}]
2025-11-10T17:28:07+0100 [ERROR] [common.py :rollBackExceptionHandler : 185] Exception: FwoApiWriteError
2025-11-10T17:28:09+0100 [INFO ] [rollback.py :rollbackCurrentImport : 37] import 435 has been rolled back successfully
2025-11-10T17:28:09+0100 [INFO ] [import_state_controller.p:delete_import : 435] removed import with id 435 completely
2025-11-10T17:28:09+0100 [INFO ] [fwo_api_call.py :complete_import : 329] import_management: import no. 435 for management fwmgmt01-f (id=252) threw errors, total change count: 22074, rule change count: 0, duration: 256s, ERRORS: ['FwoApiWriteError or FwoImporterError: ('failed to write new RulesMetadata: Traceback (most recent call last):\n File "/usr/local/fworch/importer/model_controllers/fwconfig_import_rule.py", line 612, in addNewRuleMetadata\n import_result = self.import_details.api_call.call(addNewRuleMetadataMutation, query_variables=query_variables, debug_level=self.import_details.DebugLevel, analyze_payload=True)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/fworch/importer/fwo_api.py", line 65, in call\n return_object = self._call_chunked(session, query, query_variables, fwo_globals.debug_level)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/fworch/importer/fwo_api.py", line 252, in _call_chunked\n return_object = self._handle_chunked_calls_response(return_object, response)\n ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^\n File "/usr/local/fworch/importer/fwo_api.py", line 290, in _handle_chunked_calls_response\n raise FwoImporterError(error_txt)\nfwo_exceptions.FwoImporterError: encountered error while handling chunked call: [{\'message\': \'Not-NULL violation. null value in column "mgm_id" of relation "rule_metadata" violates not-null constraint\', \'extensions\': {\'path\': \'$.selectionSet.insert_rule_metadata.args.objects\', \'code\': \'constraint-violation\'}}]\n',) - aborting import']
(importer-venv) fworch@FWORCH1:~/importer$
Contributor
Author
'Not-NULL violation. null value in column "mgm_id" of relation "rule_metadata" violates not-null constraint' is caused by the change that made mgm_id in rule_metadata NOT NULL. We can discuss how to handle it. |
…//github.com/Imat00/firewall-orchestrator into feature/importer-rework-rule_metadataEnhance
tpurschke
requested changes
Nov 11, 2025
Contributor
tpurschke
left a comment
tpurschke
left a comment
There was a problem hiding this comment.
see below - displaying violations that prevent an update would be helpful
- missing Rule_uid from rule_metadata on rule - of Rule_uid and multiple mgm_id(s)
…//github.com/Imat00/firewall-orchestrator into feature/importer-rework-rule_metadataEnhance
…//github.com/Imat00/firewall-orchestrator into feature/importer-rework-rule_metadataEnhance
…) on rule_uid get displayed
…//github.com/Imat00/firewall-orchestrator into feature/importer-rework-rule_metadataEnhance
|
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Addresses #3863
Add column mgm_id INT with foreign key reference to management
Remove rule_uid unique constraint and its fk(temporarily)
Upgrade script:
Post-processing:
update: