Skip to content

Feat/bulk json import#12677

Open
StuCM wants to merge 8 commits intoarchesproject:dev/8.2.xfrom
flaxandteal:feat/bulk-json-import
Open

Feat/bulk json import#12677
StuCM wants to merge 8 commits intoarchesproject:dev/8.2.xfrom
flaxandteal:feat/bulk-json-import

Conversation

@StuCM
Copy link
Copy Markdown
Contributor

@StuCM StuCM commented Mar 31, 2026

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)

Description of Change

  • Create a bulk_archesfile script to handle import of 100+ resources
  • Maintained original logic for smaller imports but add a length check to route to the correct function
  • Added validation to apply the same checks as previously ran through tile.save()
    • pre_tile_save
    • check_for_missing_nodes
    • populate_missing_nodes
    • validate
  • Added a flag to fire functions if needed for the bulk import
  • Caching serialized graph per graph instead of loading it per tile
  • Replaced the per tile DB constraint query with a single in memory constraint call and checks against this.
  • ES indexing called after the function is run
  • Improved error reports and created list of failed resources. If a resource fails, it does not stop the entire import.

Issues Solved

  • Increasing speed of imports
  • Improving error reporting with data issues

Checklist

  • Unit tests pass locally with my changes
  • I have added tests that prove my fix is effective or that my feature works
  • I have added necessary documentation (if appropriate)

Ticket Background

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants