OpenCTI connector can leave partial objects on failure (no IDs stored)

[gks281263]

Summary

The OpenCTI connector does many sequential API calls in run() (org, marking, observable, labels, report, external ref, linking). If something fails in the middle, some objects are already created in OpenCTI. The connector goes to FAILED but does not clean them or store their IDs in report.errors. Operators cannot see what was created in that failed run, and retry can create duplicates.

What happens today

• File: api_app/connectors_manager/connectors/opencti.py, method run().
• Steps are separate API calls: org L79–91, marking L96–103, observable L124–128, labels loop L133–138, report L141–158, external ref L161–165, linking L163–171.
• No transaction or rollback. On any exception, Plugin.start() in api_app/classes.py calls after_run_failed(e), which only appends str(e) to report.errors and sets status FAILED. No cleanup, no recording of created OpenCTI IDs.

Why this is practical

Example: connector has soft_time_limit 30s. If the sequence takes longer, Celery raises SoftTimeLimitExceeded mid-run. Everything created before that (observable, labels, report, etc.) stays in OpenCTI; IntelOwl shows FAILED with no list of IDs. I simulated a failure after report creation and verified partial objects remain in OpenCTI while IntelOwl shows FAILED.

Impact

• Partial objects in OpenCTI (e.g. report not linked to observable) with no way to see which IDs were created in that run.
• report.errors only has the exception text; operators must search OpenCTI manually.
• Retry runs the same sequence again and creates duplicate report/observable/labels because there is no idempotency by job_id.

Minimal suggested fix

Track created OpenCTI IDs inside run() in a small local dict and append them to report.errors on failure (e.g. observable, report, external_reference, labels — "OpenCTI created IDs before failure: observable=…, report=…, …"). Scope: only OpenCTI connector, no public API change, no success-path change, only extra context on failure.

---

If you agree this is valid, I can open a small PR with only the ID tracking part first, keeping the change minimal and safe. We can then discuss rollback / idempotency separately if you prefer.

[gks281263]

Hi @mlodic , I kept this change small and focused only on improving failure visibility, without changing the connector flow. My idea is to strengthen reliability step by step. If this direction looks correct, later we can also discuss better rollback or idempotency handling in a structured way.

[gks281263]

I have opened PR #3337 which handles the partial-state tracking part in a minimal way.

For the rollback / idempotency improvements, do you prefer to continue that discussion in this issue, or should I open a separate issue to keep the scope clean?

[github-actions]

This issue has been marked as stale because it has had no activity for 30 days. If you are still working on this, please provide some updates.

[gks281263]

This is being addressed in PR #3337 .