fix(graphql): prevent memory leak and deadlock in subscription resolvers

[Sanchit2662]

Summary

This PR fixes a critical concurrency issue in the ChaosCenter GraphQL subscription layer that could lead to unbounded memory growth and a process-wide deadlock under normal UI usage.

Specifically, GetInfraEvents subscriptions were leaking channels after client disconnects, and SendInfraEvent could block indefinitely while holding a shared mutex. Over time, this caused the GraphQL server to become unresponsive with no crash logs or clear error signals.

The fix ensures proper subscription cleanup, prevents blocking sends, and hardens related cleanup paths against concurrent map access.

---

Fix

1. Proper subscription cleanup on disconnect

Channels are now removed from the publisher slice when the subscription context is cancelled:

go func() {
    <-ctx.Done()
    data_store.Store.Mutex.Lock()
    channels := data_store.Store.InfraEventPublish[projectID]
    for i, ch := range channels {
        if ch == infraEvent {
            data_store.Store.InfraEventPublish[projectID] =
                append(channels[:i], channels[i+1:]...)
            break
        }
    }
    data_store.Store.Mutex.Unlock()
}()

---

2. Non-blocking event delivery to prevent deadlocks

Event publishing no longer blocks on slow or disconnected subscribers:

for _, observer := range r.InfraEventPublish[infra.ProjectID] {
    select {
    case observer <- &newEvent:
    default:
        // skip slow/dead subscriber
    }
}

This ensures one stalled subscription cannot block the entire system.

---

3. Thread-safe cleanup in related subscriptions

Cleanup paths in GetPodLog, GetKubeObject, and GetKubeNamespace now properly guard map deletes with the shared mutex, preventing concurrent map access panics.

---

Impact

• Memory leak eliminated: subscription channels are no longer leaked.
• Deadlock prevented: event publishing cannot block while holding the mutex.
• Improved resilience: slow or disconnected clients degrade gracefully.
• Stability improved: prevents rare but severe production outages in ChaosCenter.

---

Types of changes

• Bugfix (non-breaking change which fixes an issue)

---

Checklist

• I have read the [CONTRIBUTING](https://github.com/litmuschaos/litmus/blob/master/CONTRIBUTING.md) doc
• I have signed the commit for DCO to be passed
• Lint and unit tests pass locally with my changes

[Sanchit2662]

Hi @PriteshKiri, @amityt , @SarthakJain26
I’ve updated the PR to address the issue and adjusted the implementation accordingly. This helps avoid a potential memory leak and deadlock in the GraphQL subscription flow by improving how subscriptions are cleaned up and how events are delivered.

Whenever you get a chance, I’d really appreciate a review. Thanks!

[Copilot]

Pull request overview

This PR addresses concurrency problems in the ChaosCenter GraphQL subscription layer, focusing on preventing blocked publishers and cleaning up subscription listeners to avoid leaked channels and map access hazards.

Changes:

• Made SendInfraEvent publish using a non-blocking channel send to avoid indefinitely blocking while holding the shared mutex.
• Added GetInfraEvents subscription cleanup to remove the subscriber channel on ctx.Done().
• Wrapped several subscription cleanup delete(...) operations (ExperimentLog, KubeObjectData, KubeNamespaceData) with the shared mutex.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 5 comments.

File	Description
chaoscenter/graphql/server/pkg/chaos_infrastructure/service.go	Switches infra event fan-out to non-blocking sends to prevent deadlocks.
chaoscenter/graphql/server/graph/chaos_infrastructure.resolvers.go	Adds disconnect cleanup for infra event subscriptions and mutex-protects cleanup deletes for several subscription maps.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[SarthakJain26]

@Sanchit2662 please check the comments from copilot