Open
Conversation
add AWS CLI in dockerfile update kafka with kafka related environments (because there's no kafka entrypoint /env stuff
…d, and also fix the issue of scheduled deletions of env
kafka : fitting a whole ass http is bad, so i p use a non-http workloads
1. Created EFS Module - Full Terraform module
for AWS EFS file systems
2. Added EFS Support to ECS - Updated ECS
service module to support EFS volume mounts
3. Created EFS File System -
fs-0081e926b6106e69d with:
- Encrypted storage
- General Purpose performance mode
- Bursting throughput
- Mount targets in both availability zones
- Access point with proper POSIX permissions
(UID/GID 1000)
4. Updated Kafka Configuration - Mounted EFS at
/var/lib/kafka/data for persistent storage
5. Added Security Groups - Configured NFS (port
2049) access between ECS and EFS
6. Updated IAM Permissions - Added EFS mount
permissions to ECS task role
✅ Current Status:
- Kafka Task: RUNNING and HEALTHY
- Task Definition: Revision 14 (with EFS)
- Error Count: 0 DUPLICATE_BROKER_REGISTRATION
errors
- Storage: Data persisted to EFS at
/var/lib/kafka/data
- Schema Registry: Connected and operational
📊 Verification:
✓ EFS File System: fs-0081e926b6106e69d
(available)
✓ Kafka Health: HEALTHY
✓ Logs: No duplicate registration errors
✓ Consumer Offsets: Loading from persistent
storage
✓ Topics: _schemas partition created
successfully
🔧 What This Fixes:
- ✅ Kafka data survives container restarts
- ✅ Broker registration state persists
- ✅ No more DUPLICATE_BROKER_REGISTRATION
errors
- ✅ Consumer offsets and metadata preserved
- ✅ Topics and partitions persist across
deployments
How It Works Now 1. Client creates WebSocket connection 2. Server accepts the connection 3. Server records the connection in its registry 4. Server sends confirmation message to client 5. Client receives confirmation and resolves the connection promise 6. Client sends match request to API 7. Server checks WebSocket registry (with retry backup) 8. Match request succeeds This eliminates the race condition by ensuring both sides agree the connection is ready before proceeding with the matching flow.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.