When facing production incidents we usually need quick mitigation to put out the fire and there is no time to deeply look at the underlying causes, treating symptoms and not the problem.
The goal of an RCA process is to discover the real cause behind the incident to fully understand how to solve it, prevent it in the future and have a record of successful strategies in order to share knowledge and repeat things that worked.
Incidents could be defined as events that cause disruption to or a reduction in the quality of a production service or product feature which requires an emergency response.
In addition to discovering the root cause, we should strive to provide context and information that will result in an action or a decision: good analysis is actionable analysis.
The content on this repository should be shared once the incidents are mitigated or resolved so that potential vulnerabilities are not exposed.
In order to perform an Incident Analysis, you can choose the tool that better fits the situation, some common examples are: 5 Whys, Fishbone Diagram (many potential causes, cause & effect) or Change Impact Analysis
Incidents Severity is categorized on the following levels:
- SEV-1: Critical issues impacting more than 50% of our users (a.k.a “Oh Fuck!”). The incident degrades the experience to a point in which the user decides to drop Decentraland Platform. Requires immediate resolution
- SEV-2: Critical system issue actively impacting a limited number of users. The users can still interact in Decentraland word but they get frustrated by the inability to live a full experience. Requires immediate resolution
- SEV-3: Stability or minor user impacting issue that requires immediate attention from the service owner, otherwise it might become a SEV-2 incident. Very restricted incident that is internally visible and should be mitigated as soon as possible; without extended user awareness and impacting non-critical flows
- SEV-4: Minor issue requiring action but not affecting the ability to use the platform
- SEV-5: Cosmetic issues or bugs not affecting the users’ ability to use the platform, but it's relevant to give awareness to the other teams
To add new incidents use the date of the event as the Id with the following format YYYY-MM-DD. If there is more than one event on the same date you may need to use a suffix as part of the file name.
- 2026-07-07 Remote users disappear when teleporting back to Genesis Plaza from a World
- 2026-06-18 Mass service request failures ("Premature close") after Node.js security patch hardened keep-alive socket pooling
- 2026-05-04 Intermittent Crashing in Genesis Plaza (Part #2)
- 2026-04-17 Intermittent Crashing in Genesis Plaza (Mitigation)
- 2026-03-17 Users unable to login with Crypto wallets / Magic options
- 2026-03-11 Profile dapp wallet connection broken due to thirdweb dependency mismatch
- 2026-03-04 Avatar creation flow hung and login failures due to corrupt identity
- 2025-11-18 Cloudflare Global Network experiencing issues
- 2025-11-15 Genesis Plaza appearing in Worlds
- 2023-10-03 Marketplace search not working
- 2023-08-30 Users are not able access authenticated services
- 2023-08-24 Users are not able to join Decentraland
- 2023-08-17 Mic remains open when releasing the T key
- 2023-05-23 Cannot log in to Goerli network
- 2023-05-16 Places API is inaccessible
- 2023-05-08 Some users are not loading
- 2023-04-13 Users are not visible in any realm other than Heimdallr
- 2023-03-28 User connections constantly reconnected to the same realm
- 2023-03-23 Chats showing out of order
- 2023-03-22 NFT names not displaying as alias
- 2023-03-09-2 Ghost mode in builder
- 2023-03-09 Decentraland Homepage is down
- 2023-02-27 Get Friends, Private Chat and Friends Requests not working
- 2023-02-14 Reference client cannot be launched
- 2023-02-06 Social Service Migration
- 2023-02-02-2 Transak widget not working
- 2023-02-02 The teleport get freezed for all users using DEBUG_MODE
- 2023-02-01 Loading an avatar with Thunder Earrings is crashing the client
- 2023-01-12-2 Users unable to login to DG realm
- 2023-01-12 Some users are not able to list or make friend requests
- 2023-01-11 Users with many wearables are being shown an empty list
- 2023-01-09 Users unable to obtain their correct profiles
- 2023-01-05 Desktop launcher doesn't launch correct version
- 2022-12-27 SDK Preview doesn't work
- 2022-12-09 Wrong online users metric on the status page
- 2022-12-07-3 Some users do not see the execution of Custom Emotes
- 2022-12-07-2 Higher than normal crashes on desktop platforms (windows)
- 2022-12-07 Some Realms show partial info of others
- 2022-12-05-03 Events API inaccessible
- 2022-12-05-02 Crash when opening backpack
- 2022-12-05 Chat & Friends service unavailable
- 2022-11-28 Scenes MessageBus not working in production
- 2022-11-17 Changes in user profile not updating in peer perspective
- 2022-11-15 NFTs with animated gif thumbnails have stopped showing thumbnails
- 2022-11-13 Users joined to the #mvmf channel noticing huge lags
- 2022-11-04 3d models from other scenes appearing on other scense
- 2022-10-27 Marketplace failed to display NFTs
- 2022-10-17 Some 3D models not rendering
- 2022-09-08 Users not able to see or chat with each other
- 2022-09-01 Users not able to save passport
- 2022-08-18 Scenes not loading in Europe region
- 2022-08-02 The Graph indexing delay prevented users from changing their wearables
- 2022-07-25 Cloudflare XSS protection prevented some users to deploy scenes or smart wearables
- 2022-07-12 Catalyst node continuously rebooted after an update rollout
- 2022-06-06 Social metrics tracking discrepancies
- 2022-05-18 Some issues were detected after the explorer release on
- 2022-04-22 Infura outage caused problem with different Services
- 2022-02-12 CDN proxies stopped working affecting the ability to join Decentraland and some sites
- 2022-02-05 Wearables not loading on some users backpack due to corrupted dropped wearable
- 2026-09-02 A stored javascript: forum link in Builder ran in a curator's session
- 2026-09-02 The Auth request page previewed one typed-data payload while the wallet signed another
- 2026-09-02 Rental listing cancellation ignored which signer bumped the asset index
- 2026-08-27 The Auth request page dispatched transaction fields its review never showed
- 2026-08-25 Signed-fetch metadata keys fell outside the signature, letting a scene act as the visiting user
- 2026-08-13 A profile deployment could carry another user's address in its avatar identity fields
- 2026-08-12 Invalid scene base breaks deployment identity across Genesis City and Worlds
- 2026-08-11 Case-sensitive scene-signer check in signed-fetch lets a scene act as the visiting user
- 2026-08-07 Unauthenticated worker bootstrap re-handed the engine's shared memory in the Bevy Explorer web client
- 2026-08-07 A URL query parameter selected the trusted super-user scene in the Bevy Explorer web client
- 2026-08-07 Scene sandbox escape reaches same-origin capabilities and persisted session credentials in the Bevy Explorer web client
- 2026-07-30 Unauthorized third-party emote deployment hijacks an approved third-party wearable's pointer
- 2026-07-26 Client-supplied
asset_pack_idlets any user insert assets into another user's Builder asset pack - 2026-07-23 Missing third-party manager authorization on the Builder collection publish route
- 2026-07-23 Stored CSS defacement of DAO proposal pages via unsanitized Snapshot vote rationale
- 2026-07-22 Scene thumbnail filename injects markup into the Places social endpoint (stored XSS / defacement)
- 2026-07-20 Marketplace accepts bids priced in an arbitrary ERC-20 that the UI presents to the seller as MANA
- 2026-07-01 Missing state-machine validation lets any user forge non-consensual friendships and bypass the ONLY_FRIENDS voice gate
- 2026-06-08 Parcel-restricted world collaborator can deploy to and delete scenes across the entire world
- 2026-05-18 Prompt injection in unity-explorer Claude PR review workflow
- 2026-04-07 Any Authenticated User Can Modify or Permanently Delete Any Event
- 2026-03-04 Unauthenticated newsletter unsubscription of other users
- 2025-10-23 GitHub Actions Script Injection
- 2025-09-29 Exposure of RTMP password in Admin Tools UI
- 2025-08-25 XSS through state parameter
- 2025-05-26 Cache poisoning through WAF and a big amount of headers
- 2025-05-23 Blocking users through cookie poisoning and WAF
- 2025-02-27 AWS Key Exposed
- 2025-01-07 Temp Atlas Server DoS
- 2024-12-25 Ability leaks the IP address and user agent of the other user
- 2024-12-18 Ability leaks the IP address and user agent of the other user
- 2024-12-16 Leaked email addresses from specific studio
- 2024-12-14 Unrestricted external integration of Intercom Widget
- 2024-12-07 Leaked deprecated AWS Dev account's keys
- 2024-12-03 Public IP Leaked for events
- 2024-10-08 Archipelago handshake enabled signing any message for signedFetch
- 2024-03-19 Allows user 1 to create text message for other user 2
- 2024-03-14 Stored XSS in custom link on
decentraland.org/profile/accounts/{id} - 2024-03-09 CRLF injection can make many subdomains of decentraland.org totally inaccessible to any specific user
- 2024-03-08 Open redirect on
/auth/setup?redirectTo= - 2024-03-01 Any user can add themselves as a manager of any job they know they ID
- 2024-03-01 Files uploaded by studios, job creators and for conversations are publicly available
- 2024-02-28 Leak of API token through path traversal leads to arbitrary code execution, XSS etc
- 2024-02-28 Email Verification Bypass on Decentraland Studios Signup - No User interaction required
- 2024-02-27 Misconfiguration in X causes various vulnerabilities
- 2024-2-21 peer.decentraland.org multiple sites arbitrary file reads [LFI]
- 2024-01-28 Campaign Role Misconfiguration: Owner Demotion via Collaborator Privileges
- 2024-01-26 Vulnerability in Deployment Rights Assignment for Decentraland Names
- 2024-01-14 Unrestricted Webpage Modification and Rendering of External Resources
- 2024-01-06 Adding Items to any User's Collections
- 2023-08-31 DOM-based XSS via SSO_URL parameter on https://play.decentraland.org/
- 2023-07-31 Marketplace Expired Listing Issue
- 2023-01-24 Discord Broken Link Hijack on Decentraland DAO Transparency Dashboard - Official DCL DAO Discord servers
- 2022-07-05 Potentially outdated prices provided by the implementation of ChainlinkOracle
- 2022-07-06 Take over of broken or expired Links
- 2022-07-13 Arbitrary Modification content stored on S3
- 2022-07-20 Cloudflare bypass for Biz environment
- 2022-08-11 Broken access control when deleting single items
- 2022-08-12 Subdomain takeover of osquery.decentraland.org
- 2022-08-23 Stored XSS - Execute Malicious Javascript on Victim's Browser
- 2022-08-28 AWS Credentials leaked in Docker Image
- 2022-11-07 SQL injection on governance API
- 2022-11-18 Misconfigured SSO Function Allows Authenticated Access To Grafana
- 2022-11-22 Dangling Call from wMana