What are zombie objects

[Bios-Marcel]

Hey

I have recently encountered a bug reported, where object IDs couldn't be found in the store.

The very same object IDs were however constantly being reported as Zombie objects in the commandline.

So I have some questions:

1. What exactly does Zombie mean?
2. Is this an ES bug or my fault?
3. What can I do to avoid this?
4. What can I do to fix this after it occurred?

Maybe it would be nice to have a section in the official readme, if this is something that can be avoided by the user.

[hg-ms]

The expression “Zombi-Object-ID” was coined some years ago for object ids that belonged to objects deleted from the storage files but still being referenced from another object in the storage files. This state was caused by a fixed error in the storage garbage collector that could cause a corrupted storage target.
In current versions the only known way to produce zombie ids is to have more the one active storage instance accessing (writing to) the same storage target.

[Bios-Marcel]

Hm, I see.

1. Given we have only recently introduced EclipseStore, it can't be due to that old bug.
2. I thought multi-write access is prohibited, as in you'll get an exception a-la "already in use"?

[hg-ms]

Given we have only recently introduced EclipseStore, it can't be due to that old bug.

EclipseStore has a predecessor project: Microstream.

I thought multi-write access is prohibited, as in you'll get an exception a-la "already in use"?

Yes, there should be an StorageExceptionInitializationexception, but if the access is done from different processes things get more complicated, for that case there is the option to use a Lock File

[Bios-Marcel]

EclipseStore has a predecessor project: Microstream.

I know, but we've never used that :)

Yes, there should be an StorageExceptionInitializationexception, but if the access is done from different processes things get more complicated, for that case there is the option to use a Lock File

Ah, I assumed it automatically did that.

Due to the fact we use H2, this is already guaranteed for us. But we had a short timeframe in which we did not use H2 ... maybe it happened during that time.

[Bios-Marcel]

So, is there any way to get rid of the zombie objects now or is the database a lost cause?

Because this actually causes follow-up issues.

[hg-ms]

Without more details it’s hard to say if there is a chance to repair the storage, but most likely it’s broken.
Did you already try to restart the storage?
How many missing ID do you see?

[Bios-Marcel]

Yeah, restarting made no difference.

It's just one object, given it is always the same ID.

It doesn't really contain anything critical and it happened on an internal database. I am just afraid it might happen for a customer 😓

[fh-ms]

Zombie object references mean an object cannot be found in the storage. This can only happen if data is lost at the storage level. A file/blob has been deleted, or multiple processes have accessed the storage. This scenario can be prevented by ensuring that only one application has concurrent access to the underlying storage.

One tool that helps to ensure this is Lock Files.

If a critical storage needs repairing, this is a case for professional support. Please reach out to support@microstream.one

[Bios-Marcel]

So, we actually had this issue again.

This time, I can 100% say that only one process at a time ever accessed the database.

I can't tell exactly what caused this, but my assumption is an ungraceful shutdown by the user.

Shouldn't it at least be possible to remove the faulty reference or allow the application to get information about it and allow removing it manually somehow?

[JustinVoitel]

@Bios-Marcel We're experiencing the same issue with one of our clients. The database is around 4GB in size and runs in a Tomcat Docker container using Spring Boot + Vaadin + EclipseStore.

Despite using a lockfile, we still haven't identified the root cause of the zombie entries. They seem to appear randomly—often after an application update (i.e., the Docker container is restarted). At some later point during runtime, zombies show up, and with the next restart, the database becomes corrupted.

We've discovered a few workarounds that might help:

---

1. Eager-Storing Before Shutdown

Calling Eager-Storing (we call it Deep-Store) on the DataRoot before shutting down the app seems to prevent zombie creation.

• Depending on DB size and system resources, this process may take some time (for our ~4GB DB, it takes 3–5 minutes).
• You could trigger it either manually via a UI button or automatically in a serviceDestroy listener.

---

2. Emergency Recovery from a Broken Store

If it's too late and your store is already broken (or Deep-Store didn't help), here's what worked for us:

• We forked the eclipse-store repository.
• Then, we modified this line to log the error instead of throwing an exception.
• Built and deployed this patched version to our private GitHub Maven repository.
• Temporarily switched to using our fork instead of the official release.

This allowed us to start the application without crashing and inspect the store. Interestingly, in almost every case, we couldn't find any actually missing objects (possibly due to GC-related behavior). Only once did we need to manually recreate a few entries from logs.

After this, running a Deep-Store repaired the broken references, and we were able to safely switch back to the official EclipseStore dependency.

---

3. Reliable Backups (Avoiding Broken Snapshots)

Because zombies occur at runtime, any backup taken after their appearance is already broken.

We found two ways to deal with this:

• Set up alerts (e.g., via Grafana) that notify you when zombie-related log entries appear, so you can take action immediately.

or

• Use a backup system based on Jackson serialization:

  • Ensure all objects in your DataRoot are (de)serializable.
  • Serialize the entire state to JSON, then deserialize it into a fresh DataRoot.
  • This eliminates both zombie references and outdated legacy type mappings.
  • Bonus: JSON backups are significantly smaller (~600MB vs. ~4GB for the binary store).

---

Additional Notes

• Docker suspicion: We suspect Docker might play a role here. The application ran for ~2 years without any issues before we containerized it. Shortly after moving to Docker, the zombie problems began.

• One reproducible case (sort of): While working on a new app (outside Docker), I once managed to reproduce the issue using a large Lazy object (~100k+ objects). I cleared it and re-stored a copy multiple times in quick succession. Unfortunately, I haven’t been able to reproduce it since.

[Bios-Marcel]

Thanks for the helpful information!

My assumption is also that it is related to Lazy, but I haven't done further investigation yet.

The Eager Store solution is interesting. I am not sure how it works internally, but does eager store by any chance load uncached lazies, just to store them again? I could imagine that would be slow.

Given eager storing everything does the job, one could also store everything once a night and then do a full backup of the known working state. Definitely not applicable in every environment, but works for most 9-5 applications I guess!

[JeremiasIsnardyEA]

@JustinVoitel
Thanks for sharing! I went through your message and found a lot of similarities. 😊
I also had the luck of having a full backup from a moment shortly before the data got corrupted. With that, I was able to analyze the zombie objects using the OID in the /store-data/{storageName}/object/{oid} endpoint of EclipseStore.
What I saw was that the data of those objects were single attributes of a class—nothing serious, just a string or whatever. No lazy values though!
We also went the route of persisting all data again as JSON so we can restore from there. We've already had cases where nearly all of our full backups were corrupted...
Most importantly, we also host our EclipseStore application in Docker.
I've also kept an eye on server processes (Docker updates, VM backups, etc.), but so far without reproducible evidence.

[Flo9520]

Hello all,

any news on this topic? We are facing the same issue here from time to time (also our application is inside a docker container 😄).
Most of the times when something is missing it is fields of very odd objects like the localTime field of the LocalDateTime object which you would never touch on it's own.
I am still not sure if this is a bug by ES or a misuse of the storing mechanism in my code, either way I am pretty worried of this bug happening in production.

[Bios-Marcel]

No news from my perspective, just sitting here, praying it doesn't happen in production.

[hg-ms]

Hi,
Could you provide some more information on the problem you encountered with those Zombi IDs?
In what environment do you run Eclipse-Store?
Which configuration did you apply to Eclipse-Store (Storage-target, housekeeping, channels, etc.)?
Did the storage become unloadable?
When did you get the Zombie-ID warning? E.g. during startup?
Are there any exception thrown?
Is it possible that you supply a reproduction scenario?

[JustinVoitel]

@Bios-Marcel I just found out you can override the content for specific objectId's, so it wont bother you with annoying logs (Storage GC marking encountered zombie....)
Here are the Docs: https://docs.eclipsestore.io/manual/storage/addendum/troubleshooting.html#_overwriting_and_existing_object_by_its_objectid

I Just tested it and it works perfectly!

[Bios-Marcel]

@hg-ms Do you have any idea how that could happen?

[hg-ms]

Did you run the storage inside a docker container that has been shut down without shutting down the storage?
If so you might try to add a ShutdownHook (Runtime.addShutdownHook) to do so.

You may also try to identify the missing object referents, please see storage graph analysis.

[Bios-Marcel]

I did an analysis already. There were 3 missing objects. However, for 2 of them, no parent could be found and for one of them i did find the parent. The third one was a Lazy<> object that was pointing to something non-existent.

We are running a windows desktop application. Usually it's shut down via the desktop.
In rare cases it MIGHT happen that the user shuts it down via task manager.

However, is there no way to defend against storage corruption in this case? I am a bit confused as there's a transaction concept. Are there limits to what the transaction can handle?

[hg-ms]

I have doubts that the problem is caused by the I/O issues that would be noticed by the transaction logs, that would cause other exceptions.
If the problems occur at the same objects, I think it is more likely that inconsistent object states are persisted. Maybe the graph analysis allows you to identify and check the related code in your app as the reverse search returns the full path from the storage root to the missing objects.

[Bios-Marcel]

I don't know where it occurred in the past, as there was no tooling to find out.

And as for now, it's not quite easily reproducible, so I don't know yet, whether it would keep happening with that field. Either way, I already know exactly which field was affected. However, I don't know how that helps me. What do you mean with inconsistent object states?