What are zombie objects

[Bios-Marcel]

Hey

I have recently encountered a bug reported, where object IDs couldn't be found in the store.

The very same object IDs were however constantly being reported as Zombie objects in the commandline.

So I have some questions:

1. What exactly does Zombie mean?
2. Is this an ES bug or my fault?
3. What can I do to avoid this?
4. What can I do to fix this after it occurred?

Maybe it would be nice to have a section in the official readme, if this is something that can be avoided by the user.

[hg-ms]

The expression “Zombi-Object-ID” was coined some years ago for object ids that belonged to objects deleted from the storage files but still being referenced from another object in the storage files. This state was caused by a fixed error in the storage garbage collector that could cause a corrupted storage target.
In current versions the only known way to produce zombie ids is to have more the one active storage instance accessing (writing to) the same storage target.

[Bios-Marcel]

Hm, I see.

1. Given we have only recently introduced EclipseStore, it can't be due to that old bug.
2. I thought multi-write access is prohibited, as in you'll get an exception a-la "already in use"?

[hg-ms]

Given we have only recently introduced EclipseStore, it can't be due to that old bug.

EclipseStore has a predecessor project: Microstream.

I thought multi-write access is prohibited, as in you'll get an exception a-la "already in use"?

Yes, there should be an StorageExceptionInitializationexception, but if the access is done from different processes things get more complicated, for that case there is the option to use a Lock File

[Bios-Marcel]

EclipseStore has a predecessor project: Microstream.

I know, but we've never used that :)

Yes, there should be an StorageExceptionInitializationexception, but if the access is done from different processes things get more complicated, for that case there is the option to use a Lock File

Ah, I assumed it automatically did that.

Due to the fact we use H2, this is already guaranteed for us. But we had a short timeframe in which we did not use H2 ... maybe it happened during that time.

[Bios-Marcel]

So, is there any way to get rid of the zombie objects now or is the database a lost cause?

Because this actually causes follow-up issues.

[hg-ms]

Without more details it’s hard to say if there is a chance to repair the storage, but most likely it’s broken.
Did you already try to restart the storage?
How many missing ID do you see?

[Bios-Marcel]

Yeah, restarting made no difference.

It's just one object, given it is always the same ID.

It doesn't really contain anything critical and it happened on an internal database. I am just afraid it might happen for a customer 😓

[fh-ms]

Zombie object references mean an object cannot be found in the storage. This can only happen if data is lost at the storage level. A file/blob has been deleted, or multiple processes have accessed the storage. This scenario can be prevented by ensuring that only one application has concurrent access to the underlying storage.

One tool that helps to ensure this is Lock Files.

If a critical storage needs repairing, this is a case for professional support. Please reach out to support@microstream.one