Let's figure out how Items work!

[default-kramer]

When you place a tree, at least two things happen in the STGDAT file:

• The blockdata will contain a special block ID (such as 2047 or 1780) in each space that the tree occupies. For example, a sapling is 4 blocks high (I think) so you would see the special block ID appear 4 times in the blockdata. (This special block ID is called the "Category" in the turtle-insect data files.)
• A single 24-byte record will contain details about the tree, such as which way it is facing (e.g. north) and the chunk-relative xyz coordinate of the item.

But how does the game know which 24-byte records belong to which chunks? Since the xyz coordinate is within the chunk, it must come from somewhere else.

Details of the 24B Record

The LSByte of the total number of 24B records is stored at 0024 E7CD. The 24B records begin near 0024 E???. It seems there is space reserved for up to 0xC8000 total records, meaning the records would end at (start + 24 * 0xC8000).

I'm not actually sure where one record ends and the next one starts, but it looks like the records align at 0029 5C01 (±24i). Assuming this alignment is correct, the contents of the record are:

• Bytes 0-5
  • Change every time the game is saved? (except zeros)
  • Bytes 20-21 always match bytes 4-5?
  • Appear to always be a multiple of 24 (assuming byte 0 is the LSByte)
  • Frequently count up by 24
  • All zero indicates "empty / no record"? (or maybe that's bytes 16-21)
• Bytes 6-7
  • Always zero?
  • May simply be the MSBytes of 0-5
• Byte 8
  • LSByte of the Item ID
• Byte 9 : xxxiiiii where
  • xxx is the lowest 3 bits of the 5-bit X coordinate (within the chunk, 0 <= X <= 31)
  • iiiii is the higher bits of the Item ID
• Byte 10 : yyyyyyxx where
  • yyyyyy is the lowest 6 bits of the Y coordinate (0 <= Y <= 95)
  • xx is the highest 2 bits of the 5-bit X coordinate (within the chunk, 0 <= X <= 31)
• Byte 11 : ddzzzzzy where
  • dd is the compass direction (NSEW)
  • zzzzz is the 5-bit Z coordinate (within the chunk, 0 <= Z <= 31)
  • y is the highest bit of the Y coordinate (when set, Y >= 64)
• Bytes 12-14
  • If you look only at the high nibble of byte 12, then they count up by 16 most of the time?
  • ...But not always, I've seen this counter violated near the end of the records
• Byte 15
  • ???
• Bytes 16-21
  • Change every time the game is saved (except zeros)
  • Bytes 20-21 always match bytes 4-5 ?
  • All zero indicates "empty / no record"? (or maybe that's bytes 0-5)
• Bytes 22-23
  • Always zero?
  • May simply be the MSBytes of 16-21

Bytes 0-7 and 16-23 seem similar, and appear to be related to the internals of the data structure.

[default-kramer]

But how does the game know which 24-byte records belong to which chunks? Since the xyz coordinate is within the chunk, it must come from somewhere else.

My Wild Guess: There could be an intermediate index mapping each chunk ID to a list of "pages", for example chunk 35 -> [2, 5, 33] would tell you that the 2nd, 5th, and 33rd "page" of 24-byte records all belong to chunk 35. If this is true, you would only gain insight into this mechanism when a new page is required, so maybe the key to figuring this out is filling up a few chunks with as many items as possible.

Another interesting test would be to place items that span across chunks.

[default-kramer]

Above is what I know about how items are stored in a DQB2 STGDAT file. If anyone knows more than me, please share and maybe we can finally crack this mystery! (@turtle-insect @Sapphire645 @Mugafo and anyone else)

[turtle-insect]

Just a little update.
turtle-insect/DQB2@44b2bbc

[turtle-insect]

support world coordinates
turtle-insect/DQB2@96cb509

[default-kramer]

I think you got it! I'm still investigating, but this looks amazing!

[Sapphire645]

https://drive.google.com/file/d/162ZJDUgTHHUrmA572a-jrbKEhWCqAJKS/view?usp=drive_link

Found this region of memory. Looks like it gets updated whenever an item is placed.
It's emulation so the region/location is not relevant.

[Mugafo]

Hey, been awhile since I've been able to work on this, busy busy and a cat puked on my computer frying it awhile back so I lost some item code I had at the time that I never committed to my tile editor. I had gotten to essentially the same point of what you've found on items.

As the screen vid Sapphire645 posted shows, items also place values in the block data area, these handle special checks for being able to place other blocks/items in that spot (though it doesn't appear to be the only check) and you can do neat things with that like deleting the item, but leaving the block data behind and create custom shadows etc.

From what I can remember, there was also a size setting in the bytes somewhere, as some items have a small, normal, and large setting that could be used by editing it. I remember playing around with the clock item having the different sizes down next to each other on the map. With the positioning data you can make items overlap and you can cause weird glitches with some of the other bytes. I remember making a pot indestructible when hit but would spit out 3 pots to pick up (liking tying into the block properties for the type of soil that would drop 3 seeds).

I'd need to get back into it, my assembly knowledge isn't very good, so I was having trouble at the time trying to figure out what all was happening when an item was placed.

[default-kramer]

Thank you! That is interesting, especially the bit about the sizes. I had assumed there was some lookup table for each Item ID that told you e.g. "this bed is 2x2x1" and "this tree is 1x1x5". But if that info is stored in the item record that will make certain plans of mine a lot easier.

[Mugafo]

I should have specified, from what I saw it still took up the same number of "blocks" the size difference was only visual. I'll have to see if I can find my cloud save for that map I had been editing.

[Mugafo]

Wish I had saved my stuff to my google drive. I've tried looking into a few times, but it's a little harder to debug the memory with the current pc I'm using.

[default-kramer]

Small progress update:

As turtle-insect demonstrated, there are 2 relevant ranges of memory:

• The 24-byte records, located at 0x24E7D1 + 24i
• The 4-byte records, located at 0x150E7D1 + 4i

The 4-byte record contains:

• The chunk offset data needed to know which chunk the item is in
• Some index which I am calling a "defragmentation index" - When loading the file, the game will place the records in order according to this index.

The key idea is that if you load and immediately save your file (without adding or destroying anything) then the resulting save file will Edit: always sometimes have the N 24-byte records and the N 4-byte records in order such that the record at position i will have a defragmentation index of i. This makes it easy to read the item records -- you just have to read the first N records and verify the index. If there is any fragmentation it seems much harder and I don't know how to do it.

For reference, here is an example of how fragmentation can occur:

• You load a save file with 20 items
• The records are placed in order
• You destroy the item of the 13th record
  • The 21st record (currently inactive) is assigned a defrag index of 13
  • The 13th record is deactivated and its defrag index will be something > 20
• You add a new item
  • It goes into the 21st record with the defrag index of 13

So it appears to be something like an indexed ring buffer, where new items are always added to the end, but I don't really know. For now I plan on requiring the user to use the game to defragment the item records if they want to manipulate them.

But I may not be able to resume this work for a while because I am rethinking my entire project and going to try using a relational database as the foundation.

[default-kramer]

Bummer, the game doesn't actually perform the defragmentation as reliably as I thought it did 😢

[default-kramer]

I've just learned that the debug build has a feature that will automatically build blueprints. And I do (mostly) understand how items are represented in blueprints. This means if I want to do large-scale item and block manipulation, I can just generate blueprints (well up to 4 at a time I guess) and use the autobuild to quickly complete them!

Of course, I'd still like to be able to directly manipulate items in the STGDAT file, but until that time this might be the perfect workaround, pending some testing...

[Sapphire645]

We really need to get this whole thing cracked... To start, all items's data is stored at this location 0x24E7D1 right?
So, if I were to theoretically only want a list of all the items that currently exist in the STGDAT, I just need to get the byte 8 + half of byte 9 of each item. Is this correct?

[default-kramer]

If your STGDAT file is "defragmented" (see my Jun 8 post), then you could theoretically get a list of all the item IDs in the STGDAT using bytes 8 and 9 of the 24-byte records. If it is not defragmented, you might see IDs from items that were destroyed and no longer in the stage (or maybe even garbage data; I'm not sure how it works).

[default-kramer]

I wonder if you create a brand new Buildertopia if that STGDAT is always defragmented. I would bet it is, but I should test that sometime...

[Sapphire645]

I'm making the full ID list, do you have an incomplete one?

[default-kramer]

I can't remember where I stored it, but I also had less than 10 items so probably not even worth it.

[Sapphire645]

Ok... I think, I think I got something for how the chunk number works... I did some testing and this is what I ended up with for IoA chunk values...

I don't really have any idea of what I'm doing?? so if its wrong tell me.

[Sapphire645]

Visual! Got the grid mapping working. I'll update the source code soon. I think I'll add a "Chunk Editor Mask Cropping" for the minimap exporter too.

[turtle-insect]

Successfully expanded chunks

turtle-insect/DQB2@12709fa

[default-kramer]

I did some testing and you can expand the IoA up to 700 chunks! Going up to 701 crashes. This is most likely because Moonbrooke is 700 chunks, and the game reserves enough space to uncompress a STGDAT file of 700 chunks. If you uncompress 701 chunks, it overwrites whatever comes next in memory and crashes.

(Although Moonbrooke isn't a perfect 700, my test worked when I did a perfect 700. But my 700-chunk IoA also had the "Moonbrooke bug" which makes sense because it really should be called the "700-chunk bug")

[turtle-insect]

simple chunk editor

[Sapphire645]

I have found that editing the chunk grid wont affectthe minimap. New chunks wont update, no matter what. Might need some more investigation. Perhaps there's a minimap tile grid?

EDIT:

Seems so.

[default-kramer]

I've just had another wild guess about the 24-byte data structure.

Consider that autosave probably works something like this: A background thread simply copies the main memory directly into a separate buffer to be zipped and saved to disk. While this is happening, the main thread does not pause. So the main thread might be in the middle of destroying or adding an item while the autosave copies the data. If this is true, the mystery bytes may implement something like a "checksum" [1] so that no matter what the autosave happened to capture, there is a sensible/consistent way to interpret the data when it is loaded.

For example, maybe when an item is destroyed the very first byte written invalidates the checksum. And when an item is added the very last byte written validates the checksum. So when autosave data is loaded, if the checksum is invalid it can safely be interpreted as "there's no item here."

[1] I put "checksum" in quotes because I don't know what to call it. The pattern is much more regular than any checksum I've seen. Maybe it's more analogous to a "commit protocol" or something.