unnamed fields should simply skip the stream

[fzakaria]

If i put an unnamed field in my definition I see that it stores the value in a variable (i.e. m__unnamed2 below)

void nix_nar_t::type_regular_t::file_content_t::_read() {
  m_len_contents = m__io->read_u8le();

  n__unnamed1 = true;

  if (nar_offset() < 0) {
    n__unnamed1 = false;

    m__unnamed1 = m__io->read_bytes(0);
  }

  m__unnamed2 = m__io->read_bytes(len_contents());

  m_padding = m__io->read_bytes(
      kaitai::kstream::mod(8 - kaitai::kstream::mod(len_contents(), 8), 8));
}

it would be great to instead seek the m__io forward the specified size.

The purpose of this is I am writing a spec for an archive format and I want to avoid serializing the contents of each file.
I have an instance that is a pointer to the start of the contents that I can use on a separate stream to read to if I want to emit the file.

This avoids keeping the whole structure of the archive format in memory.

I originally filed it in the runtime but I think that was incorrect and it's a compiler bug.
kaitai-io/kaitai_struct_cpp_stl_runtime#81

[Mingun]

Unnamed fields can be useful in some contexts, for example, in inspectors. The true feature, that is needed instead, it is to separate parsing logic from the storing logic. Compiler should generate two+ entities:

• parser that accepts byte stream and returns positions of the fields
• storage struct which contains necessary fields and calls the parser to fill them
  • depending on needs compiler can generate several structs that either to store all fields, or only named; stored fields can be stored as byte slices and/or typed fields (for example, KSY u2 can be stored as Rust [u8; 2] and as u16)

[fzakaria]

@Mingun yes that would help or to somehow make it lazy.
I see the C++ output has a --no-auto-read but that reads whole types still and I found it didn't accomplish what I needed.

This is something I had to fix in our hand-rolled parser NixOS/nix#14273 which I was hoping to replace with Kaitai.

Do you have any advice?

[generalmimon]

@fzakaria There is a general consensus that unnamed fields should be stored, see #664 (comment) and #1064. #664 actually proposes something similar to what you suggest, but only for _unnamed fields using contents. I tend to agree with @dgelessus's comment there - #664 (comment):

I think this would be better implemented in combination with private/hidden fields (#525) - those are intended for use in finished specs (unlike unnamed fields) and you can be certain that they won't be used by "application code" (i. e. anything outside the code generated by ksc), so it should be safe to not store values of private/hidden contents fields.

But if I look specifically at the field where you care about this:

  m__unnamed2 = m__io->read_bytes(len_contents());

... it looks like from the semantic point of view, this shouldn't be always skipped, because you would ideally still want to see this field in a visualizer, for example. So the thing you suggest as an alternative, i.e.:

or to somehow make it lazy.

... makes much more sense. We already have an issue that proposes this feature: #133

[fzakaria]

@generalmimon thank you for your comment; I guess we can close this as a duplicate -- #133 would suffice.

As an aside:
Do you have any advice for my spec I am contributing to the Nix repo [ref]

The way I am trying to match on the closing ) is a bit lame and I have to put if: 'kind.body == "entry"' on every field in the type.
I am not sure if there is a better approach here.

[generalmimon]

@fzakaria:

Do you have any advice?

Well, if you want to make things lazy in Kaitai Struct, instances are usually the answer. It's very likely that you can achieve what you want using instances in this case too (although it won't be as nice and easy as adding lazy: true proposed in #133, because it hasn't been implemented yet).

To be honest, unfortunately, the simplest solution is most likely to patch the generated code to replace the m__unnamed2 = m__io->read_bytes(len_contents()); line with the equivalent m__io->seek(...) call, which simply skips the file contents. If I were to do this, I would create a script to compile this .ksy specification, which would first call kaitai-struct-compiler and then perform this text replacement. This would require the searched line to be found, otherwise the script would fail (i.e. we want 1 replacement to occur, not 0 replacements, which a naive sed invocation would not enforce). This should ensure that if the generated code changes in the future for some reason (due to a change in the .ksy specification or an upgrade to a new version of KSC), the ineffectiveness of the replacement will not go unnoticed.

If you want to keep it purely at the .ksy level, I'm pretty sure it's technically possible using instances, but it probably won't be very nice, as it will require some additional wrapper types and some pos calculations. I probably wouldn't recommend it...

[generalmimon]

@fzakaria:

As an aside:
Do you have any advice for my spec I am contributing to the Nix repo [ref]

You can simplify nar.ksy:33-34:

      - id: padding
        size: (8 - (len_str % 8)) % 8

In Kaitai Struct, % is actually a proper modulo (see https://doc.kaitai.io/user_guide.html#_arithmetic_operators) in the usual mathematical sense, meaning that the result will be non-negative for positive divisors even if the dividend is negative. So it can become just this (and this would be idiomatic in Kaitai Struct, because it can't get any simpler):

      - id: padding
        size: -len_str % 8

WolframAlpha can be used to verify that it's equivalent (note the (-x) mod 8 in "Alternate form"): https://www.wolframalpha.com/input?i=%288+-+%28x+mod+8%29%29+mod+8

The way I am trying to match on the closing ) is a bit lame and I have to put if: 'kind.body == "entry"' on every field in the type.
I am not sure if there is a better approach here.

This is a relatively common occurrence when writing .ksy specifications. The solution is to extract all the fields with the same if expression into a new user-defined type. This way, you can apply the if only once where you use this type, instead of having to repeat it for each individual field.