Is there an easier way to convert kaitai to raw bytes？

[silver-fang1]

I'm using kaitai parsing network protocol，it works well. But when I want to develop a client to communicate with the server using a specified protocol, I need to generate raw data packets，copy each field into an array，at the end send the array.
So is there an easier way to do this?

[generalmimon]

See https://doc.kaitai.io/faq.html#writing and the issue #27 - Kaitai Struct does not yet officially support writing structures back to bytes. However, there are Java PoC branches called serialization in relevant repositories (kaitai_struct_java_runtime, kaitai_struct_compiler) which can already work well enough for simple formats right now. If Java is acceptable for you to use, you'll need to build compiler from source - just install sbt, checkout the serialization branches in Git and follow this comment: #27 (comment).

The compiler doesn't have writing support for other languages than Java, but there are serialization routines available for some runtime libraries largely made by @jchv: C++ (kaitai-io/kaitai_struct_cpp_stl_runtime#30), C# (kaitai-io/kaitai_struct_csharp_runtime#14), Go (kaitai/writer.go) and Python (kaitai-io/kaitai_struct_python_runtime#48 - this probably still has some flaws pointed out in the review, though). They can make the serialization code easier to make. However, assembling the actual serializing code will be up to you, unfortunately, unless you add serialization support for your target language into the compiler using the existing JavaCompiler code on branch serialization as a template (that should not be too hard, in theory it's just about copying a few methods from JavaCompiler that generate Java-specific serialization method calls and adjusting them so that they generate code valid in your language, but there may be some hidden pitfalls and you still need some basic Scala understanding). But even if you decide not to follow this route and you would just enrich the generated read-only code with _write() method declarations by hand instead, I'd still strongly advise you to at least build the KSC's serialization branch and compile the same format into Java with --read-write CLI option so that you get an idea how to deal with writing enum values, fixed-size byte arrays and strings, delimited byte arrays/strings with terminator, substructures, substreams, repeated structures, etc. (if your format spec uses such things, of course). It might save you a lot of time devising things that have been already invented.

I wish I could give you a more satisfying answer, but this is the best I can tell you right now.

[silver-fang1]

Thanks for your advises, I'm trying to build the KSC's serialization branch and see how it works, but i'm new to java and scala. I installed Intellij and scala plugin, download serialization branch and open it as a project, when I tried to build the project lots of errors appeared. Is there a specific tutorial to build and run kaitai_struct_compiler？

[generalmimon]

@Fgod:

when I tried to build the project lots of errors appeared

It would help if you posted the console output (messages marked as [warn] can be ignored, they do not prevent compilation; but if there are some [error], that is a problem) because I don't know for sure what exact problems occur in your environment. But I also get some cryptic error

[error] java.lang.NoSuchMethodError: sbt.Keys$.reresolveSbtArtifacts()Lsbt/SettingKey;

when I checkout the serialization KSC branch in the current state. On my machine, I resolved it by editing https://github.com/kaitai-io/kaitai_struct_compiler/blob/serialization/project/build.properties like this:

-sbt.version = 1.2.7
+sbt.version = 1.4.3

If this doesn't fix it, please post the console output so that I can help you with that.

Is there a specific tutorial to build and run kaitai_struct_compiler？

Not really (at least not that I'm aware of), but it should be relatively straightforward. Building the KSC only depends on sbt and JRE (= Java Runtime Environment, which supposedly even gets downloaded automatically by sbt if you don't have it already) as written at https://doc.kaitai.io/developers.html#_prerequisites. Normal "stage" build (used for development) is triggered as sbt compilerJVM/stage from the compiler project root, which creates the $COMPILER_DIR/jvm/target/universal/stage/bin/kaitai-struct-compiler Unix shell launch script, which can be used to run the KSC to generate parsing code. It is advisable to add the $COMPILER_DIR/jvm/target/universal/stage/bin/ folder to your path so that you can run just kaitai-struct-compiler from your terminal without having to type the entire path. You can also create a symlink to it (e.g. to shorten it to ksc), but that symlink must be in the same directory (because the launch script uses relative path to reach the .jar files that it runs, and if you launch it from a folder on a different level, the relative path won't work and you get an error Could not find or load main class io.kaitai.struct.JavaMain).

If you have a problem, please don't hesistate to ask.

[silver-fang1]

I successfully build ksc with sbt and installed it on my machine.Also generated a java code with command -w.So now i'm trying to create a java project. Here is what i did:

1. create a java project with maven
2. add io.kaitai dependency in pom.xml
3. create a java class and copy parsing code generated by ksc
4. create a main class to parse data

when i tried to run main() it said:

java: can't find symbol
   Symbol: class ConsistencyError
   Location: package io.kaitai.struct 

java: Symbol not found
     Symbol: Class ReadWrite
     Location: io.kaitai.struct.KaitaiStruct class 

I think step2 was wrong, can you give me some suggestions?

[generalmimon]

@silver-fang1

I think step2 was wrong, can you give me some suggestions?

2. add io.kaitai dependency in pom.xml

The Java runtime library published in the Maven Central repository reflects the 0.9 tag on the master branch of the kaitai_struct_java_runtime repository at the time of releasing the 0.9 version. As I said, serialization is still not officially supported, so the runtime lib on master branch can be only used for parsing - it has only the read*() methods, no write*() methods used by serialization. You need to checkout the serialization branch of kaitai_struct_java_runtime for writing support.

I would simply git clone the Java runtime repo on branch serialization and then include the source code into the project build. I don't know Maven well enough to know how to exactly do this (perhaps @ams-tschoening can help), but when compiling the Java code manually from command-line with javac for example would mean to just list all .java files present in the runtime library as positional arguments (which can be done by some *.java globbing). This is how I would do it (I'm using https://gitter.im/kaitai_struct/Lobby?at=60201d1332e01b4f717f270e as a reference):

Presuming this flat structure:

├─ App.java
└─ my_format.ksy

kaitai-struct-compiler -t java --no-auto-read --read-write my_format.ksy # generates MyFormat.java
git clone --branch serialization --depth 1 https://github.com/kaitai-io/kaitai_struct_java_runtime.git # fills `kaitai_struct_java_runtime/` dir with .java files
mkdir bin/ # for Java .class bytecode files
shopt -s globstar # to find .java files in all subdirectories recursively
javac -encoding UTF-8 -d bin $(ls **/*.java)
java -cp bin App [...] # [...] are command line arguments required by the App.java (if any)

This should work.

[ams-tschoening]

I would simply git clone the Java runtime repo on branch serialization and then include the source code into the project build.

It's most likely easier to build the runtime for itself like any other MVN project, which should result in some JAR-files within the directory hierarchy of the runtime. Afterwards, adopt the build of your own project to reference those JAR files in your local file system. Dependencies required by the runtime, if any at all, can easily be added as dependencies of your own project this way, cached like all other dependencies etc. This way you don't need to care about individual source code files and stuff.

There are multiple ways to reference those JARs or put them into your local MVN mirror or alike.

<dependency>
            <groupId>anything</groupId>
            <artifactId>anything</artifactId>
            <version>anything</version>
            <scope>system</scope>
            <systemPath>${basedir}/lib/jar_name.jar</systemPath>
</dependency>

https://stackoverflow.com/a/51647143/2055163
https://stackoverflow.com/a/4955695/2055163

This is what I'm doing for Gradle:

repositories
{
	jcenter()

	flatDir
	{
		dirs '../Libs Java 3rd/Parsers/Generators/kaitai_struct/runtime/libs_java_3rd_usage/lib'
	}
}
[...]
dependencies
{
	implementation	':kaitai-struct-runtime:0.9', [...]
	runtimeOnly		[...]
}

[silver-fang1]

@generalmimon @ams-tschoening Thanks for your help! I successfully built the program, but there was a problem when parsing the data:
Exception in thread "main" java.lang.NullPointerException at main.main(main.java:34)
At first I thought it was an input data problem or a ksy file problem, but the same data and ksy file run normally with the parser that generated without the -w command.

The ksy file:

meta:
  id: enip_segment
  title: ENIP (Ethernet/IP) segment
  endian: le
doc: |
  Rockwell controller use ENIP communicate with RS5000 software
seq:
  - id: enip_header
    size: 24
    type: enip_header
  - id: command_data
    size-eos: true
    type: command_data
    
types:
  enip_header:
    seq:
        - id: enip_command
          type: u2
        - id: enip_length
          type: u2
        - id: session_handle
          type: u4
        - id: enip_status
          type: u4
        - id: sender_context
          size: 8
        - id: enip_options
          size: 4  
  command_data:
    seq:
        - id: interface_handle
          type: u4
        - id: timeout
          type: u2
        - id: item_count
          type: u2
        - id: address_item
          size: 8
          type: connected_address_item
        - id: data_item
          size: 6
          type: data_item
        - id: enip_data
          size-eos: true
  connected_address_item:
    seq:
        - id: type_id
          type: u2
        - id: address_item_len
          type: u2
        - id: connection_id
          type: u4
  data_item:
    seq:
        - id: type_id
          type: u2
        - id: data_item_len
          type: u2
        - id: cip_seq
          type: u2

The test java script:

import io.kaitai.struct.ByteBufferKaitaiStream;

public class main {
    public static void main(String[] args) {
        byte[] hexArray = new byte[] {
                0x70, 0x00, 0x7D, 0x00, (byte)0xF8, (byte)0xA8, (byte)0x84, 0x01,
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
                0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x02, 0x00,
                (byte)0xA1, 0x00, 0x04, 0x00, (byte)0xB9, 0x22, (byte)0xD3, 0x73,
                (byte)0xB1, 0x00, 0x69, 0x00, 0x44, 0x00, 0x4B, 0x02,
                0x20, 0x67, 0x24, 0x01, 0x07, 0x4D, 0x00, 0x4E,
                0x25, (byte)0x90, 0x21, 0x0F, 0x00, 0x16, 0x70, (byte)0xAA,
                0x50, 0x01, 0x49, 0x00, 0x28, 0x00, 0x00, 0x00,
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
                0x00, 0x00, 0x00, 0x01, 0x03, 0x00, 0x00, 0x00,
                0x00, 0x0F, 0x00, 0x71, (byte)0xEC, 0x09, 0x00, 0x7A,
                0x00, 0x04, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
                0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
                0x00, 0x00, 0x00, 0x00, 0x00
        };

        EnipSegment data = new EnipSegment(new ByteBufferKaitaiStream(hexArray));
        System.out.println(data.enipHeader().enipLength());
    }
}

By the way, if I want to add serialization support for python, which files need to be modified?
kaitai_struct_python_runtime and PythonCompiler in the serialization branch？

[ams-tschoening]

Exception in thread "main" java.lang.NullPointerException at main.main(main.java:34)
At first I thought it was an input data problem or a ksy file problem, but the same data and ksy file run normally with the parser

It would be of help to know which line 34 exactly is. Additionally, if you can trigger that issue by using -w or not, I suggest generating both sources and DIFF/compare them as well. There surely is some important difference if there's a problem in the parser.

[generalmimon]

there was a problem when parsing the data:
Exception in thread "main" java.lang.NullPointerException at main.main(main.java:34)

It seems to me that the NullPointerException is triggered by this line:

System.out.println(data.enipHeader().enipLength());

The problem is that when you enable the --read-write option, it turns --no-auto-read on. This causes the _read() method not to be called automatically from class constructors in the generated parser/serializer, which is the default behavior for parsing-only modules.

So you need to call _read() manually after instantiating the EnipSegment:

         EnipSegment data = new EnipSegment(new ByteBufferKaitaiStream(hexArray));
+        data._read();
         System.out.println(data.enipHeader().enipLength());

[silver-fang1]

You are my savior! data._read() works! but data._write() will throw a BufferOverflowException.
Generated _write method:

    public void _write() {
        KaitaiStream _io__raw_enipHeader = new ByteBufferKaitaiStream(24); 
        this.enipHeader._write(_io__raw_enipHeader);
        this._io.writeStream(_io__raw_enipHeader); 
        this.commandData._write(this._io);
    }

this._io.writeStream(_io__raw_enipHeader); will cause BufferOverflowException.

I tried to enlarge the ByteBufferKaitaiStream size, but it didn't work

[generalmimon]

but data._write() will throw a BufferOverflowException.

Please show the full interaction of your application code (the main.java code, presumably) with the generated module EnipSegment. How do you instantiate it? What KaitaiStream do you pass in there? Do you call _read(), set some values and then try to _write() again? To the same stream or another one? Do you call the _check() method before _write() to ensure consistency?

I need to see your code so I can help.

[silver-fang1]

my main.java file:

1. import io.kaitai.struct.ByteBufferKaitaiStream;
2. 
3. public class main {
4.      public static void main(String[] args) {
5. 
6.         byte[] hexData = new byte[] {
7.                     0x70, 0x00, 0x22, 0x00, 0x00, 0x01, 0x02, 0x12,
8.                     0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
9.                     0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
10.                     0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x02, 0x00,
11.                 (byte)0xA1, 0x00, 0x04, 0x00, 0x01, 0x2D, (byte)0x94, 0x00,
12.                 (byte)0xB1, 0x00, 0x0E, 0x00, 0x03, 0x00, 0x03, 0x02,
13.                     0x20, 0x64, 0x24, 0x01, 0x02, 0x00, 0x01, 0x00,
14.                     0x02, 0x00
15.         };
16. 
17.         EnipSegment data = new EnipSegment(new ByteBufferKaitaiStream(hexData));
18. 
19.         data._read();
20.         data.enipHeader().setEnipCommand(0x55);
21. 
22.         try {
23.             data._check();
24.             data._write();
25.         }catch (Exception e){
26.             System.out.println(e);
27.         }
28. 
29.         System.out.println(data.enipHeader().enipCommand());
30.     }
31. }

generated module:

1. // This is a generated file! Please edit source .ksy file and use kaitai-struct-compiler to rebuild
2. 
3. import io.kaitai.struct.ByteBufferKaitaiStream;
4. import io.kaitai.struct.KaitaiStruct;
5. import io.kaitai.struct.KaitaiStream;
6. import java.io.IOException;
7. import io.kaitai.struct.ConsistencyError;
8. 
9. 
10. /**
11.  * Rockwell controller use ENIP communicate with RS5000 software
12.  */
13. public class EnipSegment extends KaitaiStruct.ReadWrite {
14.     @org.jetbrains.annotations.NotNull
15.     @org.jetbrains.annotations.Contract("_ -> new")
16.     public static EnipSegment fromFile(String fileName) throws IOException {
17.         return new EnipSegment(new ByteBufferKaitaiStream(fileName));
18.     }
19.     public EnipSegment() {
20.         this(null, null, null);
21.     }
22. 
23.     public EnipSegment(KaitaiStream _io) {
24.         this(_io, null, null);
25.     }
26. 
27.     public EnipSegment(KaitaiStream _io, KaitaiStruct.ReadWrite _parent) {
28.         this(_io, _parent, null);
29.     }
30. 
31.     public EnipSegment(KaitaiStream _io, KaitaiStruct.ReadWrite _parent, EnipSegment _root) {
32.         super(_io);
33.         this._parent = _parent;
34.         this._root = _root == null ? this : _root;
35.     }
36.     public void _read() {
37.         this._raw_enipHeader = this._io.readBytes(24);
38.         KaitaiStream _io__raw_enipHeader = new ByteBufferKaitaiStream(_raw_enipHeader);
39.         this.enipHeader = new EnipHeader(_io__raw_enipHeader, this, _root);
40.         this.enipHeader._read();
41.         this._raw_commandData = this._io.readBytesFull();
42.         KaitaiStream _io__raw_commandData = new ByteBufferKaitaiStream(_raw_commandData);
43.         this.commandData = new CommandData(_io__raw_commandData, this, _root);
44.         this.commandData._read();
45.     }
46. 
47.     public void _write() {
48.         KaitaiStream _io__raw_enipHeader = new ByteBufferKaitaiStream(24);
49.         this.enipHeader._write(_io__raw_enipHeader);
50.         this._io.writeStream(_io__raw_enipHeader);
51.         this.commandData._write(this._io);
52.     }
53. 
54.     public void _check() {
55.     }
56.     public static class EnipHeader extends KaitaiStruct.ReadWrite {
57.         public static EnipHeader fromFile(String fileName) throws IOException {
58.             return new EnipHeader(new ByteBufferKaitaiStream(fileName));
59.         }
60.         public EnipHeader() {
61.             this(null, null, null);
62.         }
63. 
64.         public EnipHeader(KaitaiStream _io) {
65.             this(_io, null, null);
66.         }
67. 
68.         public EnipHeader(KaitaiStream _io, EnipSegment _parent) {
69.             this(_io, _parent, null);
70.         }
71. 
72.         public EnipHeader(KaitaiStream _io, EnipSegment _parent, EnipSegment _root) {
73.             super(_io);
74.             this._parent = _parent;
75.             this._root = _root;
76.         }
77.         public void _read() {
78.             this.enipCommand = this._io.readU2le();
79.             this.enipLength = this._io.readU2le();
80.             this.sessionHandle = this._io.readU4le();
81.             this.enipStatus = this._io.readU4le();
82.             this.senderContext = this._io.readBytes(8);
83.             this.enipOptions = this._io.readBytes(4);
84.         }
85. 
86.         public void _write() {
87.             this._io.writeU2le(this.enipCommand);
88.             this._io.writeU2le(this.enipLength);
89.             this._io.writeU4le(this.sessionHandle);
90.             this._io.writeU4le(this.enipStatus);
91.             this._io.writeBytes(this.senderContext);
92.             this._io.writeBytes(this.enipOptions);
93.         }
94. 
95.         public void _check() {
96.             if (senderContext().length != 8)
97.                 throw new ConsistencyError("sender_context", senderContext().length, 8);
98.             if (enipOptions().length != 4)
99.                 throw new ConsistencyError("enip_options", enipOptions().length, 4);
100.         }
101.         private int enipCommand;
102.         private int enipLength;
103.         private long sessionHandle;
104.         private long enipStatus;
105.         private byte[] senderContext;
106.         private byte[] enipOptions;
107.         private EnipSegment _root;
108.         private EnipSegment _parent;
109.         public int enipCommand() { return enipCommand; }
110.         public void setEnipCommand(int _v) { enipCommand = _v; }
111.         public int enipLength() { return enipLength; }
112.         public void setEnipLength(int _v) { enipLength = _v; }
113.         public long sessionHandle() { return sessionHandle; }
114.         public void setSessionHandle(long _v) { sessionHandle = _v; }
115.         public long enipStatus() { return enipStatus; }
116.         public void setEnipStatus(long _v) { enipStatus = _v; }
117.         public byte[] senderContext() { return senderContext; }
118.         public void setSenderContext(byte[] _v) { senderContext = _v; }
119.         public byte[] enipOptions() { return enipOptions; }
120.         public void setEnipOptions(byte[] _v) { enipOptions = _v; }
121.         public EnipSegment _root() { return _root; }
122.         public void set_root(EnipSegment _v) { _root = _v; }
123.         public EnipSegment _parent() { return _parent; }
124.         public void set_parent(EnipSegment _v) { _parent = _v; }
125.     }
126.     public static class CommandData extends KaitaiStruct.ReadWrite {
127.         public static CommandData fromFile(String fileName) throws IOException {
128.             return new CommandData(new ByteBufferKaitaiStream(fileName));
129.         }
130.         public CommandData() {
131.             this(null, null, null);
132.         }
133. 
134.         public CommandData(KaitaiStream _io) {
135.             this(_io, null, null);
136.         }
137. 
138.         public CommandData(KaitaiStream _io, EnipSegment _parent) {
139.             this(_io, _parent, null);
140.         }
141. 
142.         public CommandData(KaitaiStream _io, EnipSegment _parent, EnipSegment _root) {
143.             super(_io);
144.             this._parent = _parent;
145.             this._root = _root;
146.         }
147.         public void _read() {
148.             this.interfaceHandle = this._io.readU4le();
149.             this.timeout = this._io.readU2le();
150.             this.itemCount = this._io.readU2le();
151.             this._raw_addressItem = this._io.readBytes(8);
152.             KaitaiStream _io__raw_addressItem = new ByteBufferKaitaiStream(_raw_addressItem);
153.             this.addressItem = new ConnectedAddressItem(_io__raw_addressItem, this, _root);
154.             this.addressItem._read();
155.             this._raw_dataItem = this._io.readBytes(6);
156.             KaitaiStream _io__raw_dataItem = new ByteBufferKaitaiStream(_raw_dataItem);
157.             this.dataItem = new DataItem(_io__raw_dataItem, this, _root);
158.             this.dataItem._read();
159.             this.enipData = this._io.readBytesFull();
160.         }
161. 
162.         public void _write() {
163.             this._io.writeU4le(this.interfaceHandle);
164.             this._io.writeU2le(this.timeout);
165.             this._io.writeU2le(this.itemCount);
166.             KaitaiStream _io__raw_addressItem = new ByteBufferKaitaiStream(8);
167.             this.addressItem._write(_io__raw_addressItem);
168.             this._io.writeStream(_io__raw_addressItem);
169.             KaitaiStream _io__raw_dataItem = new ByteBufferKaitaiStream(6);
170.             this.dataItem._write(_io__raw_dataItem);
171.             this._io.writeStream(_io__raw_dataItem);
172.             this._io.writeBytes(this.enipData);
173.         }
174. 
175.         public void _check() {
176.         }
177.         private long interfaceHandle;
178.         private int timeout;
179.         private int itemCount;
180.         private ConnectedAddressItem addressItem;
181.         private DataItem dataItem;
182.         private byte[] enipData;
183.         private EnipSegment _root;
184.         private EnipSegment _parent;
185.         private byte[] _raw_addressItem;
186.         private byte[] _raw_dataItem;
187.         public long interfaceHandle() { return interfaceHandle; }
188.         public void setInterfaceHandle(long _v) { interfaceHandle = _v; }
189.         public int timeout() { return timeout; }
190.         public void setTimeout(int _v) { timeout = _v; }
191.         public int itemCount() { return itemCount; }
192.         public void setItemCount(int _v) { itemCount = _v; }
193.         public ConnectedAddressItem addressItem() { return addressItem; }
194.         public void setAddressItem(ConnectedAddressItem _v) { addressItem = _v; }
195.         public DataItem dataItem() { return dataItem; }
196.         public void setDataItem(DataItem _v) { dataItem = _v; }
197.         public byte[] enipData() { return enipData; }
198.         public void setEnipData(byte[] _v) { enipData = _v; }
199.         public EnipSegment _root() { return _root; }
200.         public void set_root(EnipSegment _v) { _root = _v; }
201.         public EnipSegment _parent() { return _parent; }
202.         public void set_parent(EnipSegment _v) { _parent = _v; }
203.         public byte[] _raw_addressItem() { return _raw_addressItem; }
204.         public void set_raw_AddressItem(byte[] _v) { _raw_addressItem = _v; }
205.         public byte[] _raw_dataItem() { return _raw_dataItem; }
206.         public void set_raw_DataItem(byte[] _v) { _raw_dataItem = _v; }
207.     }
208.     public static class ConnectedAddressItem extends KaitaiStruct.ReadWrite {
209.         public static ConnectedAddressItem fromFile(String fileName) throws IOException {
210.             return new ConnectedAddressItem(new ByteBufferKaitaiStream(fileName));
211.         }
212.         public ConnectedAddressItem() {
213.             this(null, null, null);
214.         }
215. 
216.         public ConnectedAddressItem(KaitaiStream _io) {
217.             this(_io, null, null);
218.         }
219. 
220.         public ConnectedAddressItem(KaitaiStream _io, EnipSegment.CommandData _parent) {
221.             this(_io, _parent, null);
222.         }
223. 
224.         public ConnectedAddressItem(KaitaiStream _io, EnipSegment.CommandData _parent, EnipSegment _root) {
225.             super(_io);
226.             this._parent = _parent;
227.             this._root = _root;
228.         }
229.         public void _read() {
230.             this.typeId = this._io.readU2le();
231.             this.addressItemLen = this._io.readU2le();
232.             this.connectionId = this._io.readU4le();
233.         }
234. 
235.         public void _write() {
236.             this._io.writeU2le(this.typeId);
237.             this._io.writeU2le(this.addressItemLen);
238.             this._io.writeU4le(this.connectionId);
239.         }
240. 
241.         public void _check() {
242.         }
243.         private int typeId;
244.         private int addressItemLen;
245.         private long connectionId;
246.         private EnipSegment _root;
247.         private EnipSegment.CommandData _parent;
248.         public int typeId() { return typeId; }
249.         public void setTypeId(int _v) { typeId = _v; }
250.         public int addressItemLen() { return addressItemLen; }
251.         public void setAddressItemLen(int _v) { addressItemLen = _v; }
252.         public long connectionId() { return connectionId; }
253.         public void setConnectionId(long _v) { connectionId = _v; }
254.         public EnipSegment _root() { return _root; }
255.         public void set_root(EnipSegment _v) { _root = _v; }
256.         public EnipSegment.CommandData _parent() { return _parent; }
257.         public void set_parent(EnipSegment.CommandData _v) { _parent = _v; }
258.     }
259.     public static class DataItem extends KaitaiStruct.ReadWrite {
260.         public static DataItem fromFile(String fileName) throws IOException {
261.             return new DataItem(new ByteBufferKaitaiStream(fileName));
262.         }
263.         public DataItem() {
264.             this(null, null, null);
265.         }
266. 
267.         public DataItem(KaitaiStream _io) {
268.             this(_io, null, null);
269.         }
270. 
271.         public DataItem(KaitaiStream _io, EnipSegment.CommandData _parent) {
272.             this(_io, _parent, null);
273.         }
274. 
275.         public DataItem(KaitaiStream _io, EnipSegment.CommandData _parent, EnipSegment _root) {
276.             super(_io);
277.             this._parent = _parent;
278.             this._root = _root;
279.         }
280.         public void _read() {
281.             this.typeId = this._io.readU2le();
282.             this.dataItemLen = this._io.readU2le();
283.             this.cipSeq = this._io.readU2le();
284.         }
285. 
286.         public void _write() {
287.             this._io.writeU2le(this.typeId);
288.             this._io.writeU2le(this.dataItemLen);
289.             this._io.writeU2le(this.cipSeq);
290.         }
291. 
292.         public void _check() {
293.         }
294.         private int typeId;
295.         private int dataItemLen;
296.         private int cipSeq;
297.         private EnipSegment _root;
298.         private EnipSegment.CommandData _parent;
299.         public int typeId() { return typeId; }
300.         public void setTypeId(int _v) { typeId = _v; }
301.         public int dataItemLen() { return dataItemLen; }
302.         public void setDataItemLen(int _v) { dataItemLen = _v; }
303.         public int cipSeq() { return cipSeq; }
304.         public void setCipSeq(int _v) { cipSeq = _v; }
305.         public EnipSegment _root() { return _root; }
306.         public void set_root(EnipSegment _v) { _root = _v; }
307.         public EnipSegment.CommandData _parent() { return _parent; }
308.         public void set_parent(EnipSegment.CommandData _v) { _parent = _v; }
309.     }
310.     private EnipHeader enipHeader;
311.     private CommandData commandData;
312.     private EnipSegment _root;
313.     private KaitaiStruct.ReadWrite _parent;
314.     private byte[] _raw_enipHeader;
315.     private byte[] _raw_commandData;
316.     public EnipHeader enipHeader() { return enipHeader; }
317.     public void setEnipHeader(EnipHeader _v) { enipHeader = _v; }
318.     public CommandData commandData() { return commandData; }
319.     public void setCommandData(CommandData _v) { commandData = _v; }
320.     public EnipSegment _root() { return _root; }
321.     public void set_root(EnipSegment _v) { _root = _v; }
322.     public KaitaiStruct.ReadWrite _parent() { return _parent; }
323.     public void set_parent(KaitaiStruct.ReadWrite _v) { _parent = _v; }
324.     public byte[] _raw_enipHeader() { return _raw_enipHeader; }
325.     public void set_raw_EnipHeader(byte[] _v) { _raw_enipHeader = _v; }
326.     public byte[] _raw_commandData() { return _raw_commandData; }
327.     public void set_raw_CommandData(byte[] _v) { _raw_commandData = _v; }
328. }

I tried calling _check() before _write() and changing input byte array, it won't work.

[generalmimon]

@silver-fang1 Thanks for the code. I can finally see the problem - after you _read() the data, the stream byte position stays at the end of the last known structure that has been parsed (which is the EOF in this case, because the input data is fully parsed till the end). If you change a value and try _write() to the same stream, it will start writing from the stream position. However, the stream position has got stuck at the EOF after _read(), so the very first write will fail (the ByteBufferKaitaiStream is a wrapper around Java ByteBuffer, which has a fixed size, it's not growable; writing beyond the end of stream is not allowed).

You need to seek() back to the beginning of the stream (_io().seek(0)):

 import io.kaitai.struct.ByteBufferKaitaiStream;

 public class main {
     public static void main(String[] args) {

         byte[] hexData = new byte[] {
                     0x70, 0x00, 0x22, 0x00, 0x00, 0x01, 0x02, 0x12,
                     0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
                     0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00,
                     0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x02, 0x00,
                 (byte)0xA1, 0x00, 0x04, 0x00, 0x01, 0x2D, (byte)0x94, 0x00,
                 (byte)0xB1, 0x00, 0x0E, 0x00, 0x03, 0x00, 0x03, 0x02,
                     0x20, 0x64, 0x24, 0x01, 0x02, 0x00, 0x01, 0x00,
                     0x02, 0x00
         };

         EnipSegment data = new EnipSegment(new ByteBufferKaitaiStream(hexData));

         data._read();
+        System.out.println("_io.pos after _read(): " + data._io().pos());
+        data._io().seek(0);
+        System.out.println("_io.pos before _write(): " + data._io().pos());
+
         data.enipHeader().setEnipCommand(0x55);
+        data.enipHeader()._check();
+
+        System.out.println("hexData before _write(): " + byteArrayToHex(hexData));

         try {
             data._check();
             data._write();
         }catch (Exception e){
             System.out.println(e);
         }

-        System.out.println(data.enipHeader().enipCommand());
+        System.out.println("hexData after  _write(): " + byteArrayToHex(hexData));
+        System.out.println("_io.pos after  _write(): " + data._io().pos());
+
+        System.out.println("enipCommand: " + data.enipHeader().enipCommand());
+    }
+
+    protected static String byteArrayToHex(byte[] arr) {
+        StringBuilder sb = new StringBuilder("[");
+        for (int i = 0; i < arr.length; i++) {
+            if (i > 0)
+                sb.append(' ');
+            sb.append(String.format("%02x", arr[i]));
+        }
+        sb.append(']');
+        return sb.toString();
     }
 }

BTW, the byteArrayToHex method is copied from the runtime library.

Output:

$ javac -encoding UTF-8 -d bin $(ls **/*.java)
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding="UTF-8"

$ java -cp bin main
Picked up JAVA_TOOL_OPTIONS: -Dfile.encoding="UTF-8"
_io.pos after _read(): 58
_io.pos before _write(): 0
hexData before _write(): [70 00 22 00 00 01 02 12 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 02 00 a1 00 04 00 01 2d 94 00 b1 00 0e 00 03 00 03 02 20 64 24 01 02 00 01 00 02 00]
hexData after  _write(): [55 00 22 00 00 01 02 12 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 01 00 02 00 a1 00 04 00 01 2d 94 00 b1 00 0e 00 03 00 03 02 20 64 24 01 02 00 01 00 02 00]
_io.pos after  _write(): 58
enipCommand: 85

[generalmimon]

@silver-fang1 Also notice that I added a _check() call after you set the enipCommand:

         data.enipHeader().setEnipCommand(0x55);
+        data.enipHeader()._check();

Calling _check() isn't absolutely necessary for writing, but it is strongly recommended, because it checks "that stuff that will be written will be read back properly" (see #27 (comment)). If it detects a problem with values that have been set by set*() methods, it triggers a ConsistencyError. This way, you get informed what value is wrong and would cause the serialized structure to be corrupt (so parsing it back would fail or get incorrect values).

Therefore, you should call _check() on each structure after you set all properties of it that you wanted to set (if any). Note that unlike _read() and _write(), _check() does not propagate to substructures. Which means that it is not enough to call it on the root object as you do now:

        try {
            data._check();
            data._write();
        }catch (Exception e){

As you can see in the generated code, data._check() actually doesn't do anything, because the seq of the root type EnipSegment does not directly contain fields that would require checking:

/**
 * Rockwell controller use ENIP communicate with RS5000 software
 */
public class EnipSegment extends KaitaiStruct.ReadWrite {
    // ...

    public void _write() {
        KaitaiStream _io__raw_enipHeader = new ByteBufferKaitaiStream(24);
        this.enipHeader._write(_io__raw_enipHeader);
        this._io.writeStream(_io__raw_enipHeader);
        this.commandData._write(this._io);
    }

    public void _check() { // NB: no checks
    }
    public static class EnipHeader extends KaitaiStruct.ReadWrite {
        // ...

... but the subtype EnipHeader has a few:

    public static class EnipHeader extends KaitaiStruct.ReadWrite {
        // ...

        public void _read() {
            this.enipCommand = this._io.readU2le();
            // ...
            this.senderContext = this._io.readBytes(8);
            this.enipOptions = this._io.readBytes(4);
        }

        public void _write() {
            this._io.writeU2le(this.enipCommand);
            // ...
            this._io.writeBytes(this.senderContext);
            this._io.writeBytes(this.enipOptions);
        }

        public void _check() {
            if (senderContext().length != 8)
                throw new ConsistencyError("sender_context", senderContext().length, 8);
            if (enipOptions().length != 4)
                throw new ConsistencyError("enip_options", enipOptions().length, 4);
        }

Calling the data._check() on the root object data, if you don't directly set any of its properties, is not necessary. It makes sense to call it just on structures that you modify. It's pointless to check the unchanged ones, because you just know that their values are OK and will read back properly, because you have just read them! I mean, you can do it if you want; it won't hurt anything, but it won't help either.

However, when you changed a field of enipHeader(), you should call _check() on it:

         data.enipHeader().setEnipCommand(0x55);
+        data.enipHeader()._check();

Although there are not necessarily any checks specifically for enipCommand yet, it is definitely a good habit to call _check() if you changed anything in the structure. Hopefully this advice helps you. I'm glad that you're bold enough to try serialization!

---

An answer to your previous question from #881 (comment):

By the way, if I want to add serialization support for python, which files need to be modified?
kaitai_struct_python_runtime and PythonCompiler in the serialization branch？

Yes, exactly. As I mentioned in #881 (comment), for Python there is already a WIP pull request with serialization routines:

The compiler doesn't have writing support for other languages than Java, but there are serialization routines available for some runtime libraries: (...) and Python (kaitai-io/kaitai_struct_python_runtime#48 - this probably still has some flaws pointed out in the review, though).

So I suggest forking from there for a faster start and making it follow the existing Java runtime library code (KaitaiStream, ByteBufferKaitaiStream) as closely as possible - but obviously you don't have to implement methods that you don't need (i.e. that are not used by EnipSegment.java or other formats that you need to serialize). Then you need to make PythonCompiler to generate the write_* method calls. Just try to run kaitai-struct-compiler --ksc-exceptions --read-write -t python *.ksy and see what errors you get. First, it will most likely complain that PythonCompiler does not have some traits that a language supporting serialization must have. You'll need to add some of these to PythonCompiler too (JavaCompiler.scala:12-25:

class JavaCompiler(val typeProvider: ClassTypeProvider, config: RuntimeConfig)
  extends LanguageCompiler(typeProvider, config)
    with SingleOutputFile
    with UpperCamelCaseClasses
    with ObjectOrientedLanguage
    with EveryReadIsExpression
    with EveryWriteIsExpression
    with GenericChecks
    with UniversalFooter
    with UniversalDoc
    with AllocateIOLocalVar
    with FixedContentsUsingArrayByteLiteral
    with SwitchIfOps
    with NoNeedForFullClassPath {

Certainly at least EveryWriteIsExpression and GenericChecks will be needed, not sure about others. Then try to invoke kaitai-struct-compiler --ksc-exceptions --read-write -t python *.ksy again. It will likely throw some NotImplementedError meaning that some method needed for writing is not implemented by PythonCompiler. Find that method up in JavaCompiler, copy it to PythonCompiler and edit it so that it outputs valid Python code. And compile again... 🙂