DPP: first inference yields incorrect results on z/OS

[tinhto-000]

Running DPP on z/OS, first inference always yields incorrect AIU output while always fine second and onward. For example running program-id 22:

First:

[203, 203, 1318, 25077, 4693, 101, 884, 2124, 322, 45055, 544, 312, 309, 607, 48643, 345, 633, 32, 11228, 7629, 322, 1869, 482, 973, 990, 461, 15682, 32, 7141, 2953, 884, 646]
[ 0/ 4]: AIU tokens:
[44322, 24349, 1170, 497, 638, 30, 309, 17548, 31, 363, 31, 83, 31, 3368, 30, 312, 286, 45, 619, 312, 5808, 30, 5808, 31, 452, 31, 3368, 30, 312, 286, 45, 6935]
[ 0/ 4]: CPU output:


The ligaments are like the stays on a sailboat mast. They hold the spine upright and stable. But they are not
[ 0/ 4]: AIU output:
aset flyby..., soda-and-a-day, ade; as a day, day-to-day, ade; days```

Second and on:

[203, 203, 1318, 25077, 4693, 101, 884, 2124, 322, 45055, 544, 312, 309, 607, 48643, 345, 633, 32, 11228, 7629, 322, 1869, 482, 973, 990, 461, 15682, 32, 7141, 2953, 884, 646]
[ 0/ 4]: AIU tokens:
[203, 203, 1318, 25077, 4693, 101, 884, 322, 2193, 5430, 30, 461, 2953, 884, 5029, 8913, 32, 7141, 2017, 884, 2329, 28543, 2612, 30, 1510, 884, 322, 5430, 688, 5008, 322, 21931]
[ 0/ 4]: CPU output:


The ligaments are like the stays on a sailboat mast. They hold the spine upright and stable. But they are not
[ 0/ 4]: AIU output:


The ligaments are the support lines, and they are very important. But there are also tendons, which are the lines that connect the mus```

It affects all program-ids regardless of sequence or resultant prompt batch-size:

22 -> 21 -> 18 (all bs 1) : 22 bad
21 -> 18 : 21 bad
22 -> 19 (bs 2) -> 15 (bs 2) : 22 bad
19 -> 15 : 19 bad

I have spoken to @JRosenkranz that it might have to do with device initialization being different on z/OS that an extra inference call (extract_validation_information) is needed after warmup.

[tinhto-000]

Also this only affects multi-AIU. No issue if nproc-per-node=1.