pgm-call: result-xml structure corrupt due to randomly added characters (urgent)

[nedi-dev]

• Node.js version: 12.14.0
• itoolkit version: 1.0.0
• IBM i version: 7.3
• XMLSERVICE version: XML Toolkit 2.0.2-dev

Describe the bug
The node-script requests IBM i DB-Data via a call to rpg-pgm.
As working with daily-data for one month the call to the program is added to a toolkit conn-object for 30/31 times.
Running the con results in a successful execution but gives back a corrupted xml-structure (e. g. ... type=a'4p2' ..., ... <d ...).
We are using different person-ids in the request. With some PER# the error is reproducible others work fine.
Maybe there is a stringbuffer involved?!?

! Running the script multiple times with identic PERS# results always in the same error at the same position in xml-string.

To Reproduce
see above

Expected behavior
Please locate the bug and provide a corrected version of itoolkit or xmlservice

Files
If needed, we can provide our code snippet and some xml-data.

[nedi-dev]

Examples for Errors:

<data type='4p2'>0<,00</data>
</parm>

<parm>
<data type='20A'></data>
<</parm>

<parm>
<data type='4p0'>1279</daita>
</parm>

<parm>
<data type='8p0'>17032020</d<ata>
</parm>

<parm>
<data type='4p0'>1431</data>
</rparm>

<parm>
<data type='4p2'a>15,04</data>
</parm>

[nedi-dev]

code for the function
Function.ts.txt

[nedi-dev]

Meantime tested with transporttype=odbc > no errors!
Maybe there is some issue in transporttype = idb ?!!

[abmusse]

Hello @nedi-dev 👋

Interesting that when using odbc transport you do not run into the errors. Does the error occur every time using idb transport or randomly?

I noticed from your code snippet that you have conn.debug(true) enabled. Does the xml input look malformed on the way in as well?

It would nice to have a simplified example of the RPG program being called (without confidential info) so that the issue can be readily reproduced.

[nedi-dev]

Hi @abmusse
The error occurs always when using 'idb' transport. Even the misspelled tag is the same for the same input parameters.
The input-xml is fine in all cases.
Find the abbreviated rpg-source attached.
ZEI109_short.rpglesrc.txt

[abmusse]

What version of idb-connector do you have installed?

$ npm ls idb-connector

I'm thinking this could be an issue with dbstmt.fetchAll.

Can you add the following to line 92-93 in lib/transports/idbTransport.js

console.log(results);
console.log('results length: ', results.length);

And check if the xml in the results array is already corrupted before its appended to xmlOutput variable.

[nedi-dev]

$ npm list idb-connector
trb-pgm-ZEI109@1.0.0 /home/NEUMANN/nodejs/trb-pgm-ZEI109
`-- itoolkit@1.0.0
  +-- idb-connector@1.2.9
  `-- idb-pconnector@1.0.8
    `-- idb-connector@1.2.9  deduped

[nedi-dev]

Thinking you are on the right track:
XML Error:
<success><![CDDATA[+++ success LGIZEIT ZEI109 ]]></success>
Duplicated "D" at pos 15!

Output from idbTransport:

      '<parm>\n' +
      "<data type='50A'>Keine Daten vorhanden !</data>\n" +
      '</parm>\n' +
      '<success><![CD'
  },
  {
    OUT151: 'DATA[+++ success LGIZEIT ZEI109 ]]></success>\n' +
      '</pgm>\n' +
      "<pgm name='ZEI109' lib='LGIZEIT' error='fast'>\n" +
      '<parm>\n' +
      "<data type='2p0'>1</data>\n" 

Same here:

<parm>
<data type='20A'></dataD>
</parm>

Capital "D" is wrong!

idbTransport:

      '<parm>\n' +
      "<data type='20A'></dataD"
  },
  {
    OUT151: '>\n' +
      '</parm>\n' +

other findings:

    '<parm>\n' +
     "<data type='8p0'>17012020D"
 },
 {
   OUT151: '</data>\n' +
     '</parm>\n' +

      '<parm>\n' +
      "<data type='4p2'>0,00D"
  },
  {
    OUT151: '</data>\n' +
      '</parm>\n' +

      '<parm>\n' +
      "<data type='4p0'>1279</date"
  },
  {
    OUT151: 'a>\n' +
      '</parm>\n' +

[ThePrez]

@abmusse , should we transfer this one over to idb-connector?

[abmusse]

From your output looks like the data retrieved using idb-connector is already corrupted before we iterate over results array and append to the xmlOutput. So seems to be an issue with idb transport since you tried odbc and did not receive an error.

@ThePrez That would make sense given the issue seems to be caused by idb-connector.

[abmusse]

@dmabupt

Any thoughts on what may be causing this issue?

[dmabupt]

@dmabupt

Any thoughts on what may be causing this issue?

Seems to be an encoding issue. Need to reproduce it for debugging.

[nedi-dev]

This is a good point. The error occurs only for employees with names including "ä ö ü Ä Ö Ü" german special-characters.
Data for other persons seem to be fine.

[abmusse]

Interesting I wonder if this issue is related to #70.