Support different EDIFACT "standards"

[nerdoc]

I am currently working/trying very hard to support the EDI data I for myself need to use, which is the EDIFACT format used in Austria for medical systems. It's horrible. Standards? we don't need that, we create our own stuff.

But even in the english Wikipedia article about EDIFACT there is an example that looks like this:

 UNA:+.? '
 UNB+IATB:1+6XPPC:ZZ+LHPPC:ZZ+940101:0950+1'
...

With few knowledge you you might have seen that the UNB header has "IATB" in its SyntaxIdentifier field. Which is None of "UNO[A-F]" - like the UNECE published as standard codes for that field.

In Austria, we use "ANSI", "DINA", "IBMA", "UNIX" as values there, basically it is "ANSI". And no, this is not ANSI/X12, it is EDIFACT. But non-standard rules. Even the UNB header has other fields: SENDER, RECEIVER, DATE, TIME are all simple data elements, not a combined one for date/tie, like in the UNECE standard:

UNB+ANSI:1+ME123456+ME654321+230403+0840+1'

does the header look like.

So I have to find a way to support all these different formats.

I suggest to use a dict for Segment.plugins with the tag name, which makes searching for tags easier and faster (as dict keys are indexed/hashed).

Segment.plugins["UNH"] -> list[Segment]

This list has only the few Segment classes that are supported - which may be 10 sometimes, but not 1000+ if we put all classes into one flat list, as we have now.
Each Segment must define a version (which can then be compared to the needed version in SegmentFactoy.create_segment(). And it must provide the supported syntax, like ["ANSI", "UNOA", "UNOB", "UNOC",...], which is sumb too if we'd have to add that to each Segment class.

But in fact, each "standard" has other segments.

@lord-haffi Ideas?

I can not think about another, better way of supporting more versions.

[nerdoc]

@sabas offered help as well in #18 - as he already started some schema plumbing for EDIFACT versions as xml files.

This could also be a good start. I'm afraid that if pydifact tries to add too much effort into using classes, it might end in having 1000s classes around, which renders it completely unmaintainable, and, it may use much more memory than it would use now.

So it would be another (better?) idea, instead of using specific Segment and DataElement subclasses, to get data from json files and validate the edifact data against that. Shouldn't be to hard to implement.

https://github.com/php-edifact/edifact-mapping from @sabas could be a start - but this is LGPL licensed; pydifact is MIT

I personally like tha LGPL license more, but it might keep some companies from using it. OTOH, using it without modifications is perfectly fine.

Oh, and I regret not having done that in a branch - i used the master branch - sorry for that.

[nerdoc]

An additional problem is that I can't determine from an EDI message, without extreme parsing efforts, which EDIFACT directory it is using (D921, D90A, D02B, D17B, etc ?!)
And there is no official, easy reachable mapping, which version (1-4) is using which directory.

So, if I get a message, and have a perfekt library that supports every single standard/direcory - how to I determine which one to use when parsing the message?

Or is it always necessary that both sender and receiver agree (before!) on a certain standard, let's say "D02B"?
The UNB header does not contain this information, right?

[nerdoc]

Even more: If we use facade classes for all the DataElements or segments like UNBSegment or CSyntaxIdentifier (for the compound data element), it must transparently support and abstract away the differences between the versions, because the sender and receiver may be the same, and the dates may be easy to read/write if n6 or n8, but when it comes to all the small differences I don't know, we`d have to add parser methods and serializers for all the differences.

When I think at this, I get a bit stomach pain... This can't be a good way to solve this problem.

Specialized DataElement -> Specialized CompoundDataElement -> Specialized Segment -> and then Specialized Interchange. we would need Interchanges for all versions.

No... can't be that way...

ATM the Interchange class does this:

     def get_header_segment(self) -> Segment:
       return Segment(
            self.HEADER_TAG,
            [self.syntax_identifier[0], str(self.syntax_identifier[1])],
            self.sender,
            self.recipient,
            [f"{self.timestamp:%y%m%d}", f"{self.timestamp:%H%M}"],
            self.control_reference,
            *self.extra_header_elements,
        )

which is a bit of nonsense, as it hardcodes a certain version of UNB segment (v4).

It needs to

• determine the correct EDIFACT syntax/version
• use that version of UNBSegment
• don't hardcode the date/time fields, but delegate it to the UNBSegment's serializer.

And the __init__ method too:

    def __init__(
        self,
        sender: Element,
        recipient: Element,
        control_reference: Element,
        syntax_identifier: tuple[str, int] | CSyntaxIdentifier,
        ...

How do we tell mypy/the IDE which CSyntaxIdentifier is meant here? It depends on the EDIFACT version.

With some segments, this won't be a problem, but it could lead to major differences between versions. Abstraction is not easy here.

[sabas]

To answer some of your questions, the choice of the license was made to get contributions at the start, nowadays I do MIT as well (or WTFPL :D ), for the xml mappings it's totally made by me so I can dual license or switch if you'd prefer....
To detect the message type, look at the parser
https://github.com/php-edifact/edifact/blob/master/src/EDI/Parser.php

I get the fields while parsing the message to detect version and type...

In an EDI exchange usually you agree on the messages by looking at samples your partner is using...

[nerdoc]

I get the fields while parsing the message to detect version and type...

sure, for standards this is ok, but if some folks do not adhere to them...

In an EDI exchange usually you agree on the messages by looking at samples your partner is using...

OMG. I was afraid that this would be the case. So in other words: guess. Because If I look at the samples I have (see the tests/data directory: patient*.edi), I can see only non-standard UNB headers. I have a description of that, but it adheres to none of any Dxx[AB] direcory standards of UNECE.

But even if we use your xml (-> JSON) files as base for validation, this would not solve the issue that a pythonic approach is a bit tricky:

from pydifact.somewhere-dynamic import UNBSegment
e = UNBSegment(
    sender="ME123456", receiver="ME654321", datetime=now()
).serialize(syntax="ANSI", version=1)

must create (for "my" funny format I want to support)

UNB+UNOB:1+ME123456+ME654321+250504+2025+1'

and for another serialize(syntax="UNOB", version=4):

UNB+UNOB:1+ME123456+ME654321+20250504:2025+1'

Mind the difference in the datetime, + vs : and n6 vs n8.

I think that we'd have to use a "minimum" Segment as facade, that handles all the differences in serializers and readers.

So in fact, we'd produce a Meta-EDIFACT that handles all the versions. This is crazy. I'm not IBM with 50 people working on it ;-) I'm just one person doing that in spare time...

But OTOH, I don't think we'd have to handle ALL variants from 1996 until now. There are many outdated versions, and some are so similar, that the differences don't matter at all. The mandatory fields have not changed too much.

From an architect's POV, we wouldn't have to implement many UNBSegments - there is only one class that should be imported. Else we would need dynamic imports (using importlib) which renders type annotations really useless.

I think that the approach with fixed classes for segments is a good one, and the moving parts are the serializers and importers for reading and writing different versions. This could be done dynamically, e.g., by loading xml/json definition files...

[sabas]

For the generation part, I started creating classes which generalized the messages I exchanged, but after some contributions I wrote some new messages with a procedural style because it would have been a mess
https://github.com/php-edifact/edifact-generator
Behold the IFTMIN message which is a "instruction message" and used in different contexts, all segments would be too long to support, my code for now is like this

$oInterchange = (new \EDI\Generator\Interchange('ME', 'YOU'))
                ->setCharset('UNOC', 2);


$oBase = (new \EDI\Generator\Message('IFTMIN', 'D', '04A', 'UN'));

$bgm = (new \EDI\Generator\Segment\BeginningOfMessage())
    ->setDocument('99')
    ->setDocumentIdentification($oBase->getMessageID())
    ->setMessageFunctionCode('9');

$oBase->addSegment($bgm);

$dtm = (new \EDI\Generator\Segment\DateTimePeriod())
    ->setDateOrTimeOrPeriodFunctionCodeQualifier('137')
    ->setDateOrTimeOrPeriodText(date('YmdHi'))
    ->setDateOrTimeOrPeriodFormatCode('203');

$oBase->addSegment($dtm);

$oBase = $oBase->compose();
$aComposed = $oInterchange->addMessage($oBase)->getComposed();

echo nl2br((new \EDI\Encoder($aComposed, false))->get());

So yes, segments should be the simplest generalization....

The pydifact code I have starts like this

    unique_interchange_id = 'I'+str(uuid.uuid4().hex.upper())[:13]
    interchange = Interchange('SUPPORT', 'CARRIER', unique_interchange_id, ('UNOA', 2))

    unique_message_id = 'M'+str(uuid.uuid4().hex.upper())[:13]
    message = Message(unique_message_id, ('VERMAS', 'D', '16A', 'UN', 'SMDG10'))
    message.add_segment(Segment("BGM", "749", unique_message_id, "5"))
    dtm = datetime.strftime(datetime.now(), "%Y%m%d%H%M")
    message.add_segment(Segment("DTM", ["137", dtm, "203"]))

[nerdoc]

I think that @sabas' https://github.com/php-edifact/edifact-mapping is extremely interesting. I think the reason I did not mention or recognize it earlier is that I did not understand it in the first place. Now it's getting clearer.

This is the only way that makes sense on the long term...

The pydifact code I have starts like this

    unique_interchange_id = 'I'+str(uuid.uuid4().hex.upper())[:13]
    interchange = Interchange('SUPPORT', 'CARRIER', unique_interchange_id, ('UNOA', 2))

    unique_message_id = 'M'+str(uuid.uuid4().hex.upper())[:13]
    message = Message(unique_message_id, ('VERMAS', 'D', '16A', 'UN', 'SMDG10'))
    message.add_segment(Segment("BGM", "749", unique_message_id, "5"))
    dtm = datetime.strftime(datetime.now(), "%Y%m%d%H%M")
    message.add_segment(Segment("DTM", ["137", dtm, "203"]))

Ok, this is just the low level access, with Segment - which works reasonably well. IMHO. But to support different Segments, it will need something like your mapping.

I think that the Austrian mis-interpretation of the EDIFACT standard (making two DataElements out of one ComposedDataElement for the UNB header's date and time, which effectively breaks the whole parsing) is a good example of "we need to support very different formats".

And maybe this could be solved using your mapping.

You are doing that by reading the message (like pydifact) as war, and use a different analyzer class to do the validation. This is also an approach which could make sense - but I'd like to integrate that into the segment code - as we need to read AND write the segments in the right way.

But that's not easy - to be honest I can't really wrap my head around it. Because You need to "interpret" the xml. For validation, xml is cool, as you can parse the elements bit for bit, and look if the schema matches. I am doing that in my code (not fully committed yet) too.
But for flexible writing, you'll need code, not xml - which "knows" where the right values should be written.

e.g. in https://github.com/php-edifact/edifact-mapping/blob/master/src/Mapping/Service_V4/segments.xml :

  <segment id="UNB" name="interchangeHeader" desc="To identify an interchange.">
...
    <composite_data_element id="S004" required="true" name="dateAndTimeOfPreparation" desc="Date and time of preparation of the interchange.">
      <data_element id="0017" required="true" name="date" desc="Local date when an interchange or a group was prepared." type="n" length="8"/>
      <data_element id="0019" required="true" name="time" desc="Local time of day when an interchange or a group was prepared." type="n" length="4"/>
    </composite_data_element>
...
</segment>

when I want to write code like

s = UNBSegment(syntax="UNOB",version=4, sender="ME", receiver="YOU", timestamp=now())

How should that code know that it should produce

UNB+UNOB:4+ME+YOU+20250504:2025+1'

With the above xml mapping?

And with this:

  <segment id="UNB" name="interchangeHeader" desc="To identify an interchange.">
...
      <data_element id="0017" required="true" name="date" desc="Local date when an interchange or a group was prepared." type="n" length="6"/>
      <data_element id="0019" required="true" name="time" desc="Local time of day when an interchange or a group was prepared." type="n" length="4"/>
...
</segment>

It should write

UNB+UNOB:4+ME+YOU+250504+2025+1'

Even if the code manages to interpret the <composite_data_element> and <data_element> nods correctly and translates them into the EDIFACT tree, there must be some code that knows that length="6" means YYMMDD and length="8" means YYYYMMDD - this information is not there.

It could be added as another xml attribute like strftime="%Y%m%d" or strftime="%y%m%d" for the two formats, but even that needs to be interpreted.

So I think that writing EDIFACT is a bit more complicated than validating it.

[sabas]

the xml mappings are used only when reading the EDIFACT (in the main edifact repo, they're used by the interpreter class), to provide a structure and array keys
to write I don't follow the mappings, although for example the BGM template (https://github.com/php-edifact/edifact-generator/blob/master/src/Generator/Segment/BeginningOfMessage.php) I made it by providing chatgpt another existing segment implementation and the relevant xml from the mappings hahaha (yes LLM know EDIFACT)

[nerdoc]

OK, I see. But then you can't support more versions when writing, right?

Oh, and therefore I developed the EDIFACT-syntax-in-code concept, which has the advantage that classes can inherit from others, as many of the changes in EDIFACT versions are minimal, so the "old" classes could be reused.

It's just a memory problem maybe, and you have to write the code manually (except for AI helpers) because there is no way of creating it by script.

However, it should be done in "plugins" - xml or code. With xml, this is easy, as you simply create directories, and load them.
With code, I can use plugins that can be loaded by importing them...

Yes, I do the boilerplate code too with LLMs - I even came to the problem of the Austrian "standard" misinterpreting the EDIFACT standards by asking ChatGPT...

[nerdoc]

I e.g. use code like this for creating the date time DataElement:

class Date(DataElement):
    code = "0017"
    repr = "n6"
    title = "Date"
    description = "Date in the format YYMMDD"

class Time(DataElement):
    code = "0019"
    repr = "n4"
    title = "Time"
    description = "Time in the format HHMM"

class CDateAndTimeOfPreparation(CompositeDataElement):
    code: str = "S004"
    schema: CompositeDataElementSchema = [
        (Date, M, "n6"),
        (Time, M, "n4"),
    ]
    datetime_format: str = "%y%m%d%H%M"

    def __init__(self, *elements):
        super().__init__(*elements)
        self.datetime = datetime.strptime(
            self.elements[0] + self.elements[1], self.datetime_format
        )

For the version 4, I have to change the datetime_format and the n6->n8:

class Date(DataElement):
    code = "0017"
    repr = "n8"
    title = "Date"
    description = "Date in the format YYYYMMDD"

class CDateAndTimeOfPreparation(v1.CDateAndTimeOfPreparation):
    schema: CompositeDataElementSchema = [
        (Date, M, "n8"),
        (Time, M, "n4"),
    ]
    datetime_format: str = "%Y%m%d%H%M"

But to write that for EVERY directory from D90A until D18A - I'd grow old.

[nerdoc]

Maybe it is not necessary to determine the syntax completely without external input, only from the file itself. because people are creative. In Austria, we use files like this:

UNB+IBMA:1+LABSBG+ME123456+181207+1707+1'
H`Normalverbraucher`Otto`W`20.07.1921`L1812412`07.12.2019`07.12.2019
E`1`MYCOG```negativ```Mycoplasma pneumoniae IgG
E`1`MYCOM```negativ```Mycoplasma pneumoniae IgM
H`Tester`Diana`W`05.08.1946`L2523546`07.12.2012`07.12.2012
E`1`BSG1```10`mm`< 30`BSG 1. Stunde
E`1`BSG2```22`mm`< 53`BSG 2. Stunde
E`1`HBA1C```6.4`%`< 6.0`HBA1c
E`2````5.7-6.4: erhöhtes DM-Risiko laut ÖDG-Leitlinie 2012```
E`1`HBA1CM```39.8`mmol/mol`13.7 - 42.1`HBA1c IFCC
E`1`KR```0.64`mg/dl`< 1.10`Kreatinin
E`1`GFRE```103`ml/min/1.73m2`> 90`glom. Filtrationsrate CKD-EPI
E`1`TSH```2.77`µU/ml`0.50 - 4.20`TSH
H`HUBER`Fritzi`M`02.01.1962`L189275920`07.12.2018`07.12.2018
E`1`BSG1`*`+`21`mm`< 15`BSG 1. Stunde
E`1`BSG2`*`+`43`mm`< 40`BSG 2. Stunde
E`1`KR```0.93`mg/dl`< 1.10`Kreatinin
E`1`GFRE```107`ml/min/1.73m2`> 90`glom. Filtrationsrate CKD-EPI
E`1`TSH```1.58`µU/ml`0.50 - 4.20`TSH
UNZ+1+1'

Which is basically a dumb UNB/UNZ envelope, enclosing a format that has nothing to do with EDIFACT.
And worse. We get files that are actually PDFs (yes, the binary blob format) that are enclosed in an EDIFACT header, of course with the wrong date/time DateElements.

So I'm facing a happy time finding out how to read that stuff ;-)

But it would suffice if I tell the Parser it should use the D96A plugin for that file. It'll raise errors anyway if it does not work.

And I don't have to support ALL directories. EDIFACT v4 and the Frankenstein EDIFACT format of the Austrian Medical Association would be a start - at least for me. And if it's pluggable, anyone could add his/her support too.

[sabas]

WTF!? Why they do the envelope with the normal separators and then change it?
I have a function to change the separators if there's no UNA segment, but it applies also to the message envelope, so it doesn't split UNB..
And that isn't an UN/EDIFACT message, so perhaps you have some guideline for it?

$c = new EDI\Parser();
$c->load('wtf.txt');
$c->analyseUNA(":`.?*'");
$c->parse();

This is a custom flat-file medical data export, wrapped in EDIFACT-like envelope tags for transmission.

IBMA is a custom syntax identifier, probably short for IBM Austria, IBM Application, or similar.

Not a standard EDIFACT message like LABRES or MEDRPT.

You likely need custom parsing logic or documentation from the sending party (probably a lab using an IBM-based or legacy system).

If you're interfacing with this system, treat it as semi-structured medical lab data rather than full EDIFACT.

Thanks ChatGPT :|