Transfer syntax mismatch when sending a dataset to C-STORE SCP

[isolovey]

I have a question about the way pynetdicom's Association.send_c_store method picks a presentation context for a dataset it is about to transfer.

The method first obtains the Transfer Syntax from the dataset (file meta and the dataset itself), and then looks for a negotiated Presentation Context for that Transfer Syntax and SOP Class.

However, what if the negotiated Presentation Context for that SOP Class is for a different Transfer Syntax? For example, my dataset is encoded with Implicit Little Endian, but the destination Store SCP accepts only Explicit Little Endian? In this case, send_c_store will raise an exception (_get_valid_context will fail to find a match). If I instead simply don't read the Transfer Syntax from the dataset, it will succeed. The Presentation Context will be with the Explicit Little Endian TS, and the stream will be re-encoded to match it, further down in send_c_store.

I validated this by patching the method to not use the transfer syntax obtained from the dataset:

index b648fa282..ba5d32851 100644
--- a/pynetdicom/association.py
+++ b/pynetdicom/association.py
@@ -1931,7 +1931,7 @@ class Association(threading.Thread):
 
         # Get a Presentation Context to use for sending the message
         context = self._get_valid_context(
-            sop_class, tsyntax, "scu", allow_conversion=allow_conversion
+            sop_class, "", "scu", allow_conversion=allow_conversion
         )
         transfer_syntax = context.transfer_syntax[0]

I'm implementing a C-MOVE proxy which receives datasets from a DICOM server modifies them (for compatibility reasons) before sending off to a legacy device. The legacy device only uses the Explicit Little Endian transfer syntax. I can force the DICOM server to also use this transfer syntax, so that the datasets don't have to be re-encoded in my proxy. But I don't see why I should have to do this, and why the Transfer Syntax from the dataset is matched to the list of negotiated Presentation Contexts at all. I'm sure there are pitfalls to my "naive" patch. I'm probably missing something and would appreciate any input.

[scaramallion]

pydicom can convert between uncompressed transfer syntaxes like Implicit VR Little Endian, Explicit VR Little Endian and Deflated Explicit VR Little Endian without any issues, so when sending uncompressed datasets pynetdicom should automatically convert to match the accepted presentation context's transfer syntax:

from pydicom import examples
from pydicom.uid import ExplicitVRLittleEndian
from pynetdicom import AE, debug_logger
from pynetdicom.sop_class import RTDoseStorage

ds = examples.rt_dose
print(ds.file_meta.TransferSyntaxUID.name)  # Implicit VR Little Endian

debug_logger()

ae = AE()
ae.add_requested_context(RTDoseStorage, ExplicitVRLittleEndian)
assoc = ae.associate("localhost", 11112)  # To DCMTK's storescp
assoc.send_c_store(ds)

I: Requesting Association
D: Request Parameters:
D: ======================= OUTGOING A-ASSOCIATE-RQ PDU ========================
D: Our Implementation Class UID:      1.2.826.0.1.3680043.9.3811.3.0.1
D: Our Implementation Version Name:   PYNETDICOM_301
D: Application Context Name:    1.2.840.10008.3.1.1.1
D: Calling Application Name:    PYNETDICOM
D: Called Application Name:     ANY-SCP
D: Our Max PDU Receive Size:    16382
D: Presentation Context:
D:   Context ID:        1 (Proposed)
D:     Abstract Syntax: =RT Dose Storage
D:     Proposed SCP/SCU Role: Default
D:     Proposed Transfer Syntax:
D:       =Explicit VR Little Endian
D: Requested Extended Negotiation: None
D: Requested Common Extended Negotiation: None
D: Requested Asynchronous Operations Window Negotiation: None
D: Requested User Identity Negotiation: None
D: ========================== END A-ASSOCIATE-RQ PDU ==========================
D: Accept Parameters:
D: ======================= INCOMING A-ASSOCIATE-AC PDU ========================
D: Their Implementation Class UID:    1.2.276.0.7230010.3.0.3.6.4
D: Their Implementation Version Name: OFFIS_DCMTK_364
D: Application Context Name:    1.2.840.10008.3.1.1.1
D: Calling Application Name:    PYNETDICOM
D: Called Application Name:     ANY-SCP
D: Their Max PDU Receive Size:  16384
D: Presentation Contexts:
D:   Context ID:        1 (Accepted)
D:     Abstract Syntax: =RT Dose Storage
D:     Accepted SCP/SCU Role: Default
D:     Accepted Transfer Syntax: =Explicit VR Little Endian
D: Accepted Extended Negotiation: None
D: Accepted Asynchronous Operations Window Negotiation: None
D: User Identity Negotiation Response: None
D: ========================== END A-ASSOCIATE-AC PDU ==========================
I: Association Accepted
I: Sending Store Request: MsgID 1
D: ========================== OUTGOING DIMSE MESSAGE ==========================
D: Message Type                  : C-STORE RQ
D: Message ID                    : 1
D: Affected SOP Class UID        : RT Dose Storage
D: Affected SOP Instance UID     : 1.9.999.999.99.9.9999.9999.20030818153516
D: Data Set                      : Present
D: Priority                      : Low
D: ============================ END DIMSE MESSAGE =============================
D: pydicom.read_dataset() TransferSyntax="Little Endian Implicit"
I: Received Store Response (Status: 0x0000 - Success)
D: ========================== INCOMING DIMSE MESSAGE ==========================
D: Message Type                  : C-STORE RSP
D: Presentation Context ID       : 1
D: Message ID Being Responded To : 1
D: Affected SOP Class UID        : RT Dose Storage
D: Affected SOP Instance UID     : 1.9.999.999.99.9.9999.9999.20030818153516
D: Status                        : 0x0000 - Success
D: ============================ END DIMSE MESSAGE =============================

However, if _config.STORE_SEND_CHUNKED_DATASET is True then pynetdicom requires the dataset's Transfer Syntax UID be an exact match to the accepted transfer syntax, because the dataset isn't decoded at all (just sent in chunks) and so cannot be converted.

Without seeing your code, an example anonymised dataset or any logging output I don't know why the conversion would be failing. I wouldn't recommend your patch, though, as (generally speaking) you'll be sending datasets that may be encoded with an incorrect transfer syntax.

[isolovey]

Sorry for the confusion @scaramallion , I have misidentified the issue as being a mismatch between Explicit vs Implicit VR. In fact, the mismatch causing the problem is the endianness. I can see that in #484 , a check was added to enforce matching endianness because of pydicom's inability to convert it. However, if I remove that check

index b648fa282..9a4a42dd3 100644
--- a/pynetdicom/association.py
+++ b/pynetdicom/association.py
@@ -483,10 +483,6 @@ class Association(threading.Thread):
                 if tr_syntax.is_compressed or cx_syntax.is_compressed:
                     continue
 
-                # Filter out contexts where the endianness doesn't match
-                if tr_syntax.is_little_endian != cx_syntax.is_little_endian:
-                    continue
-
             # Match to convertible transfer syntaxes
             #   Allowable matches:
             #       explicit VR <-> implicit VR

The endianness conversion seems to happen without issue. Using your snippet with either ExplicitVRBigEndian and ExplicitVRBigEndian requested context, I've looked at the datasets received via DCMTK's storescp with a DICOM hex viewer and they both look as expected, the endianness matches the transfer syntax.

Perhaps pydicom started supporting endianness conversion and the check in #484 isn't needed anymore?

(I attached the two datasets as received by the store scp: received_from_pynetdicom.zip . )

[scaramallion]

No, your "big endian" dataset hasn't been converted (pydicom doesn't do so):

>>> [v for v in little.PixelData[:8]
[232, 14, 19, 0, 232, 14, 19, 0]
>>> [v for v in big.PixelData[:8]
[232, 14, 19, 0, 232, 14, 19, 0]

If it were converted the little endian Pixel Data (VR OW with Bits Allocated 32) values would be byte swapped to: [0, 19, 14, 232, 0, 19, 14, 232]

To do the conversion properly you need to identify which elements in your dataset use binary encodings. This will include (but may not be limited to):

• Pixel Data, which needs to be byte swapped based off the container size from Bits Allocated
• Elements with a multi-byte O* VR such as OW, OF, OD, etc.
• Elements with a VR of AT
• Possibly elements encoding multi-byte unicode text, I'm not sure if these are endianness specific or not
• Multi-byte VRs that encode floats and integers: FL, FD, SL, SS, SV, UL, US, UV
• Any binary encoded private elements that may be of interest, although honestly it's easier to just remove any private elements if you don't need them, or change the VR to UN.

These will have to be byte swapped to use the correct byte ordering, then once that is done you can change the dataset's file meta Transfer Syntax UID to Explicit VR Big Endian and transfer with pynetdicom.