Pull transfer with multipart requests does not work

[Moritz72]

Describe the bug

When using a multipart request to perform a pull transfer, the request arrives at the data source in a malformed state.

To Reproduce

1. Create an offer for some REST API (for example fastapi) that has an endpoint that expects some number of files as part of a multipart request (provider side).
2. Negotiate the offer per EDR negotiation
3. Perform a pull transfer using the EDR with a multipart request

Expected behavior

The request arrives at the REST API, is correctly processed and eventually a response with status code 200 or similar is returned.

Screenshots/Error Messages

In my case I have a fastapi REST API. When performing the above steps I get

{"errors":["Received code transferring HTTP data: 400 - ."]}

When directly calling the REST API instead of going through the Connector it works though.

So this works

curl -X PUT "<REST_API_URL>/<API_PATH>" \
  -F "file=@/path/to/file"

but this does not

curl -X PUT "<CONNECTOR_PUBLIC_API>/<API_PATH>" \
  -H "Authorization: <TOKEN>" \
  -F "file=@/path/to/file"

I looked into the tcpdump and saw that for the PULL the REST API receives the request without a carriage return after a boundary.

This is an excert of the tcpdump for the working one (direct)

0x01c0:  6f6e 7465 6e74 2d74 7970 653a 206d 756c  ontent-type:.mul
0x01d0:  7469 7061 7274 2f66 6f72 6d2d 6461 7461  tipart/form-data
0x01e0:  3b20 626f 756e 6461 7279 3d2d 2d2d 2d2d  ;.boundary=-----
0x01f0:  2d2d 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d  ----------------
0x0200:  2d2d 2d61 3639 3135 6336 6137 3266 6664  ---a6915c6a72ffd
0x0210:  3336 380d 0a0d 0a2d 2d2d 2d2d 2d2d 2d2d  368....---------
0x0220:  2d2d 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d  ----------------
0x0230:  2d61 3639 3135 6336 6137 3266 6664 3336  -a6915c6a72ffd36
0x0240:  380d 0a43 6f6e 7465 6e74 2d44 6973 706f  8..Content-Dispo
0x0250:  7369 7469 6f6e 3a20 666f 726d 2d64 6174  sition:.form-dat
0x0260:  613b 206e 616d 653d 2266 696c 6522 3b20  a;.name="file";.
0x0270:  6669 6c65 6e61 6d65 3d22 6375 7374 6f6d  filename="custom
0x0280:  6572 732d 3130 302e 6373 7622 0d0a 436f  ers-100.csv"..Co
0x0290:  6e74 656e 742d 5479 7065 3a20 6170 706c  ntent-Type:.appl
0x02a0:  6963 6174 696f 6e2f 6f63 7465 742d 7374  ication/octet-st
0x02b0:  7265 616d 0d0a 0d0a 496e 6465 782c 4375  ream....Index,Cu

and this for the one that does not work (pull transfer)

0x01f0:  6666 320d 0a63 6f6e 7465 6e74 2d74 7970  ff2..content-typ
0x0200:  653a 206d 756c 7469 7061 7274 2f66 6f72  e:.multipart/for
0x0210:  6d2d 6461 7461 3b62 6f75 6e64 6172 793d  m-data;boundary=
0x0220:  2d2d 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d  ----------------
0x0230:  2d2d 2d2d 2d2d 2d2d 3266 6363 3330 6432  --------2fcc30d2
0x0240:  6438 3530 6138 3137 0d0a 6163 6365 7074  d850a817..accept
0x0250:  2d65 6e63 6f64 696e 673a 2067 7a69 700d  -encoding:.gzip.
0x0260:  0a75 7365 722d 6167 656e 743a 206f 6b68  .user-agent:.okh
0x0270:  7474 702f 352e 312e 300d 0a0d 0a2d 2d2d  ttp/5.1.0....---
0x0280:  2d2d 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d 2d2d  ----------------
0x0290:  2d2d 2d2d 2d2d 2d32 6663 6333 3064 3264  -------2fcc30d2d
0x02a0:  3835 3061 3831 370a 436f 6e74 656e 742d  850a817.Content-
0x02b0:  4469 7370 6f73 6974 696f 6e3a 2066 6f72  Disposition:.for
0x02c0:  6d2d 6461 7461 3b20 6e61 6d65 3d22 6669  m-data;.name="fi
0x02d0:  6c65 223b 2066 696c 656e 616d 653d 2263  le";.filename="c
0x02e0:  7573 746f 6d65 7273 2d31 3030 2e63 7376  ustomers-100.csv
0x02f0:  220a 436f 6e74 656e 742d 5479 7065 3a20  ".Content-Type:.
0x0300:  6170 706c 6963 6174 696f 6e2f 6f63 7465  application/octe
0x0310:  742d 7374 7265 616d 0a0a 496e 6465 782c  t-stream..Index,

Note that there is no carriage return (0d) after the boundary before Content-Dispostion in the second one.

Context Information

• Tractus-X EDC 0.11.0 (running in Kubernetes)
• REST API with python (fastapi)

[Moritz72]

The issue seems to be caused by the ContainerRequestContextApi. Specifically the following method

@Override
public String body() {
    try (BufferedReader br = new BufferedReader(new InputStreamReader(context.getEntityStream()))) {
        return br.lines().collect(Collectors.joining("\n"));
    } catch (IOException e) {
        throw new EdcException("Failed to read request body: " + e.getMessage());
    }
}

[Moritz72]

I "fixed" it now by replacing the above mentioned code with

@Override
public String body() {
    try {
        return new String(context.getEntityStream().readAllBytes(), StandardCharsets.UTF_8);
    } catch (IOException e) {
        throw new EdcException("Failed to read request body: " + e.getMessage());
    }
}

I'm not sure about the full implications of this change as I'm not overly familiar with java, but it seems to work fine for what I tried. I would be willing to submit a pull request if this is a desired change.

[lgblaumeiser]

I'll check it! Thanks!

[lgblaumeiser]

Hello @Moritz72, sorry, but finally, I found some time to check your change. It definitely improves the situation, as the previous implementation slices the input on line endings and add the line endings again, but sometimes not completely. That is definitely a bug. Your proposed solution does get around that by taking the original data and encoding it into a string as is.

So I suggest to go ahead and do a contribution PR with your proposed change. One small change I propose is, to leave away the charset parameter when creating the String. The point is, that the previous implementation also used the default charset, so leaving it our would be the minimal change.

I might add, that I am not sure, whether this part of the proxy is properly thought through, as it assumes, that the body is a stream of characters, which might not always be the case. By translating the bytes to a string, there might be unwanted side effects for binary data streams. But this is part of the original implementation which comes from upstream, as the overridden body method is expected to return a String object. So, I would go with the minimal implementation, which is as proposed by you but without the charset to prevent unwanted side effects.

[lgblaumeiser]

I took the freedom to assign this to you, if you want us to do the change, please come back, but I would in general appreciate contributions from others than the committers 😃