AWS Cloudfront returning 502 for all requests handled by Quarkus Lambda

[fage88]

I am using AWS Cloudfront with an Application Load balancer as the origin.
On the application Load balancer I have a rule to send some traffic to a AWS Lambda function.
I had originally implemented this is vanilla Java and it was all working, but I wanted to look at moving it to quarkus so I could eventually move it to a GraalVm native image.
My handler looks like this

@Override
    public ApplicationLoadBalancerResponseEvent handleRequest(ApplicationLoadBalancerRequestEvent input, Context context) {

        Log.infof(INFO_HANDLING_REQUEST_FORMAT, input.getPath());

        var response = new ApplicationLoadBalancerResponseEvent();
        if (StringUtils.contains(input.getPath(), WATERMARK_PDF_URI_PARTIAL))  {
            try {
                HttpResponse<byte[]> downloadResponse = pdfRetrieverService.getDownloadRequest(input);

                response.setStatusCode(downloadResponse.statusCode());
                if (downloadResponse.statusCode() == 200) {
                    response.setBody(Base64.getEncoder().encodeToString(
                            watermark(downloadResponse, watermarkTextService.getWatermarkingLines(input))));
                    var preservedHeaders = new HashMap<String, String>(PRESERVED_HEADERS.size());
                    for (var preserveHeader : PRESERVED_HEADERS) {
                        for (var header : downloadResponse.headers().map().entrySet()) {
                            if (StringUtils.equalsIgnoreCase(preserveHeader, header.getKey())) {
                                preservedHeaders.put(header.getKey(), header.getValue().get(0));
                            }
                        }
                    }
                    response.setHeaders(preservedHeaders);
                    response.setIsBase64Encoded(true);
                } else {
                    response.setBody(new String(downloadResponse.body()));
                }
            } catch (Exception e) {
                response.setStatusCode(500);
                response.setBody(String.format(UNEXPECTED_ERROR_FORMAT, e));
                response.setHeaders(Map.of(CONTENT_TYPE_HEADER, TEXT_HTML_HEADER_VALUE));
            }
        } else {
            response.setStatusCode(400);
            response.setBody(BAD_REQUEST_STATUS_DESCRIPTION);
            response.setHeaders(Map.of(CONTENT_TYPE_HEADER, TEXT_HTML_HEADER_VALUE));
        }
        if (response.getStatusCode() == 200) {
            Log.infof(INFO_PDF_WATERMARKING_COMPLETE_FORMAT, input.getPath());
        } else {
            Log.warnf(WARNING_PDF_WATERMARKING_FAILED_FORMAT, input.getPath(), response.getStatusCode(), response.getBody());
        }
        return response;
    }

It doesn't seem to matter what response I send back cloudfront always returns a 502.
The test function in aws console returns the responses I expect.

Anyone have any idea of what could be wrong here? It feels like somehow the result is not leaving the lambda function correctly.

thank you,
-Jeff

[quarkus-bot[bot]]

/cc @matejvasek, @patriot1burke

[patriot1burke]

We wrap your lambda with a stream handler so that we can control lifecycle of the lambda for things like CDI/Spring injections. The stream handler will unmarshal/marshal the lambda input/output using Jackson. My guess is that there is an error in the marshalling of the load balancer event. I would need log output of your lambda.

[fage88]

I turned on DEBUG logging but its not too interesting looking

START RequestId: 884d2cf3-573e-40d6-8ce8-ce3bb3eaf231 Version: $LATEST
--
__  ____  __  _____   ___  __ ____  ______
--/ __ \/ / / / _ \| / _ \/ //_/ / / / __/
-/ /_/ / /_/ / __ \|/ , _/ ,< / /_/ /\ \
--\___\_\____/_/ \|_/_/\|_/_/\|_\|\____/___/
2022-06-21 15:24:53,177 DEBUG [org.jbo.logging] (main) Logging Provider: org.jboss.logging.JBossLogManagerProvider
2022-06-21 15:24:53,529 DEBUG [io.qua.arc.impl] (main) ArC DI container initialized [beans=15, observers=0]
2022-06-21 15:24:54,085 DEBUG [org.jbo.threads] (main) JBoss Threads version 3.4.2.Final
2022-06-21 15:24:54,275 INFO  [io.quarkus] (main) prime-pdf-watermark-lambda 1.0-SNAPSHOT on JVM (powered by Quarkus 2.9.2.Final) started in 1.701s.
2022-06-21 15:24:54,275 INFO  [io.quarkus] (main) Profile prime-dev activated.
2022-06-21 15:24:54,276 INFO  [io.quarkus] (main) Installed features: [amazon-lambda, cdi]
2022-06-21 15:24:56,470 INFO  [com.pub.pri.pdf.han.WatermarkingHandler] (main) Handling PDF request /rest/content/v1/watermark-pdf/astmh/astmh/draft/journals/tpmd/102/3/article-p629.pdf
2022-06-21 15:24:56,530 INFO  [com.pub.pri.pdf.ser.PdfRetrieverService] (main) Download request http://prime-content.dev.prime.pubfactory.net/rest/content/v1/download/astmh/astmh/draft/journals/tpmd/102/3/article-p629.pdf
2022-06-21 15:24:59,264 FINE  [org.apa.fon.ttf.GlyphSubstitutionTable] (main) FeatureRecord array not alphabetically sorted by FeatureTag: fina < medi
2022-06-21 15:24:59,285 FINE  [org.apa.fon.ttf.GlyphSubstitutionTable] (main) Type 4 GSUB lookup table is not supported and will be ignored
2022-06-21 15:24:59,285 FINE  [org.apa.fon.ttf.GlyphSubstitutionTable] (main) Type 4 GSUB lookup table is not supported and will be ignored
2022-06-21 15:24:59,945 INFO  [com.pub.pri.pdf.han.WatermarkingHandler] (main) PDF watermarking complete for /rest/content/v1/watermark-pdf/astmh/astmh/draft/journals/tpmd/102/3/article-p629.pdf
END RequestId: 884d2cf3-573e-40d6-8ce8-ce3bb3eaf231
REPORT RequestId: 884d2cf3-573e-40d6-8ce8-ce3bb3eaf231	Duration: 5758.05 ms	Billed Duration: 5759 ms	Memory Size: 514 MB	Max Memory Used: 155 MB	Init Duration: 2051.84 ms

Is there another logging setting I should use?

[patriot1burke]

My guess is that the ALB doesn't like the json response it is given.

[patriot1burke]

Could compare the output of the vanilla Java vs. the Quarkus one.

[fage88]

here is the vanilla java aws console test output

{
  "statusCode": 200,
  "isBase64Encoded": true,
  "headers": {
    "cache-control": "no-cache, no-transform, no-store, max-age=0, private",
    "etag": "\"/journals/tpmd/102/3/article-p629.pdf\""
  },
  "body": "base 64 encoded stream"
}

vrs quarkus

{
  "statusCode": 200,
  "statusDescription": null,
  "isBase64Encoded": true,
  "headers": {
    "cache-control": "no-cache, no-transform, no-store, max-age=0, private",
    "content-type": "application/pdf",
    "etag": "\"/journals/tpmd/102/3/article-p629.pdf\""
  },
  "multiValueHeaders": null,
  "body": "base64 encoded stream"
}

It looks like Quarkus has an empty status description, I wonder if it is simple as that, I'll try populating that field

[fage88]

YES! that did it,
Serializing the status Description with Null was 100% the issue.

Thank you for your help @patriot1burke

[patriot1burke]

Ok, that's a simple fix on our side so you don't have to manually do it.

[fage88]

great, for now I am adding the dependency

<dependency>
                <groupId>commons-httpclient</groupId>
                <artifactId>commons-httpclient</artifactId>
                <version>${commons-httpclient.version}</version>
</dependency>

and this workaround

 response.setStatusCode(downloadResponse.statusCode());
 response.setStatusDescription(HttpStatus.getStatusText(downloadResponse.statusCode()));

But if you guys are gonna fix it I'd be great. Then I can remove that dependency which I am not using anywhere else