acme/http-01: no challenge registered

[bmos1]

Bug Overview

We configure the nginx-acme module to replace the working certbot solution. The User-Agent EJBCA ACME Challenge Validator/1.0 requests a /.well-known path. The debug error message states acme/http-01: no challenge registered for ... The nginx-acme module responds with 404. Finally we see urn:ietf:params:acme:error:incorrectResponse: Response received didn't match the challenge's requirements in the logs and no certificate is issued.

Expected Behavior

The nginx-acme module provisions the key authorization as a resource an endpoint /.well-known/acme-challenge/TOKEN on the HTTP server for the domain in question.

Steps to Reproduce the Bug

nginx config

pid /var/tmp/nginx/nginx.pid;
error_log /var/tmp/nginx/error.log debug;

load_module modules/ngx_http_acme_module-debug.so;

http {
    # Config access.log for all http servers where nginx has proper write access
    access_log /var/tmp/nginx/access.log;
    
    # Hide Nginx version
    server_tokens off;

    ...

    # Internal DNS resolver
    resolver 1.1.1.1:53;

    # ACME (ngx_http_acme_module is required) 
    acme_issuer pki {
        uri         https://pki.example.com/ejbca/acme/signserver/directory;
        account_key ecdsa:256;
        contact     pki@example.com;
        state_path  /var/cache/nginx/acme;
        accept_terms_of_service;
    }

    acme_shared_zone zone=ngx_acme_shared:1M;

    server { 
        # listener on port 80 is required to process ACME HTTP-01 challenges 
        listen 80; 

        location / { 
           return 404; 
        }
    }

    server {
        server_name signserver.example.com;

        listen 443 ssl;
        http2 on;

        # ACME (ngx_http_acme_module is required) 
        acme_certificate     pki signserver.example.com key=ecdsa:384;
        ssl_certificate       $acme_certificate; 
        ssl_certificate_key   $acme_certificate_key;
        ssl_certificate_cache max=2;
        
        location / {
        proxy_pass http://127.0.0.1:8082/;
        ...
        }
    }
}

events {

}

Restarting nginx-debug results in requests from the EJBCA ACME Challenge Validator, where all requests result in 404.
The debug error log shows: acme/http-01: no challenge registered for urHN7xquTdZtjrO0R0Q8Iw

sending request to https://pki.example.com/ejbca/acme/signserver/acct/0T8HbZjFyuO5ATyceI6bwg/chall/OdYQJ1C18t9rbxhRw6-HJg

http request line: "GET /.well-known/acme-challenge/urHN7xquTdZtjrO0R0Q8Iw HTTP/1.1"

http process request header line
http header: "Connection: close"
http header: "Host: pki-uctserver01.commend.intern"
http header: "User-Agent: EJBCA ACME Challenge Validator/1.0"
http header: "Accept-Encoding: gzip,deflate"
http header done

acme/http-01: no challenge registered for urHN7xquTdZtjrO0R0Q8Iw
generic phase: 1
rewrite phase: 2
test location: "/"
using configuration "/"
http cl:-1 max:1048576
rewrite phase: 4
http finalize request: 404, "/.well-known/acme-challenge/urHN7xquTdZtjrO0R0Q8Iw?" a:1, c:1
http special response: 404, "/.well-known/acme-challenge/urHN7xquTdZtjrO0R0Q8Iw?"
http set discard body
HTTP/1.1 404 Not Found
Server: nginx
Date: Fri, 22 Aug 2025 07:25:30 GMT
Content-Type: text/html
Content-Length: 146
Connection: close

write new buf t:1 f:0 000056623BBC33D0, pos 000056623BBC33D0, size: 143 file: 0, size: 0
http write filter: l:0 f:0 s:143
http output filter "/.well-known/acme-challenge/urHN7xquTdZtjrO0R0Q8Iw?"
http copy filter: "/.well-known/acme-challenge/urHN7xquTdZtjrO0R0Q8Iw?"
http postpone filter "/.well-known/acme-challenge/urHN7xquTdZtjrO0R0Q8Iw?" 000056623BD50AB0

http write filter: l:1 f:0 s:289
http write filter limit 2097152

http write filter 0000000000000000
http copy filter: 0 "/.well-known/acme-challenge/urHN7xquTdZtjrO0R0Q8Iw?"
http finalize request: 0, "/.well-known/acme-challenge/urHN7xquTdZtjrO0R0Q8Iw?" a:1, c:1
http request count:1 blk:0
http close request
http log handler

close http connection: 3
reusable connection: 0

worker cycle
epoll timer: 59984

SSL_read: 455
SSL_read: 0
SSL_get_error: 6
peer shutdown SSL cleanly
shmtx lock
slab alloc: 105 slot: 4
slab alloc: 00007311171AD080
shmtx unlock
acme certificate "pki/signserver.example.com-f28176657b37216b" request failed: urn:ietf:params:acme:error:incorrectResponse: Response received didn't match the challenge's requirements

Environment Details

Target deployment platform: VPS
Target OS: Ubuntu 24.04
Version of this project or specific commit: 1.29.1+0.1.1-1 (from nginx mainline Ubuntu repo)
Version of any relevant project languages: None

Additional Context

We do NOT believe that it is the IPv6 timeout bug. It looks like the nginx-acme-module does not provide the /.well-known path with the token.
We notice an appendix of ? on each token.

http finalize request: 404, "/.well-known/acme-challenge/urHN7xquTdZtjrO0R0Q8Iw?" a:1, c:1
http special response: 404, "/.well-known/acme-challenge/urHN7xquTdZtjrO0R0Q8Iw?"

http copy filter: 0 "/.well-known/acme-challenge/hTg3_Ya2WjG8SMOpOm5nqg?"
http finalize request: 0, "/.well-known/acme-challenge/hTg3_Ya2WjG8SMOpOm5nqg?" a:1, c:1

[bavshin-f5]

Thanks for the report.
I'm afraid there's not enough information to figure out where the problem is. Would it be possible to share full debug log with timestamps?

It might be still insufficient, so I'm planning to improve debug logs and test with EJBCA before the next release.

[bmos1]

Thanks for the report. I'm afraid there's not enough information to figure out where the problem is. Would it be possible to share full debug log with timestamps?

It might be still insufficient, so I'm planning to improve debug logs and test with EJBCA before the next release.

Thanks for your fast reply. We appreciate that you plan to improve the debug logs. Of course, we can share a full debug logs, but we must be careful about making them public on Github. Or should we provide a redacted log where all sensitive information has been removed/replaced?

[bavshin-f5]

Thanks for your fast reply. We appreciate that you plan to improve the debug logs. Of course, we can share a full debug logs, but we must be careful about making them public on Github. Or should we provide a redacted log where all sensitive information has been removed/replaced?

A redacted fragment with timestamps starting from "acme: updating certificates" and to the incorrectResponse error should be sufficient.

[bmos1]

Here are the debug logs. Additionally we provide the redacted nginx.conf and access.log file. The fragment does NOT contain the required line "acme: updating certificates" anyways.

nginx-debug.zip

Here are the installed packages:

Package: nginx-module-acme
Version: 1.29.1+0.1.1-1~noble
Priority: optional
Section: httpd
Maintainer: NGINX Packaging nginx-packaging@f5.com
Installed-Size: 7,905 kB
Provides: nginx-module-acme-r1.29.1
Depends: libc6 (>= 2.39), libgcc-s1 (>= 4.2), nginx-r1.29.1
Homepage: https://nginx.org/
Download-Size: 2,122 kB
APT-Manual-Installed: yes
APT-Sources: https://nginx.org/packages/mainline/ubuntu noble/nginx amd64 Packages
Description: nginx nginx-acme
nginx-acme for nginx

Package: nginx
Version: 1.29.1-1~noble
Priority: optional
Section: httpd
Maintainer: NGINX Packaging nginx-packaging@f5.com
Installed-Size: 3,750 kB
Provides: httpd, nginx, nginx-r1.29.1
Depends: libc6 (>= 2.34), libcrypt1 (>= 1:4.1.0), libpcre2-8-0 (>= 10.22), libssl3t64 (>= 3.0.0), zlib1g (>= 1:1.1.4), lsb-base (>= 3.0-6)
Recommends: logrotate
Conflicts: nginx-common, nginx-core
Replaces: nginx-common, nginx-core
Homepage: https://nginx.org
Download-Size: 1,161 kB
APT-Manual-Installed: yes
APT-Sources: https://nginx.org/packages/mainline/ubuntu noble/nginx amd64 Packages
Description: high performance web server

[bavshin-f5]

That is an interesting sequence of events:

1. NGINX sends an initial POST-as-GET to the challenge URL to fetch the status.
2. EJBCA starts the challenge and fails.
3. EJBCA responds to the initial POST with urn:ietf:params:acme:error:incorrectResponse.

It is a spec violation in our code: the only specified way to interact with the challenge resource is to activate it by sending a POST with payload {}. The status check and polling should be done on the authorization resource (RFC 8555 § 7.5.1).
All the server implementation I tested before allowed POST-as-GET on the challenge URL though, so it was a plausible misinterpretation.

EJBCA seems to deviate from the spec in a different way, by initiating the challenge verification on POST-as-GET, before the client confirms its readiness by submitting a POST with payload {} to the challenge URL. It is acceptable if we assume that no other requests are permitted on the challenge resource, albeit different from other implementations.

I'm planning to rework the authorization status polling in a spec-compliant way and address this in the next release in a week or two.

[bmos1]

Thanks for your fast response. That sounds like we identified issues in both implementation. :-)
We opened another bug issue on Keyfactors support website too. Here is the issue content.

We utilize Keyfactors EJBCA and the ACME protocol to automate the certificate issuing of internal services. We want to inform you about a ACME protocol deviation that leads to incompatibility with the NGINX Web Server ACME module.

We opened a bug issue on Github #40 that the HTTP-01 challenge is NOT working. Together with the F5 developers we found out that NGINX sends an POST-as-GET to the challenge URL.

Problem:
Keyfactors EJBCA seems to deviate from the ACME spec, by initiating the challenge verification on POST-as-GET, before the client confirms its readiness by submitting a POST with payload {} to the challenge URL.

[bmos1]

Hi bavshin,

FYI, here is the Keyfactors support response.

This sentence “Keyfactors EJBCA seems to deviate from the ACME spec, by initiating the challenge verification on POST-as-GET, before the client confirms its readiness by submitting a POST with payload {} to the challenge URL.“ is definitely wrong.

We support an empty string beside {} (e30 in base64Url) since the beginning. There are historical reasons. To disallow an empty string rather could cause trouble. This is also not an issue.

Keyfactor won't fix its behavior.