[BUG] Endless checking of an unpaid invoice (processor LNBits)

[Perlover]

Describe the bug

Every minute, several times a second, LNBits is contacted to check the status of the LN invoice (my nostream is paid relay), which was canceled 6 days ago. LNBits returns a 404 error, since this hash of the invoice no longer exists for him. But norstream keeps checking the bill again and again every minute. A restart of nostream doesn't help.

To Reproduce

I do not know, I know that on that day I just deployed the nostream server. Perhaps this is due to the fact that at the beginning I mistakenly indicated in settings.yml:

• payments.feeSchedules.admission.amount: 199000
• limits.event.pubkey.minBalance: 1000000

I forgot to adjust the second parameter. Having paid 199 sats for a user, I thought that his account would work. But it turned out that the minimum was 1000 sats. Then I paid for him 5 more times and his account started working. Later, I changed the second parameter to 199 sats. I don't know if it caused this error or not, but it's the only thing that comes to my mind.

Expected behavior

I think that if the payment verification already shows a HTTP 404 error from LNBits - this should stop. Or stop at least for the reason that the account has expired a long time ago (it had an expiration date of 10 minutes)

Screenshots

To see logs below

System (please complete the following information):

• OS: Ubuntu 22.04
• Platform: Docker
• Version: d1d4cb9e2512e44812f539a5da61db0e97cec266 (v1.25.2 + 3 commits)

Logs

Jul 12 16:57:53 my.host.com start[1141393]: nostream          | Unable to get invoice 523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a. Reason: AxiosError: Request failed with status code 404
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     at settle (/app/node_modules/axios/dist/node/axios.cjs:1859:12)
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     at IncomingMessage.handleStreamEnd (/app/node_modules/axios/dist/node/axios.cjs:2723:11)
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     at IncomingMessage.emit (node:events:525:35)
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     at endReadableNT (node:internal/streams/readable:1359:12)
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |   code: 'ERR_BAD_REQUEST',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |   config: {
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     transitional: {
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       silentJSONParsing: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       forcedJSONParsing: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       clarifyTimeoutError: false
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     },
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     adapter: [ 'xhr', 'http' ],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     transformRequest: [ [Function: transformRequest] ],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     transformResponse: [ [Function: transformResponse] ],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     timeout: 0,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     xsrfCookieName: 'XSRF-TOKEN',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     xsrfHeaderName: 'X-XSRF-TOKEN',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     maxContentLength: -1,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     maxBodyLength: -1,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     env: { FormData: [Function], Blob: [class Blob] },
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     validateStatus: [Function: validateStatus],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     headers: AxiosHeaders {
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       Accept: 'application/json, text/plain, */*',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       'content-type': 'application/json',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       'x-api-key': 'XXXXXXXXXXXXXXXXXXXXXXXXXXXX',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       'User-Agent': 'axios/1.2.6',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       'Accept-Encoding': 'gzip, compress, deflate, br'
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     },
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     baseURL: 'https://lnbits.my.host.com/',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     maxRedirects: 1,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     method: 'get',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     url: '/api/v1/payments/523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     data: undefined
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |   },
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |   request: <ref *1> ClientRequest {
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     _events: [Object: null prototype] {
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       abort: [Function (anonymous)],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       aborted: [Function (anonymous)],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       connect: [Function (anonymous)],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       error: [Function (anonymous)],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       socket: [Function (anonymous)],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       timeout: [Function (anonymous)],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       finish: [Function: requestOnFinish]
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     },
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     _eventsCount: 7,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     _maxListeners: undefined,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     outputData: [],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     outputSize: 0,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     writable: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     destroyed: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     _last: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     chunkedEncoding: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     shouldKeepAlive: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     maxRequestsOnConnectionReached: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     _defaultKeepAlive: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     useChunkedEncodingByDefault: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     sendDate: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     _removedConnection: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     _removedContLen: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     _removedTE: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     strictContentLength: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     _contentLength: 0,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     _hasBody: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     _trailer: '',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     finished: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     _headerSent: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     _closed: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     socket: TLSSocket {
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _tlsOptions: [Object],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _secureEstablished: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _securePending: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _newSessionPending: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _controlReleased: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       secureConnecting: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _SNICallback: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       servername: 'lnbits.my.host.com',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       alpnProtocol: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       authorized: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       authorizationError: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       encrypted: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _events: [Object: null prototype],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _eventsCount: 10,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       connecting: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _hadError: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _parent: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _host: 'lnbits.my.host.com',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _closeAfterHandlingError: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _readableState: [ReadableState],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _maxListeners: undefined,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _writableState: [WritableState],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       allowHalfOpen: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _sockname: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _pendingData: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _pendingEncoding: '',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       server: undefined,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _server: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       ssl: [TLSWrap],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _requestCert: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _rejectUnauthorized: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       parser: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _httpMessage: [Circular *1],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(res)]: [TLSWrap],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(verified)]: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(pendingSession)]: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(async_id_symbol)]: 78,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kHandle)]: [TLSWrap],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(lastWriteQueueSize)]: 0,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(timeout)]: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kBuffer)]: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kBufferCb)]: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kBufferGen)]: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kCapture)]: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kSetNoDelay)]: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kSetKeepAlive)]: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kSetKeepAliveInitialDelay)]: 60,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kBytesRead)]: 0,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kBytesWritten)]: 0,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(connect-options)]: [Object]
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     },
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     _header: 'GET /api/v1/payments/523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a HTTP/1.1\r\n' +
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       'Accept: application/json, text/plain, */*\r\n' +
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       'content-type: application/json\r\n' +
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       'x-api-key: XXXXXXXXXXXXXXXXXXXXXXXXXXXX\r\n' +
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       'User-Agent: axios/1.2.6\r\n' +
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       'Accept-Encoding: gzip, compress, deflate, br\r\n' +
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       'Host: lnbits.my.host.com\r\n' +
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       'Connection: close\r\n' +
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       '\r\n',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     _keepAliveTimeout: 0,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     _onPendingData: [Function: nop],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     agent: Agent {
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _events: [Object: null prototype],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _eventsCount: 2,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _maxListeners: undefined,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       defaultPort: 443,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       protocol: 'https:',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       options: [Object: null prototype],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       requests: [Object: null prototype] {},
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       sockets: [Object: null prototype],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       freeSockets: [Object: null prototype] {},
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       keepAliveMsecs: 1000,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       keepAlive: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       maxSockets: Infinity,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       maxFreeSockets: 256,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       scheduling: 'lifo',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       maxTotalSockets: Infinity,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       totalSocketCount: 1,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       maxCachedSessions: 100,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _sessionCache: [Object],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kCapture)]: false
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     },
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     socketPath: undefined,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     method: 'GET',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     maxHeaderSize: undefined,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     insecureHTTPParser: undefined,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     joinDuplicateHeaders: undefined,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     path: '/api/v1/payments/523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     _ended: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     res: IncomingMessage {
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _readableState: [ReadableState],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _events: [Object: null prototype],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _eventsCount: 4,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _maxListeners: undefined,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       socket: [TLSSocket],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       httpVersionMajor: 1,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       httpVersionMinor: 1,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       httpVersion: '1.1',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       complete: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       rawHeaders: [Array],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       rawTrailers: [],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       joinDuplicateHeaders: undefined,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       aborted: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       upgrade: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       url: '',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       method: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       statusCode: 404,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       statusMessage: 'Not Found',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       client: [TLSSocket],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _consuming: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _dumped: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       req: [Circular *1],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       responseUrl: 'https://lnbits.my.host.com/api/v1/payments/523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       redirects: [],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kCapture)]: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kHeaders)]: [Object],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kHeadersCount)]: 10,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kTrailers)]: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kTrailersCount)]: 0
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     },
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     aborted: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     timeoutCb: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     upgradeOrConnect: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     parser: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     maxHeadersCount: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     reusedSocket: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     host: 'lnbits.my.host.com',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     protocol: 'https:',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     _redirectable: Writable {
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _writableState: [WritableState],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _events: [Object: null prototype],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _eventsCount: 3,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _maxListeners: undefined,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _options: [Object],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _ended: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _ending: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _redirectCount: 0,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _redirects: [],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _requestBodyLength: 0,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _requestBodyBuffers: [],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _onNativeResponse: [Function (anonymous)],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _currentRequest: [Circular *1],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _currentUrl: 'https://lnbits.my.host.com/api/v1/payments/523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kCapture)]: false
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     },
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     [Symbol(kCapture)]: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     [Symbol(kBytesWritten)]: 0,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     [Symbol(kNeedDrain)]: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     [Symbol(corked)]: 0,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     [Symbol(kOutHeaders)]: [Object: null prototype] {
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       accept: [Array],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       'content-type': [Array],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       'x-api-key': [Array],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       'user-agent': [Array],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       'accept-encoding': [Array],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       host: [Array]
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     },
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     [Symbol(errored)]: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     [Symbol(kUniqueHeaders)]: null
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |   },
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |   response: {
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     status: 404,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     statusText: 'Not Found',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     headers: AxiosHeaders {
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       server: 'nginx/1.18.0 (Ubuntu)',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       date: 'Wed, 12 Jul 2023 16:57:53 GMT',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       'content-type': 'application/json',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       'content-length': '36',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       connection: 'close'
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     },
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     config: {
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       transitional: [Object],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       adapter: [Array],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       transformRequest: [Array],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       transformResponse: [Array],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       timeout: 0,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       xsrfCookieName: 'XSRF-TOKEN',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       xsrfHeaderName: 'X-XSRF-TOKEN',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       maxContentLength: -1,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       maxBodyLength: -1,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       env: [Object],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       validateStatus: [Function: validateStatus],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       headers: [AxiosHeaders],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       baseURL: 'https://lnbits.my.host.com/',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       maxRedirects: 1,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       method: 'get',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       url: '/api/v1/payments/523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       data: undefined
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     },
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     request: <ref *1> ClientRequest {
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _events: [Object: null prototype],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _eventsCount: 7,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _maxListeners: undefined,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       outputData: [],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       outputSize: 0,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       writable: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       destroyed: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _last: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       chunkedEncoding: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       shouldKeepAlive: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       maxRequestsOnConnectionReached: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _defaultKeepAlive: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       useChunkedEncodingByDefault: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       sendDate: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _removedConnection: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _removedContLen: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _removedTE: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       strictContentLength: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _contentLength: 0,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _hasBody: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _trailer: '',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       finished: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _headerSent: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _closed: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       socket: [TLSSocket],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _header: 'GET /api/v1/payments/523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a HTTP/1.1\r\n' +
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |         'Accept: application/json, text/plain, */*\r\n' +
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |         'content-type: application/json\r\n' +
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |         'x-api-key: XXXXXXXXXXXXXXXXXXXXXXXXXXXX\r\n' +
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |         'User-Agent: axios/1.2.6\r\n' +
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |         'Accept-Encoding: gzip, compress, deflate, br\r\n' +
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |         'Host: lnbits.my.host.com\r\n' +
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |         'Connection: close\r\n' +
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |         '\r\n',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _keepAliveTimeout: 0,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _onPendingData: [Function: nop],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       agent: [Agent],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       socketPath: undefined,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       method: 'GET',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       maxHeaderSize: undefined,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       insecureHTTPParser: undefined,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       joinDuplicateHeaders: undefined,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       path: '/api/v1/payments/523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _ended: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       res: [IncomingMessage],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       aborted: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       timeoutCb: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       upgradeOrConnect: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       parser: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       maxHeadersCount: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       reusedSocket: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       host: 'lnbits.my.host.com',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       protocol: 'https:',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       _redirectable: [Writable],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kCapture)]: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kBytesWritten)]: 0,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kNeedDrain)]: false,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(corked)]: 0,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kOutHeaders)]: [Object: null prototype],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(errored)]: null,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       [Symbol(kUniqueHeaders)]: null
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     },
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     data: { detail: 'Payment does not exist.' }
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |   }
Jul 12 16:57:53 my.host.com start[1141393]: nostream          | }
Jul 12 16:57:53 my.host.com start[1141393]: nostream          | Unable to get invoice from payments processor. Reason: AxiosError: Request failed with status code 404
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     at settle (/app/node_modules/axios/dist/node/axios.cjs:1859:12)
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     at IncomingMessage.handleStreamEnd (/app/node_modules/axios/dist/node/axios.cjs:2723:11)
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     at IncomingMessage.emit (node:events:525:35)
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     at endReadableNT (node:internal/streams/readable:1359:12)
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |   code: 'ERR_BAD_REQUEST',
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |   config: {
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     transitional: {
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       silentJSONParsing: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       forcedJSONParsing: true,
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |       clarifyTimeoutError: false
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     },
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     adapter: [ 'xhr', 'http' ],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     transformRequest: [ [Function: transformRequest] ],
Jul 12 16:57:53 my.host.com start[1141393]: nostream          |     transformResponse: [ [Function: transformResponse] ],
...

Additional context

LND side - invoice data:

lncli lookupinvoice 523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a
{
    "memo": "YEStr.ME Admission Fee for npub17XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
    "r_preimage": "158c56c7YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY",
    "r_hash": "523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a",
    "value": "199",
    "value_msat": "199000",
    "settled": false,
    "creation_date": "1688624770",
    "settle_date": "0",
    "payment_request": "lnbc19XXXXXXXXXXXXXXXXXXXXXXXXX",
    "description_hash": null,
    "expiry": "600",
    "fallback_addr": "",
    "cltv_expiry": "80",
    "route_hints": [
    ],
    "private": false,
    "add_index": "1277",
    "settle_index": "0",
    "amt_paid": "0",
    "amt_paid_sat": "0",
    "amt_paid_msat": "0",
    "state": "CANCELED",
    "htlcs": [
    ],
    "features": {
        "9": {
            "name": "tlv-onion",
            "is_required": false,
            "is_known": true
        },
        "14": {
            "name": "payment-addr",
            "is_required": true,
            "is_known": true
        },
        "17": {
            "name": "multi-path-payments",
            "is_required": false,
            "is_known": true
        }
    },
    "is_keysend": false,
    "payment_addr": "b103eec86ba64f2170be7cd735c084067d5f073018d26011aeab305e5a2f241c",
    "is_amp": false,
    "amp_invoice_state": {
    }
}

Here:

$ date --date='@1688624770'
Thu Jul  6 06:26:10 UTC 2023

[Perlover]

And few other log pieces - there is "Unable to update invoice from payment processor." too..

Jul 12 17:23:53 my.host.com start[1141393]: nostream          | Unable to update invoice from payment processor. Reason: AxiosError: Request failed with status code 404
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     at settle (/app/node_modules/axios/dist/node/axios.cjs:1859:12)
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     at IncomingMessage.handleStreamEnd (/app/node_modules/axios/dist/node/axios.cjs:2723:11)
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     at IncomingMessage.emit (node:events:525:35)
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     at endReadableNT (node:internal/streams/readable:1359:12)
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |   code: 'ERR_BAD_REQUEST',
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |   config: {
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     transitional: {
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |       silentJSONParsing: true,
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |       forcedJSONParsing: true,
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |       clarifyTimeoutError: false
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     },
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     adapter: [ 'xhr', 'http' ],
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     transformRequest: [ [Function: transformRequest] ],
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     transformResponse: [ [Function: transformResponse] ],
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     timeout: 0,
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     xsrfCookieName: 'XSRF-TOKEN',
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     xsrfHeaderName: 'X-XSRF-TOKEN',
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     maxContentLength: -1,
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     maxBodyLength: -1,
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     env: { FormData: [Function], Blob: [class Blob] },
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     validateStatus: [Function: validateStatus],
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     headers: AxiosHeaders {
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |       Accept: 'application/json, text/plain, */*',
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |       'content-type': 'application/json',
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |       'x-api-key': 'XXXXXXXXXXXXXXXXXXXXXXXXXXXX',
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |       'User-Agent': 'axios/1.2.6',
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |       'Accept-Encoding': 'gzip, compress, deflate, br'
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     },
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     baseURL: 'https://lnbits.my.host.com/',
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     maxRedirects: 1,
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     method: 'get',
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     url: '/api/v1/payments/523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a',
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     data: undefined
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |   },
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |   request: <ref *1> ClientRequest {
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |     _events: [Object: null prototype] {
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |       abort: [Function (anonymous)],
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |       aborted: [Function (anonymous)],
Jul 12 17:23:53 my.host.com start[1141393]: nostream          |       connect: [Function (anonymous)],

[Perlover]

I think this is not full fixed may be this bug: #193

Because this error happens for pending invoices...

[Perlover]

After 3 weeks right now I see same errors for the same invoice 523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a...

[cameri]

@Perlover I will look into reproducing this issue.

[cameri]

@Perlover in the meantime, try removing the invoice from your invoices table

[Perlover]

@Perlover in the meantime, try removing the invoice from your invoices table

What is way to remove the invoice? SQL commands in the nostream SQL database?

[cameri]

You can check first that the invoice exists with:

docker compose exec nostream-db psql -U nostr_ts_relay -c "select * from invoices where id = '523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a';"

And then delete it with:

docker compose exec nostream-db psql -U nostr_ts_relay -c "delete from invoices where id = '523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a';"

[cameri]

@Perlover let me know if you were able to solve the issue

[Perlover]

Sorry for delay. I will do it on Monday.

[Perlover]

You can check first that the invoice exists with:

docker compose exec nostream-db psql -U nostr_ts_relay -c "select * from invoices where id = '523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a';"

$ docker compose exec nostream-db psql -U nostr_ts_relay -c "select * from invoices where id = '523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a';"
WARN[0000] The "OPENNODE_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "NODELESS_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "ZEBEDEE_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "NODELESS_WEBHOOK_SECRET" variable is not set. Defaulting to a blank string. 
                                id                                |                               pubkey                               |                                                                                                                                                                                      
                bolt11                                                                                                                                                                                                       | amount_requested | amount_paid | unit | status  |                                        descri
ption                                         | confirmed_at |     expires_at      |         created_at         |         updated_at         | verify_url 
------------------------------------------------------------------+--------------------------------------------------------------------+--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+------------------+-------------+------+---------+----------------------------------------------
----------------------------------------------+--------------+---------------------+----------------------------+----------------------------+------------
 523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a | \xf16ac67afff1XXXXXXXXXXXXXXXXXXXXX8425a73f2 | lnbc1990n1pj2vh5zpp5XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXv8q58gpfa4253 |              199 |             | sats | pending | YEStr.ME Admission Fee for npub17XXXXXXXXXXXXXXXXXXXXXjhd |              | 2023-07-06 06:36:10 | 2023-07-06 06:26:10.263+00 | 2023-07-06 06:36:03.156+00 | 
(1 row)

And then delete it with:

```shell
docker compose exec nostream-db psql -U nostr_ts_relay -c "delete from invoices where id = '523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a';"

$ docker compose exec nostream-db psql -U nostr_ts_relay -c "delete from invoices where id = '523a9e10e1a318b0a8e049a88b87e911afae5792b1b4f400805e660dd9366f1a';"

WARN[0000] The "NODELESS_WEBHOOK_SECRET" variable is not set. Defaulting to a blank string. 
WARN[0000] The "ZEBEDEE_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "NODELESS_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "OPENNODE_API_KEY" variable is not set. Defaulting to a blank string. 
DELETE 1

Thanks, it seems to have been successfully deleted. But judging by the logs - this invoice is not the only one, but there are others. I'll delete them :)

P.S. This invoice has been checked in the logs to this day.....

[Perlover]

And, without private data some info about the pending invoices in my DB:

$ docker compose exec nostream-db psql -U nostr_ts_relay -c "select id, amount_requested, amount_paid, status, expires_at, created_at, updated_at from invoices where status= 'pending';"
WARN[0000] The "NODELESS_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "NODELESS_WEBHOOK_SECRET" variable is not set. Defaulting to a blank string. 
WARN[0000] The "ZEBEDEE_API_KEY" variable is not set. Defaulting to a blank string. 
WARN[0000] The "OPENNODE_API_KEY" variable is not set. Defaulting to a blank string. 
                                id                                | amount_requested | amount_paid | status  |     expires_at      |         created_at         |         updated_at         
------------------------------------------------------------------+------------------+-------------+---------+---------------------+----------------------------+----------------------------
 01dd8fcf008675d383d61e959639572bb1e97763424696cb5f4f42be97dc8d4c |              199 |             | pending | 2023-08-24 05:53:23 | 2023-08-24 05:43:23.214+00 | 2023-08-24 05:52:37.57+00
 a1476ba1d681d24d6ef4a32824e1bb60141e0e8cf35df88f2cdf220b8ec5acaa |              199 |             | pending | 2023-08-24 05:51:28 | 2023-08-24 05:41:29.049+00 | 2023-08-24 05:50:37.522+00
 63b836d863bbb1ec03322cd057187e58ea4c9f784f743722b3e4794792ccc208 |              199 |             | pending | 2023-08-24 05:53:49 | 2023-08-24 05:43:50.185+00 | 2023-08-24 05:53:37.35+00
 c93101b1197c26b328b0cc595b78bfb5a3cd739be271b7c059322bf53345cc46 |              199 |             | pending | 2023-09-09 11:10:31 | 2023-09-09 11:00:31.921+00 | 2023-09-09 11:09:53.208+00
 5a889a5e5d0eb5500c7561fba502b4ea9e540c54f2fc28edfe920cecdb8915df |              199 |             | pending | 2023-09-09 11:10:41 | 2023-09-09 11:00:42.26+00  | 2023-09-09 11:09:53.383+00
 39a8907d1e505be4db0dc6c6fc5a11414e2e606221dbf14ebf13573aa6dfc1d3 |              199 |             | pending | 2023-09-09 11:10:49 | 2023-09-09 11:00:49.909+00 | 2023-09-09 11:09:53.558+00
 7b2d26c8f70272471f79533f3bc161430190d4d76c3e890b3d704fffdddaa1d5 |              199 |             | pending | 2023-09-09 11:10:22 | 2023-09-09 11:00:22.803+00 | 2023-09-09 11:09:53.034+00
 eececd36b3589c15bdee9e835a73e34068c6be590de54d4350f5451c88eaefaa |              199 |             | pending | 2023-11-01 10:03:49 | 2023-10-31 10:03:49.575+00 | 2023-11-01 10:03:15.002+00
 767c52863853c30b7ba01f4075441b4989da04f23cbc87a9df51d652fdd1de5d |              199 |             | pending | 2023-11-27 00:56:03 | 2023-11-26 00:56:03.531+00 | 2023-11-27 00:55:38.325+00
(9 rows)

For example right now in my logs:

Unable to get invoice from payments processor. Reason: AxiosError: Request failed with status code 404
    at settle (/app/node_modules/axios/dist/node/axios.cjs:1859:12)
    at IncomingMessage.handleStreamEnd (/app/node_modules/axios/dist/node/axios.cjs:2723:11)
    at IncomingMessage.emit (node:events:525:35)
    at endReadableNT (node:internal/streams/readable:1359:12)
    at process.processTicksAndRejections (node:internal/process/task_queues:82:21) {
  code: 'ERR_BAD_REQUEST',
  config: {
    transitional: {
      silentJSONParsing: true,
      forcedJSONParsing: true,
      clarifyTimeoutError: false
    },
    adapter: [ 'xhr', 'http' ],
    transformRequest: [ [Function: transformRequest] ],
    transformResponse: [ [Function: transformResponse] ],
    timeout: 0,
    xsrfCookieName: 'XSRF-TOKEN',
    xsrfHeaderName: 'X-XSRF-TOKEN',
    maxContentLength: -1,
    maxBodyLength: -1,
    env: { FormData: [Function], Blob: [class Blob] },
    validateStatus: [Function: validateStatus],
    headers: AxiosHeaders {
      Accept: 'application/json, text/plain, */*',
      'content-type': 'application/json',
      'x-api-key': 'XXXXXXXXXXXXXXXXXXX',
      'User-Agent': 'axios/1.2.6',
      'Accept-Encoding': 'gzip, compress, deflate, br'
    },
    baseURL: 'https://lnbits.XXXXXXXXXXXXXX.com/',
    maxRedirects: 1,
    method: 'get',
    url: '/api/v1/payments/01dd8fcf008675d383d61e959639572bb1e97763424696cb5f4f42be97dc8d4c',
    data: undefined
  },

....

    path: '/api/v1/payments/767c52863853c30b7ba01f4075441b4989da04f23cbc87a9df51d652fdd1de5d',
    _ended: true,
    res: IncomingMessage {
      _readableState: [ReadableState],
      _events: [Object: null prototype],
      _eventsCount: 4,
      _maxListeners: undefined,
      socket: [TLSSocket],
      httpVersionMajor: 1,
      httpVersionMinor: 1,
      httpVersion: '1.1',
      complete: true,
      rawHeaders: [Array],
      rawTrailers: [],
      joinDuplicateHeaders: undefined,
      aborted: false,
      upgrade: false,
      url: '',
      method: null,
      statusCode: 404,
      statusMessage: 'Not Found',