@aws-sdk/client-cloudwatch-logs::GetQueryResultsCommand - Log Insights results missing parsed fields

[stripedpajamas]

Checkboxes for prior research

• I've gone through Developer Guide and API reference
• I've checked AWS Forums and StackOverflow.
• I've searched for previous similar issues and didn't find any solution.

Describe the bug

Log Insights queries that use parse don't seem to work properly. The returned data only has the @ptr fields, not the parsed fields. This behavior differs from using the CLI or other SDKs (e.g. the Rust SDK). I found this behavior with v2 and v3 of the JS SDK.

Regression Issue

• Select this option if this issue appears to be a regression.

SDK version number

@aws-sdk/client-cloudwatch-logs@3.758.0

Which JavaScript Runtime is this issue in?

Node.js

Details of the browser/Node.js/ReactNative version

18.12.1

Reproduction Steps

For instance, parsing the REPORT line from logs emitted from a Lambda function:

const queryString = "filter @type = 'REPORT' | parse @message /REPORT (?:.+)Memory Size: (?<memoryLimit>\d+) MB\sMax Memory Used: (?<memoryUsed>\d+) MB/";
const queryProps = {
  logGroupName: '/aws/lambda/my-lambda-fn',
  startTime: 1741390525,
  endTime: 1741392337,
  queryString,
};
const { queryId } = await logsClient.send(new StartQueryCommand(queryProps));
let res;
for (let i = 0; i < 5; i++) {
  res = await logsClient.send(new GetQueryResultsCommand({ queryId }));
  if (res.status === 'Complete') {
    break;
  }
  await sleep(1000);
}
console.log(JSON.stringify(res, null, 2));

Outputs:

{
  "$metadata": {
    "httpStatusCode": 200,
    "requestId": "0992f423-7b6d-489d-acaf-dc5ff1c0886a",
    "attempts": 1,
    "totalRetryDelay": 0
  },
  "queryLanguage": "CWLI",
  "results": [
    [
      {
        "field": "@ptr",
        "value": "<snip>"
      }
    ],
    [
      {
        "field": "@ptr",
        "value": "<snip>"
      }
    ]
  ],
  "statistics": {
    "bytesScanned": 185824,
    "estimatedBytesSkipped": 0,
    "estimatedRecordsSkipped": 0,
    "logGroupsScanned": 1,
    "recordsMatched": 2,
    "recordsScanned": 708
  },
  "status": "Complete"
}

Whereas if I run the same query through the AWS CLI:

$ aws --region us-west-2 logs start-query \
  --log-group-name '/aws/lambda/my-lambda-fn' \
  --query-string "filter @type = 'REPORT' | parse @message /REPORT (?:.+)Memory Size: (?<memoryLimit>\d+) MB\sMax Memory Used: (?<memoryUsed>\d+) MB/" \
  --start-time 1741390525 --end-time 1741392337
{
    "queryId": "a16d6798-ee95-4d04-a02f-ff670d2de19b"
}
$ aws --region us-west-2 logs get-query-results --query-id "a16d6798-ee95-4d04-a02f-ff670d2de19b"

{
    "results": [
        [
            {
                "field": "memoryLimit",
                "value": "600"
            },
            {
                "field": "memoryUsed",
                "value": "295"
            },
            {
                "field": "@ptr",
                "value": "<snip>"
            }
        ],
        [
            {
                "field": "memoryLimit",
                "value": "600"
            },
            {
                "field": "memoryUsed",
                "value": "296"
            },
            {
                "field": "@ptr",
                "value": "<snip>"
            }
        ]
    ],
    "statistics": {
        "recordsMatched": 2.0,
        "recordsScanned": 708.0,
        "bytesScanned": 185824.0
    },
    "status": "Complete"
}

Code emitted for brevity, but the same query using the Rust SDK outputs the parsed fields like the AWS CLI does.

Observed Behavior

The parsed fields in the query were not included in the output of GetQueryResults.

Expected Behavior

The parsed fields in the query to be included in the output of GetQueryResults. I am aware that the docs for GetQueryResults say "Only the fields requested in the query are returned, along with a @ptr field..." but I'm not sure exactly what that means (seeing as my results differ based on what tool I use to make the same query). I have tried adjusting the query in various ways, but can never get the parsed fields to show up in the return data.

Possible Solution

No response

Additional Information/Context

No response

[zshzbh]

Hey @stripedpajamas ,

I used JS SDK in lambda to generate a queryId and used the queryId in the cli, I got the same result which looks like

    [
      {
        "field": "@ptr",
        "value": "Cm4KMQodNDcxMTEyNjIzMjA2Oi9hd3MvbGFtYmRhLzY4OTYQByIOCJbXx6/aMhC/pfmw2jISNRoYAgZ7DX9hAAAAAEc3l8sABn2JSYAAAAESIAEojNSSsNoyMMvUlrDaMjgJQOMMSO4UUMEMGAAgARAIGAE="
      }
    ],
    [
      {
        "field": "@ptr",
        "value": "Cm4KMQodNDcxMTEyNjIzMjA2Oi9hd3MvbGFtYmRhLzY4OTYQByIOCJbXx6/aMhC/pfmw2jISNRoYAgZ7DX9hAAAAAEc3l8sABn2JSYAAAAESIAEojNSSsNoyMMvUlrDaMjgJQOMMSO4UUMEMGAAgARAFGAE="
      }
    ],
          

But if I use cli to generate queryId and get query result in cli, I got the log with parsed field, which looks like -

       [
            {
                "field": "memoryLimit",
                "value": "128"
            },
            {
                "field": "memoryUsed",
                "value": "81"
            },
            {
                "field": "@ptr",
                "value": "Cm4KMQodNDcxMTEyNjIzMjA2Oi9hd3MvbGFtYmRhLzY4OTYQAiIOCJbXx6/aMhC/pfmw2jISNRoYAgZ3MGjxAAAAAG84KacABn2JYAAAAAUSIAEoncOssNoyMIzGrLDaMjgEQL8ESKQRUPcIGAAgARADGAE="
            }
        ],

The code I have in lambda -

import {CloudWatchLogsClient, StartQueryCommand, GetQueryResultsCommand} from '@aws-sdk/client-cloudwatch-logs'
export const handler = async (event) => {
  // TODO implement
  const sleep = (ms) => new Promise(resolve => setTimeout(resolve, ms));

  const logsClient = new CloudWatchLogsClient({ region: 'us-east-1' });
  const queryString = "filter @type = 'REPORT' | parse @message /REPORT (?:.+)Memory Size: (?<memoryLimit>\d+) MB\sMax Memory Used: (?<memoryUsed>\d+) MB/";
const queryProps = {
  logGroupName: '/aws/lambda/6896',
  startTime: 1742205559,//	Mon Mar 17 2025 02:59:19 GMT-0700 (Pacific Daylight Time)
  endTime: 1742248759, //Mon Mar 17 2025 14:59:19 GMT-0700 (Pacific Daylight Time)
  queryString,
};
const { queryId } = await logsClient.send(new StartQueryCommand(queryProps));


let res;
for (let i = 0; i < 5; i++) {
  res = await logsClient.send(new GetQueryResultsCommand({ queryId }));
  if (res.status === 'Complete') {
    break;
  }
  await sleep(1000);// wait for 1 s
}
console.log("queryID", queryId);
console.log("status", res.status);
console.log(JSON.stringify(res, null, 2));
  return queryId;
};
I edited the lambda code in local development env and got the same result. 

I will talk about it with the team tomorrow and see if it's a designed behavior or a bug.

Thanks!
Maggie

[zshzbh]

Upate -

I added --debug in cli and added middlewareStack at the finalizeRequest step to log the serialized queryString that SDK generated. Posting my findings -

queryString after serialization generated by sdk - \n filter @type = 'REPORT' | parse @message /REPORT (?:.+)Memory Size: (?<memoryLimit>d+) MBsMax Memory Used: (?<memoryUsed>d+) MB/

query-string in cli - filter @type = 'REPORT' | parse @message /REPORT (?:.+)Memory Size: (?<memoryLimit>\\d+) MB\\sMax Memory Used: (?<memoryUsed>\\d+) MB/

Sharing full logs in this repo's README
Comparing two queryStrings -
CLI is the one on top.

[stripedpajamas]

Oh I see, it's just an escaping thing. The JS SDK works fine if I add the extra slashes in my regex:

const queryString = "filter @type = 'REPORT' | parse @message /REPORT (?:.+)Memory Size: (?<memoryLimit>\\d+) MB\\sMax Memory Used: (?<memoryUsed>\\d+) MB/";
const queryProps = {
  logGroupName: '/aws/lambda/my-lambda-fn',
  startTime: 1741390525,
  endTime: 1741392337,
  queryString,
};
const { queryId } = await logsClient.send(new StartQueryCommand(queryProps));
let res;
for (let i = 0; i < 5; i++) {
  res = await logsClient.send(new GetQueryResultsCommand({ queryId }));
  if (res.status === 'Complete') {
    break;
  }
  await sleep(1000);
}
console.log(JSON.stringify(res, null, 2));

Response:

{
  "$metadata": {
    "httpStatusCode": 200,
    "requestId": "880b9e76-d85d-4407-90b7-bd6e2bf93ecb",
    "attempts": 1,
    "totalRetryDelay": 0
  },
  "queryLanguage": "CWLI",
  "results": [
    [
      {
        "field": "memoryLimit",
        "value": "600"
      },
      {
        "field": "memoryUsed",
        "value": "295"
      },
      {
        "field": "@ptr",
        "value": "<snip>"
      }
    ],
    [
      {
        "field": "memoryLimit",
        "value": "600"
      },
      {
        "field": "memoryUsed",
        "value": "296"
      },
      {
        "field": "@ptr",
        "value": "<snip>"
      }
    ]
  ],
  "statistics": {
    "bytesScanned": 185824,
    "estimatedBytesSkipped": 0,
    "estimatedRecordsSkipped": 0,
    "logGroupsScanned": 1,
    "recordsMatched": 2,
    "recordsScanned": 708
  },
  "status": "Complete"
}

This issue can likely be closed, thanks for the help @zshzbh