Bug: Custom log formatter code not being reached #5503
Description
Expected Behaviour
We have hundreds of lambda functions deployed with SAM. We globally set POWERTOOLS_LOGGER_LOG_EVENT to True in order for the event of each invocation to be logged. In order to not log sensitve information, such as authoriztion headers from an APIGW event, I'm redacting certain fields due to security and in efforts to also reduce the overall size of the log in order to keep CloudWatch costs down.
The custom logger should delete these certain fields from the event payload before being logged.
Current Behaviour
The serialize() method never gets reached and none of my logger logic is performed. When I import the logger in a python shell, it works as expected and the custom code is ran. However, my logger doesn't appear to call the def serializer(self) method unless I add these two lines below. Problem with this method is that the lambda event also gets modified. So when I try to redact the Authorization key in the headers, my lambda authorizer throws a key error because it's missing from the event payload now.:
handler = logger.handlers[0]
handler.setFormatter(CustomFormatter())
Code snippet
Lambda handler:
logger = Logger(logger_formatter=CustomFormatter())
@tracer.capture_lambda_handler(
capture_response=False
)
@logger.inject_lambda_context()
def lambda_authorizer_handler(event: dict, context: LambdaContext) -> dict:
return authorizer.handler(event, context, logger)
Custom Logger
class CustomFormatter(LambdaPowertoolsFormatter):
"""
A custom logging formatter that redacts specific fields from log data before serialization.
This is in order to remove sensitive informtion and reduce CloudWatch log costs as it greatly
reduces the size of the log
Attributes:
HEADERS_TO_REDACT (Set[str]): Headers to be redacted from the log's 'headers' field.
REQUEST_CONTEXT_TO_REDACT (Set[str]): Request context fields to redact from the log's 'requestContext'.
MESSAGE_FIELDS_TO_REDACT (Set[str]): Message-level fields to redact from the log's 'message' field.
LAMBDA_EVENT_FIELDS_TO_REDACT (Set[str]): Top-level Fields specific to Lambda events to redact from the main log body.
"""
HEADERS_TO_REDACT: Set[str] = {
"Authorization",
"baggage",
}
REQUEST_CONTEXT_TO_REDACT: Set[str] = {
"resourcePath",
"extendedRequestId",
"protocol",
"stage",
"domainPrefix",
"identity",
"domainName",
"deploymentId",
"apiId",
}
MESSAGE_FIELDS_TO_REDACT: Set[str] = {
"multiValueHeaders",
"multiValueQueryStringParameters",
"pathParameters",
"stageVariables",
"isBase64Encoded",
}
LAMBDA_EVENT_FIELDS_TO_REDACT: Set[str] = {
"function_memory_size",
"function_arn",
"cold_start",
}
def __init__(self, *args, **kwargs):
print("Inside INIT of custom logger")
super().__init__(*args, **kwargs)
def serialize(self, log: LogRecord) -> str:
print(f"In Serialize method: {log}")
try:
self._redact_fields(log)
except Exception:
print("FAILURE")
pass
return self.json_serializer(log)
def _pop_fields(self, log: Dict[str, Any], keys_to_redact: Set[str]) -> None:
for key in keys_to_redact:
log.pop(key, None)
def _redact_fields(self, log: LogRecord) -> None:
message = log.get("message", {})
if isinstance(message, dict):
headers = message.get("headers")
if isinstance(headers, dict):
self._pop_fields(headers, self.HEADERS_TO_REDACT)
request_context = message.get("requestContext")
if isinstance(request_context, dict):
self._pop_fields(request_context, self.REQUEST_CONTEXT_TO_REDACT)
self._pop_fields(message, self.MESSAGE_FIELDS_TO_REDACT)
self._pop_fields(log, self.LAMBDA_EVENT_FIELDS_TO_REDACT) # type: ignore
log["message"] = message
Possible Solution
No response
Steps to Reproduce
Setup lambda function to use the CustomLogger as the logger_formatter and invoke the function locally or in the cloud
Powertools for AWS Lambda (Python) version
3.2.0
AWS Lambda function runtime
3.9
Packaging format used
Lambda Layers
Debugging logs
No response