Asynchronous uploading of images results in memory leakage

[hotwinters]

Describe the bug

I used ffmpeg to decode and encode the image, and passed it to aws sdk cpp through a pointer. I found that my program's memory would continue to rise. I tried valgrind, but I couldn't find a solution. I found that as long as I commented out client ->PutObject Asynchronous, my memory wouldn't continue to rise

code.txt

Regression Issue

• Select this option if this issue appears to be a regression.

Expected Behavior

1

Current Behavior

1

Reproduction Steps

1

Possible Solution

No response

Additional Information/Context

No response

AWS CPP SDK version used

1.11.111and1.11.400

Compiler and Version used

8.5

Operating System and version

centos8

[sbiscigl]

The code you attached wont compile on its own. please create a minimal, reproducible example. Since we are talking about memory here please run with valgrind to check if and where a leak is.

We can dive deeper once you have a full reproducible example and valgrind reporting where a leak would be.

AWS CPP SDK version used 1.11.111and1.11.400

We dont patch older versions, please use the most up to date version to get patched the fastest

[sbiscigl]

I spend some time creating a minimal example for you

memory-leak-reproduciton.zip

Given a project setup

~/memory-leak-reproduciton ❯❯❯ tree -L 1
.
├── CMakeLists.txt
├── Dockerfile
├── main.cpp
└── replicate.sh

CMakeLists.txt

cmake_minimum_required(VERSION 3.13)
project(sdk_usage_workspace)
set(CMAKE_CXX_STANDARD 20)
find_package(AWSSDK REQUIRED COMPONENTS s3)
add_executable(${PROJECT_NAME} "main.cpp")
target_link_libraries(${PROJECT_NAME} PRIVATE  ${AWSSDK_LINK_LIBRARIES})

Dockerfile

FROM public.ecr.aws/amazonlinux/amazonlinux:2023
RUN yum groupinstall "Development Tools" -y
RUN yum install -y curl-devel openssl-devel ninja-build cmake3 valgrind

# build and install SDK
RUN git clone --depth 1 --recurse-submodules https://github.com/aws/aws-sdk-cpp && \
    cd aws-sdk-cpp && \
    mkdir build && \
    cd build && \
    cmake -DBUILD_ONLY="s3" -DCMAKE_INSTALL_PREFIX=/sdk-install -DAUTORUN_UNIT_TESTS=OFF .. && \
    cmake --build . && \
    cmake --install .

# Copy over and build
RUN mkdir sdk-example
COPY CMakeLists.txt /sdk-example/CMakeLists.txt
COPY main.cpp /sdk-example/main.cpp
RUN cd sdk-example &&\
    mkdir build &&\
    cd build &&\
    cmake -DCMAKE_PREFIX_PATH=/sdk-install .. && \
    cmake --build .

main.cpp

#include <aws/core/Aws.h>
#include <aws/s3/S3Client.h>
#include <aws/s3/model/PutObjectRequest.h>

using namespace Aws;
using namespace Aws::S3;
using namespace Aws::S3::Model;
using namespace Aws::Utils;

namespace {
const char* LOG_TAG = "test-app";
const char* BUCKET_NAME = "yourbucketname"; //TODO: replace with your bucket
const char* KEY = "yourkey"; //TODO: replace with your key
}

auto main() -> int {
  SDKOptions options{};
  options.loggingOptions.logLevel = Logging::LogLevel::Debug;
  InitAPI(options);
  {
    S3Client client{};
    PutObjectRequest put_object_request{};
    put_object_request.SetBucket(BUCKET_NAME);
    put_object_request.SetKey(KEY);
    const auto body = Aws::MakeShared<StringStream>(LOG_TAG, "test body");
    put_object_request.SetBody(body);
    std::condition_variable cv{};
    std::mutex mtx{};
    bool request_completed = false;
    client.PutObjectAsync(put_object_request,
      [&mtx, & request_completed, &cv](const S3Client*,
        const Model::PutObjectRequest&,
        const Model::PutObjectOutcome& outcome,
        const std::shared_ptr<const Aws::Client::AsyncCallerContext>&) -> void {
          std::unique_lock lock{mtx};
          assert(outcome.IsSuccess());
          request_completed = true;
          cv.notify_one();
      });

    std::unique_lock lock{mtx};
    cv.wait(lock, [&request_completed] { return request_completed; });
  }
  ShutdownAPI(options);
  return 0;
}

replicate.sh

#!/bin/zsh

set -u

# build image
docker build -t test-image .

# run example
docker run \
    -e AWS_ACCESS_KEY_ID=${AWS_ACCESS_KEY_ID} \
    -e AWS_SECRET_ACCESS_KEY=${AWS_SECRET_ACCESS_KEY} \
    -e AWS_SESSION_TOKEN=${AWS_SESSION_TOKEN} \
    --name test-image test-image valgrind --leak-check=yes /sdk-example/build/sdk_usage_workspace

when i run ./replicate.sh given that i have a valid AWS_ACCESS_KEY_ID/AWS_SECRET_ACCESS_KEY/AWS_SESSION_TOKEN environment variable set. and that I have replaced the bucket and key in the code to reflect my bucket and key i see:

==1== Memcheck, a memory error detector
==1== Copyright (C) 2002-2022, and GNU GPL'd, by Julian Seward et al.
==1== Using Valgrind-3.19.0 and LibVEX; rerun with -h for copyright info
==1== Command: /sdk-example/build/sdk_usage_workspace
==1==
==1==
==1== HEAP SUMMARY:
==1==     in use at exit: 16 bytes in 1 blocks
==1==   total heap usage: 186,251 allocs, 186,250 frees, 18,013,258 bytes allocated
==1==
==1== LEAK SUMMARY:
==1==    definitely lost: 0 bytes in 0 blocks
==1==    indirectly lost: 0 bytes in 0 blocks
==1==      possibly lost: 0 bytes in 0 blocks
==1==    still reachable: 16 bytes in 1 blocks
==1==         suppressed: 0 bytes in 0 blocks
==1== Reachable blocks (those to which a pointer was found) are not shown.
==1== To see them, rerun with: --leak-check=full --show-leak-kinds=all
==1==
==1== For lists of detected and suppressed errors, rerun with: -s
==1== ERROR SUMMARY: 0 errors from 0 contexts (suppressed: 0 from 0)

which is showing that there is no memory leaks (please refer to this about still reachable, they are not memory leaks).

If you can update this example to successfully show a memory leak, id be more than happy to dig into it.

Additionally if you run valgrind on your application it may reveal something we do not see in our sample, so if you have the output of that it could also be helpful

[github-actions]

Greetings! It looks like this issue hasn’t been active in longer than a week. We encourage you to check if this is still an issue in the latest release. Because it has been longer than a week since the last update on this, and in the absence of more information, we will be closing this issue soon. If you find that this is still a problem, please feel free to provide a comment or add an upvote to prevent automatic closure, or if the issue is already closed, please feel free to open a new one.