Upgrade to mountpoint-s3-client-0.8.0 and update logging (#176)

mountpoint-s3-client-0.8.0 contains changes incompatible with the logging setup.
Given the previous issues we had with this, we are separating these as follows:
- Logging in the main Python component remains handled by Python logger.
- Logging in Rust components is handled by tracing_subscriber, set up through S3_TORCH_CONNECTOR_DEBUG_LOGS and S3_TORCH_CONNECTOR_LOGS_DIR_PATH
environment variables.

Author: dnanuti

CHANGELOG.md

@@ -1,3 +1,12 @@
+## Unreleased
+
+### New features
+
+### Breaking changes
+* Separate completely Rust logs and Python logs. Logs from Rust components, used for debugging purposes 
+are configured through the following environment variables: S3_TORCH_CONNECTOR_DEBUG_LOGS, 
+S3_TORCH_CONNECTOR_LOGS_DIR_PATH.
+
 ## v1.2.0 (March 13, 2024)
 
 ### New features
@@ -11,6 +20,7 @@
 ### Breaking changes
 * No breaking changes.
 
+
 ## v1.1.4 (February 26, 2024)
 
 ### New features
@@ -37,7 +47,6 @@
 * No breaking changes.
 
 
-
 ## v1.1.2 (January 19, 2024)
 
 ### New features
@@ -48,7 +57,6 @@
 * No breaking changes.
 
 
-
 ## v1.1.1 (December 11, 2023)
 
 ### New features
@@ -69,12 +77,11 @@
 * No breaking changes.
 
 
-
 ## v1.0.0 (November 22, 2023)
 * The Amazon S3 Connector for PyTorch delivers high throughput for PyTorch training jobs that access and store data in Amazon S3.
 
 ### New features
-* S3IterableDataset and S3MapDataset, which allow building either an iterable-style or map-style dataset, using your S3 
+* S3IterableDataset and S3MapDataset, which allow building either an iterable-style or map-style dataset, using your S3
 stored data, by specifying an S3 URI (a bucket and optional prefix) and the region the bucket is in.
 * Support for multiprocess data loading for the above datasets.
 * S3Checkpoint, an interface for saving and loading model checkpoints directly to and from an S3 bucket.

doc/DEVELOPMENT.md

@@ -92,3 +92,49 @@ Fill in the path of the Python executable in your virtual environment (`venv/bin
 as the program argument.
 Then put a breakpoint in the Rust/C code and try running it.
 
+#### Enabling Debug Logging
+The [Python logger](https://docs.python.org/3/library/logging.html) handles logging messages from the Python-side 
+of our implementation.
+For debug purposes, you can also enable the logs for our Rust components, which are off by default. 
+These are handled by [tracing_subscriber](https://docs.rs/tracing-subscriber/latest/tracing_subscriber/) and can be 
+configured through the following environment variables:
+- `S3_TORCH_CONNECTOR_DEBUG_LOGS` - Configured similarly to the
+[RUST_LOG](https://docs.rs/env_logger/latest/env_logger/#enabling-logging) variable for
+filtering logs from our Rust components. This includes finer granularity logs from 
+[AWS Common Runtime (CRT)](https://docs.aws.amazon.com/sdkref/latest/guide/common-runtime.html).
+**Please note that the AWS CRT logs are very noisy. We recommend to filter them out by appending `"awscrt=off"` to
+your S3_TORCH_CONNECTOR_DEBUG_LOGS setup.**
+- `S3_TORCH_CONNECTOR_LOGS_DIR_PATH` - The path to a local directory where you have write permissions. 
+When configured, the logs from the Rust components will be appended to a file at this location. 
+This will result in a log file located at 
+`${S3_TORCH_CONNECTOR_LOGS_DIR_PATH}/s3torchconnectorclient.log.yyyy-MM-dd-HH`, rolled on an hourly basis. 
+The log messages of the latest run are appended to the end of the most recent log file.
+
+**Examples**
+- Configure INFO level logs to be written to STDOUT:
+```sh
+  export S3_TORCH_CONNECTOR_DEBUG_LOGS=info
+```
+
+- Enable TRACE level logs (most verbose) to be written at `/tmp/s3torchconnector-logs`:
+```sh
+  export S3_TORCH_CONNECTOR_DEBUG_LOGS=trace
+  export S3_TORCH_CONNECTOR_LOGS_DIR_PATH="/tmp/s3torchconnector-logs"
+```
+After running your script, you will find the logs under `/tmp/s3torchconnector-logs`.
+The file will include AWS CRT logs. 
+
+- Enable TRACE level logs with AWS CRT logs filtered out, written at `/tmp/s3torchconnector-logs`:
+```sh
+  export S3_TORCH_CONNECTOR_DEBUG_LOGS=trace,awscrt=off
+  export S3_TORCH_CONNECTOR_LOGS_DIR_PATH="/tmp/s3torchconnector-logs"
+```
+
+- Set up different levels for inner components:
+```sh
+  export S3_TORCH_CONNECTOR_DEBUG_LOGS=trace,mountpoint_s3_client=debug,awscrt=error
+```
+This will set the log level to TRACE by default, DEBUG for mountpoint-s3-client and ERROR for AWS CRT.
+
+For more examples please check the 
+[env_logger documentation](https://docs.rs/env_logger/latest/env_logger/#enabling-logging).

s3torchconnectorclient/Cargo.lock

@@ -19,8 +19,8 @@ built = "0.7"
 pyo3 = { version = "0.19.2" }
 pyo3-log = "0.8.3"
 futures = "0.3.28"
-mountpoint-s3-client = { version = "0.7.0", features = ["mock"] }
-mountpoint-s3-crt = "0.6.1"
+mountpoint-s3-client = { version = "0.8.0", features = ["mock"] }
+mountpoint-s3-crt = "0.6.2"
 log = "0.4.20"
 tracing = { version = "0.1.40", default-features = false, features = ["std", "log"] }
 tracing-subscriber = { version = "0.3.18", features = ["fmt", "env-filter"]}