Skip to content

Do not start any new threads during shutdown#211

Merged
kislyuk merged 1 commit intokislyuk:mainfrom
tor-gu:handle-msg-during-shutdown
Nov 24, 2025
Merged

Do not start any new threads during shutdown#211
kislyuk merged 1 commit intokislyuk:mainfrom
tor-gu:handle-msg-during-shutdown

Conversation

@tor-gu
Copy link
Contributor

@tor-gu tor-gu commented Nov 23, 2025

This is a fix for #193

I was only able to reproduce the issue in an AWS Glue container. Here is a minimal example.

# gluejob.py
import watchtower
import logging
from awsglue.context import GlueContext
from pyspark.context import SparkContext


if __name__ == '__main__':
    # Set up just enough of Glue to trigger the issue
    sc = SparkContext()
    glueContext = GlueContext(sc)

    # Add a watchtower handler to the root logger
    # Make sure the log_stream depends on the logger_name
    logging.basicConfig(level="INFO")
    logger = logging.getLogger()
    handler = watchtower.CloudWatchLogHandler(
        log_group_name="/foo", 
        log_stream_name="stream_{logger_name}"  # the default log_stream would also work
    )
    logger.addHandler(handler)

    logger.info("Exiting...")
SCRIPT_FILE_NAME=gluejob.py

# Install watchtower and submit the glue job
docker run -it --rm \
    -v ~/.aws:/home/hadoop/.aws \
    -v $PWD:/home/hadoop/workspace/ \
    -e AWS_PROFILE=$AWS_PROFILE \
    --name glue_watchtower \
    public.ecr.aws/glue/aws-glue-libs:5 \
    -c "pip3.11 install watchtower==3.4.0; spark-submit /home/hadoop/workspace/$SCRIPT_FILE_NAME"

On my machine, it hangs with this message most (but not every!) time. There are evidently some race conditions as things are deallocated.

INFO:py4j.clientserver:Closing down clientserver connection

@kislyuk kislyuk merged commit ba5133e into kislyuk:main Nov 24, 2025
2 of 14 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

Comments