[🐛 Bug]: Temporary firefox folder not removed upon KeyboardInterrupt

[Ozoniuss]

Description

If using the firefox profile, it will make a copy of your profile folder. Then, when instantiating a firefox profile, if you send an interrupt signal fast enough such that your driver is not fully initialized, this copy will remain. If your profile is large enough and you do this a number of times for various reasons, you may end up with very large copies that consume your available space (which is something that occured to me today with a 1.3GB profile).

Reproducible Code

I have a setup where I use the following code to set up a driver:

def init_driver(run_headless: bool) -> tuple[WebDriver, str]:
    try:
        # Set Firefox options to use the existing profile
        firefox_options = Options()

        if run_headless:
            firefox_options.add_argument("--headless")

        profile_path = os.getenv("OBLIO_FIREFOX_PROFILE_PATH")
        if profile_path not in [None, ""]:

            logger.debug("using profile path %s", profile_path)

            firefox_profile = FirefoxProfile(profile_path)
            firefox_options.profile = firefox_profile
            logger.debug(f"creating temp profile {firefox_profile._profile_dir}")

        
        driver = webdriver.Firefox(options=firefox_options)

        return driver

My main function looks something like this:

def main():
    # other code
    try:
        driver = init_driver(run_headless)
        logger.info("driver initialized")
        login(driver=driver)
        download_oblio_data_locally(
            driver=driver, download_directory=DOWNLOADS_DIRECTORY
        )

    except WebDriverException as e:
        logger.error(f"got webdriver error: {str(e)}")
    except KeyboardInterrupt:
        logger.info("exiting program")

    finally:
        logger.info("closing driver")
        driver.quit()

If I quit my program soon enough such that driver is not set, but late enough such that profile had been created, it will leave the copy dangling.

I tried to work around it by writing some code that tries to clean up that folder, but was wondering if this could be incorporated into the library code (e.g. catching it in the driver constructor code). If you're willing to make the change, I am open to make the contributions.

Debugging Logs

Not necessary. Check the full code here.

What version of Selenium are you currently using?

4.30.0

The following statements are true

• This applies to the most recent version of Selenium (we can't fix old versions)
• This hasn't already been reported (I searched and didn't find it)
• All information necessary to reproduce the issue has been provided above

Did this work for you before?

Not sure, this is my first time trying it

If yes, what version of Selenium did it work with?

No response

Operating System

Linux pop-os 6.12.10-76061203-generic #202412060638174015461722.04~b4b3ebc SMP PREEMPT_DYNAMIC Fri F x86_64 x86_64 x86_64 GNU/Linux

Selenium Language Binding

Python

Which browsers are you experiencing the issue with?

Firefox

Are you using Selenium Grid?

No

[titusfortner]

@Ozoniuss, thank you for creating this issue. We will troubleshoot it as soon as we can.

---

Info for maintainers

Triage this issue by using labels.

If information is missing, add a helpful comment and then J-issue-template label.

If the issue is a question, add the I-question label.

If the issue is valid but there is no time to troubleshoot it, consider adding the help wanted label.

If the issue requires changes or fixes from an external project (e.g., ChromeDriver, GeckoDriver, MSEdgeDriver, W3C), add the applicable G-* label, and it will provide the correct link and auto-close the issue.

After troubleshooting the issue, please add the R-awaiting answer label.

Thank you!

[titusfortner]

So a few things.

KeyboardInterrupt: I don't think this should be considered part of normal processing, and not something Selenium itself should handle. We would end up needing to wrap everything to catch the interruption and determine what should be cleaned up.

That said, that isn't the issue, because Selenium does not clean up the copied profile even when it is shut down normally. (not just in Python, but in all the bindings).

You can get profile_dir() from firefox_profile and make sure it is deleted atexit with shutil.rmtree or whatnot. I don't think Selenium should do that. @cgoldberg or @shbenzer feel free to chime in on this if you disagree or have an alternate suggestion.

[cgoldberg]

@titusfortner
That's correct.. We don't clean up profiles at all (whether copied or newly created).

We could probably clean it up when driver.quit() is called (or even in the Service class that will get destroyed even if quit() isn't explicitly called), but that's new functionality and would be different than how other bindings handle it.

@Ozoniuss
You can see by running this code:

from selenium.webdriver.firefox.firefox_profile import FirefoxProfile
for _ in range(100):
    FirefoxProfile()

Then check your temp directory :)

$ ls /tmp | grep tmp | wc -l
100

[Ozoniuss]

Thanks for the responses!

That's correct.. We don't clean up profiles at all (whether copied or newly created).

I see, I hadn't noticed it doesn't get cleaned up at all. I was able to verify this today. To me it would make sense if Selenium cleaned up temporary resources, but I understand this is a larger change scoped not only to the Python bidings.

KeyboardInterrupt: I don't think this should be considered part of normal processing, and not something Selenium itself should handle. We would end up needing to wrap everything to catch the interruption and determine what should be cleaned up.

I was thinking of an approach similar to what cgoldberg suggested, e.g. a finalizer, where the cleanup happens when the object is destroyed, not necessarily upon receiving a KeyboardInterrupt exception. This would work since the driver needs to reference the profile and also, would not require wrapping with try catches. In addition, it would work even if the driver itself is not created, when the FirefoxProfile object is destroyed (my original problem).

Furthermore, I can't think of a scenario where the copy would be useful unless it's not reused as a cache for a subsequent execution, which strengthens the argument for doing a cleanup at the end of the lifecycle. At the moment, the copy is not reused. (not suggesting an actual Python finalizer would be the best way of handling it, but at least something similar)

You can get profile_dir() from firefox_profile and make sure it is deleted atexit with shutil.rmtree or whatnot. I don't think Selenium should do that.

If keeping the current behaviour the same, would maybe extending the documentation to suggest cleaning up the temporary directory make sense, since it's exposed as path?

In any case, I have all I need to move forward from here, thanks!

[cgoldberg]

@Ozoniuss
I think a note in the docs about profiles not being cleaned up is appropriate. Probably here:

https://www.selenium.dev/documentation/webdriver/browsers/firefox/

The docs are generated from code here:

https://github.com/SeleniumHQ/seleniumhq.github.io

... If you would like to contribute a PR to update them. Otherwise, I can do it later today.

[Ozoniuss]

I'll make the PR later today or tomorrow, if I don't see it already there:)

[Ozoniuss]

SeleniumHQ/seleniumhq.github.io#2262

[github-actions]

This issue has been automatically locked since there has not been any recent activity since it was closed. Please open a new issue for related bugs.