Fix consecutive build failures

[flofriday]

Explanation

TL;DR: There are some build artifacts that are left behind in the cpython repo that might mess up future builds.

For example:

every-python install v3.12.0    # Works
every-python install v3.11.0    # Will fail (but works on its own)

Finding such cases can be difficult and from 3.12 till 3.15 I didn't find a single example but I also didn't try all commits. It seems that they are easier to trigger in earlier python versions <=3.4 but that's mostly a guess.

To clarify, it's not that every-python is unable to build 3.11.0 because it is, it just fails if 3.12.0 was build before.

My docker setup

For reproducability here is again my docker setup which failed before the change.

FROM ubuntu:22.04

ENV DEBIAN_FRONTEND=noninteractive

# Install build dependencies and curl for uv
RUN apt-get update && apt-get install -y build-essential gdb lcov pkg-config \
    libbz2-dev libffi-dev libgdbm-dev libgdbm-compat-dev liblzma-dev \
    libncurses5-dev libreadline6-dev libsqlite3-dev libssl-dev \
    lzma lzma-dev tk-dev uuid-dev zlib1g-dev libzstd-dev \
    inetutils-inetd curl git python2.7

# Link python2 as python
RUN ln -s /usr/bin/python2.7 /usr/bin/python

# Install uv
RUN curl -LsSf https://astral.sh/uv/install.sh | sh
ENV PATH="/root/.local/bin:$PATH"

WORKDIR /root/

COPY . . 

RUN uv run every-python install v3.12.0
RUN uv run -- every-python run v3.12.0 -- python --version

RUN uv run every-python install v3.11.0 --verbose
RUN uv run -- every-python run v3.11.0 -- python --version

CMD ["/bin/bash"]

I don't think for those two versions the x86 compatability mode is required but for completeness I want to state that I still used it: docker build --platform linux/amd64 -t every-python .

Implemented fix

The developer guide mentions to run make clean to resolve such problems.

You might need to run make clean before or after re-running configure in a particular build directory.

However at some point in time make clean also ran the ./configure script which then would end up running twice in our setup, making the builds a lot slower. So took inspiration from the footnotes instead and choose git clean -fdx.

A screenshot of the repo after every-python install v3.14.0 where make clean will execute the configure script (I cannot explain why).

Considerations

1) Compile Times

Since we are reseting the working dir before every build and therefore removing all object files I feared that consecutive builds might be a lot slower the object files can no longer be used as a cache. However, in my experience the configure step often dominated the build-time but this could also be because I'm running this on a beefy machine. Another point is that in general successive builds rarely were faster than the initial build pointing to the fact the the object-file cache doesn't work as well as I would have thought.

Anyway here are some compile times from my machine:

	first build (python3.14.1)	second build (python3.14.2)
before	94.39s	100.35s
after	101.64s	97.03s

As you can see they are all about the same time.

2) Alternative solutions

Instead of running the command automatically we could suggest running the commands manually when the build fails, or add it as an optional flag.

I saw that in the bisect command you use git reset --hard && git clean -fd to clean up but this didn't work for some of my usecases because object files caused the issue which are ignored in .gitignore and therefore won't be cleaned with the clean command. This could be fixed by applying git clean -fdx.

We could also spend more time figuring out why the build fails and develop a tailored solution that does the least required change, this however is less general and we might have to develop multiple such fixes for cleanups.

Thank you ✨

I spent way to much time trying to figure out to install every major release of python in a container (which is a niche project but still) before I discovered this tool and even with the time I spent triaging some bugs I'm still very happy I found it.

So finally with this fix and #6 I finally got all 3.x.0 releases (15 in total) installed in a single container 😊

[savannahostrowski]

Hey thanks for this! Happy to hear that you found the project useful 😄 !

Let's add a test for this too. I have CI that runs on all platforms so we can make sure this works on Windows as well (it should!).

[flofriday]

I incorporated your suggestions and added the example from the description above as a test for the CI. 😁

[savannahostrowski]

Oops, I think this got missed

#7 (comment)

[flofriday]

Oh noooooo 🙈

Yeah, i forgot to cleanup the expression after the colon 😅
Should be fixed now 😊

[savannahostrowski]

Thanks again for all the fixes!