Cross compiled python installed wrong version of lib2to3/Grammar pickle

[andrew-aladev]

BPO	39221
Nosy	@doko42, @andrew-aladev
Files	remove_python_version_from_pickle.patch remove_python_version_from_pickle.patch: patch v2

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields:

assignee = None
closed_at = None
created_at = <Date 2020-01-05.13:41:23.189>
labels = ['3.7', '3.8', 'type-feature', 'library', '3.9']
title = 'Cross compiled python installed wrong version of lib2to3/Grammar pickle'
updated_at = <Date 2021-07-17.00:34:02.371>
user = 'https://github.com/andrew-aladev'

bugs.python.org fields:

activity = <Date 2021-07-17.00:34:02.371>
actor = 'puchenyaka'
assignee = 'none'
closed = False
closed_date = None
closer = None
components = ['Library (Lib)']
creation = <Date 2020-01-05.13:41:23.189>
creator = 'puchenyaka'
dependencies = []
files = ['48828', '50153']
hgrepos = []
issue_num = 39221
keywords = ['patch']
message_count = 4.0
messages = ['359343', '359344', '359375', '397677']
nosy_count = 2.0
nosy_names = ['doko', 'puchenyaka']
pr_nums = []
priority = 'normal'
resolution = None
stage = None
status = 'open'
superseder = None
type = 'enhancement'
url = 'https://bugs.python.org/issue39221'
versions = ['Python 3.7', 'Python 3.8', 'Python 3.9']

[andrew-aladev]

Please see the following gentoo bug https://bugs.gentoo.org/704816

https://github.com/python/cpython/blob/master/Lib/lib2to3/pgen2/driver.py#L110

head + tail + ".".join(map(str, sys.version_info)) + ".pickle"

I've tried "print(sys.version_info)" during compilation and received:

sys.version_info(major=3, minor=6, micro=9, releaselevel='final', serial=0)

"sys.version_info" is not the target python version, this is the version of python that is running compilation. This variable needs to be replace with something like "sys.target_python_version".

This issue looks simple but I can't fix it by myself. Please assign this issue to core developer. We need to find all places where "sys.version_info" is used as target python version during compilation and replace it.

[andrew-aladev]

This is build.log:

https://gist.githubusercontent.com/andrew-aladev/f36e8e754278e4fad50ff13238326f7a/raw/245a72e8bd0fb3dc6e4185ec757b2d3ab59d5861/gistfile1.txt

We can see that python installed "Grammar3.6.9.final.0.pickle" instead of "Grammar3.6.10.final.0.pickle"

[andrew-aladev]

For now I've created a workaround - just removed python version from pickle generator. For my current container it works perfect.

[andrew-aladev]

Hello. I've received more problems with pickle generation, reviewed source code, invented better bike for solving it. So I can provide full description of this issue and reduce other developers time on debugging.

Problem:

1. Martin v. Löwis <martin@v.loewis.de> introduced pickle generation Wed Mar 19 04:43:46 2008 commit 5e37bae#diff-c9bd4064884726c21716df7118dcaeb72a63cc8ddd49484c7f4676934b37d9bb

2. Martin provided code for generating of pickle file: head + tail + ".".join(map(str, sys.version_info)) + ".pickle". This code has never been changed and comes as is in 2021 year.

Code means that current python interpreter uses its own version for creating a new pickle file for new cpython interpreter. It will work only when current python interpreter equals to new cpython interpreter. This code is broken by design.

3. We can see file "Lib/lib2to3/patcomp.py" in the commit above, today (2021) it is a part of "Lib/lib2to3/pygram.py". It comes with the following code:

_GRAMMAR_FILE = ... "Grammar.txt"
_PATTERN_GRAMMAR_FILE = ... "PatternGrammar.txt"

4. We can review step-by-step "load_grammar" real world usage from "Lib/lib2to3/pgen2/driver.py":

driver.load_grammar(_GRAMMAR_FILE)
driver.load_grammar(_PATTERN_GRAMMAR_FILE)

def load_grammar(gt="Grammar.txt", gp=None
gp = head + tail + ".".join(map(str, sys.version_info)) + ".pickle"

if force or not _newer(gp, gt):
  logger.info("Generating grammar tables from %s", gt)

def _newer(a, b):
  if not os.path.exists(a):
    return False

This "code" comes as is today as a part of latest stable python 3.9.5. This workaround means that when pickle files doesn't exist - python will recreate it in runtime inside "lib/lib2to3" folder.

This workaround will ruin your production if "lib/lib2to3" is readonly and pickle files are not inside. Everybody will try to use this workaround as a security hole. But it looks like nobody from cpython cares.

Solution:

1. Apply provided remove_python_version_from_pickle.patch.
2. Cross compile new cpython.
3. Cross compile portage.
4. Chroot.
5. Run python -c "import lib2to3.pygram", it will create pickles without version postfix, stop python from creating junk in runtime, fix sandbox permission issues temporaly.
6. Reinstall portage.
7. Reinstall cpython without patch, it will recreate pickles during installation.
8. Reinstall portage.

This is just bike on the top of Martin's workaround. The best variant is to ignore cpython (and maybe python) completely, ignore software written in python where possible.