Skip to content

Commit 5202792

Browse files
tausbntausbn
authored
Merge pull request #15754 from github/tausbn/python-copy-extractor-to-external-repo
Python: Copy Python extractor to `codeql` repo
2 parents b71b43a + 4b0689b commit 5202792

File tree

369 files changed

+165347
-0
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

369 files changed

+165347
-0
lines changed

‎python/extractor/BUILD.bazel

Lines changed: 49 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,49 @@
1+
load("//:dist.bzl", "pack_zip")
2+
3+
py_binary(
4+
name = "make-zips-py",
5+
srcs = [
6+
"make_zips.py",
7+
"python_tracer.py",
8+
"unparse.py",
9+
],
10+
data = [
11+
"LICENSE-PSF.md",
12+
"__main__.py",
13+
"imp.py",
14+
] + glob([
15+
"blib2to3/**",
16+
"buildtools/**",
17+
"lark/**",
18+
"semmle/**",
19+
]),
20+
# On @criemen's machine, without this, make-zips.py can't find its imports from
21+
# python_tracer. The problem didn't show for some reason on Windows CI machines, though.
22+
imports = ["."],
23+
main = "make_zips.py",
24+
)
25+
26+
genrule(
27+
name = "python3src",
28+
outs = [
29+
"python3src.zip",
30+
],
31+
cmd = "PYTHON_INSTALLER_OUTPUT=\"$(RULEDIR)\" $(location :make-zips-py)",
32+
tools = [":make-zips-py"],
33+
)
34+
35+
pack_zip(
36+
name = "extractor-python",
37+
srcs = [
38+
"LICENSE-PSF.md", # because we distribute imp.py
39+
"convert_setup.py",
40+
"get_venv_lib.py",
41+
"imp.py",
42+
"index.py",
43+
"python_tracer.py",
44+
"setup.py",
45+
":python3src",
46+
] + glob(["data/**"]),
47+
prefix = "tools",
48+
visibility = ["//visibility:public"],
49+
)

‎python/extractor/LICENSE-PSF.md

Lines changed: 257 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,257 @@
1+
Parts of the Python extractor are derived from code in the CPython
2+
distribution. Its license is reproduced below.
3+
4+
A. HISTORY OF THE SOFTWARE
5+
==========================
6+
7+
Python was created in the early 1990s by Guido van Rossum at Stichting
8+
Mathematisch Centrum (CWI, see http://www.cwi.nl) in the Netherlands
9+
as a successor of a language called ABC. Guido remains Python's
10+
principal author, although it includes many contributions from others.
11+
12+
In 1995, Guido continued his work on Python at the Corporation for
13+
National Research Initiatives (CNRI, see http://www.cnri.reston.va.us)
14+
in Reston, Virginia where he released several versions of the
15+
software.
16+
17+
In May 2000, Guido and the Python core development team moved to
18+
BeOpen.com to form the BeOpen PythonLabs team. In October of the same
19+
year, the PythonLabs team moved to Digital Creations, which became
20+
Zope Corporation. In 2001, the Python Software Foundation (PSF, see
21+
https://www.python.org/psf/) was formed, a non-profit organization
22+
created specifically to own Python-related Intellectual Property.
23+
Zope Corporation was a sponsoring member of the PSF.
24+
25+
All Python releases are Open Source (see http://www.opensource.org for
26+
the Open Source Definition). Historically, most, but not all, Python
27+
releases have also been GPL-compatible; the table below summarizes
28+
the various releases.
29+
30+
Release Derived Year Owner GPL-
31+
from compatible? (1)
32+
33+
0.9.0 thru 1.2 1991-1995 CWI yes
34+
1.3 thru 1.5.2 1.2 1995-1999 CNRI yes
35+
1.6 1.5.2 2000 CNRI no
36+
2.0 1.6 2000 BeOpen.com no
37+
1.6.1 1.6 2001 CNRI yes (2)
38+
2.1 2.0+1.6.1 2001 PSF no
39+
2.0.1 2.0+1.6.1 2001 PSF yes
40+
2.1.1 2.1+2.0.1 2001 PSF yes
41+
2.1.2 2.1.1 2002 PSF yes
42+
2.1.3 2.1.2 2002 PSF yes
43+
2.2 and above 2.1.1 2001-now PSF yes
44+
45+
Footnotes:
46+
47+
(1) GPL-compatible doesn't mean that we're distributing Python under
48+
the GPL. All Python licenses, unlike the GPL, let you distribute
49+
a modified version without making your changes open source. The
50+
GPL-compatible licenses make it possible to combine Python with
51+
other software that is released under the GPL; the others don't.
52+
53+
(2) According to Richard Stallman, 1.6.1 is not GPL-compatible,
54+
because its license has a choice of law clause. According to
55+
CNRI, however, Stallman's lawyer has told CNRI's lawyer that 1.6.1
56+
is "not incompatible" with the GPL.
57+
58+
Thanks to the many outside volunteers who have worked under Guido's
59+
direction to make these releases possible.
60+
61+
62+
B. TERMS AND CONDITIONS FOR ACCESSING OR OTHERWISE USING PYTHON
63+
===============================================================
64+
65+
PYTHON SOFTWARE FOUNDATION LICENSE VERSION 2
66+
--------------------------------------------
67+
68+
1. This LICENSE AGREEMENT is between the Python Software Foundation
69+
("PSF"), and the Individual or Organization ("Licensee") accessing and
70+
otherwise using this software ("Python") in source or binary form and
71+
its associated documentation.
72+
73+
2. Subject to the terms and conditions of this License Agreement, PSF hereby
74+
grants Licensee a nonexclusive, royalty-free, world-wide license to reproduce,
75+
analyze, test, perform and/or display publicly, prepare derivative works,
76+
distribute, and otherwise use Python alone or in any derivative version,
77+
provided, however, that PSF's License Agreement and PSF's notice of copyright,
78+
i.e., "Copyright (c) 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008, 2009, 2010,
79+
2011, 2012, 2013, 2014, 2015, 2016, 2017, 2018, 2019 Python Software Foundation;
80+
All Rights Reserved" are retained in Python alone or in any derivative version
81+
prepared by Licensee.
82+
83+
3. In the event Licensee prepares a derivative work that is based on
84+
or incorporates Python or any part thereof, and wants to make
85+
the derivative work available to others as provided herein, then
86+
Licensee hereby agrees to include in any such work a brief summary of
87+
the changes made to Python.
88+
89+
4. PSF is making Python available to Licensee on an "AS IS"
90+
basis. PSF MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
91+
IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, PSF MAKES NO AND
92+
DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS
93+
FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON WILL NOT
94+
INFRINGE ANY THIRD PARTY RIGHTS.
95+
96+
5. PSF SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON
97+
FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS
98+
A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON,
99+
OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
100+
101+
6. This License Agreement will automatically terminate upon a material
102+
breach of its terms and conditions.
103+
104+
7. Nothing in this License Agreement shall be deemed to create any
105+
relationship of agency, partnership, or joint venture between PSF and
106+
Licensee. This License Agreement does not grant permission to use PSF
107+
trademarks or trade name in a trademark sense to endorse or promote
108+
products or services of Licensee, or any third party.
109+
110+
8. By copying, installing or otherwise using Python, Licensee
111+
agrees to be bound by the terms and conditions of this License
112+
Agreement.
113+
114+
115+
BEOPEN.COM LICENSE AGREEMENT FOR PYTHON 2.0
116+
-------------------------------------------
117+
118+
BEOPEN PYTHON OPEN SOURCE LICENSE AGREEMENT VERSION 1
119+
120+
1. This LICENSE AGREEMENT is between BeOpen.com ("BeOpen"), having an
121+
office at 160 Saratoga Avenue, Santa Clara, CA 95051, and the
122+
Individual or Organization ("Licensee") accessing and otherwise using
123+
this software in source or binary form and its associated
124+
documentation ("the Software").
125+
126+
2. Subject to the terms and conditions of this BeOpen Python License
127+
Agreement, BeOpen hereby grants Licensee a non-exclusive,
128+
royalty-free, world-wide license to reproduce, analyze, test, perform
129+
and/or display publicly, prepare derivative works, distribute, and
130+
otherwise use the Software alone or in any derivative version,
131+
provided, however, that the BeOpen Python License is retained in the
132+
Software, alone or in any derivative version prepared by Licensee.
133+
134+
3. BeOpen is making the Software available to Licensee on an "AS IS"
135+
basis. BEOPEN MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
136+
IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, BEOPEN MAKES NO AND
137+
DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS
138+
FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF THE SOFTWARE WILL NOT
139+
INFRINGE ANY THIRD PARTY RIGHTS.
140+
141+
4. BEOPEN SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF THE
142+
SOFTWARE FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS
143+
AS A RESULT OF USING, MODIFYING OR DISTRIBUTING THE SOFTWARE, OR ANY
144+
DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
145+
146+
5. This License Agreement will automatically terminate upon a material
147+
breach of its terms and conditions.
148+
149+
6. This License Agreement shall be governed by and interpreted in all
150+
respects by the law of the State of California, excluding conflict of
151+
law provisions. Nothing in this License Agreement shall be deemed to
152+
create any relationship of agency, partnership, or joint venture
153+
between BeOpen and Licensee. This License Agreement does not grant
154+
permission to use BeOpen trademarks or trade names in a trademark
155+
sense to endorse or promote products or services of Licensee, or any
156+
third party. As an exception, the "BeOpen Python" logos available at
157+
http://www.pythonlabs.com/logos.html may be used according to the
158+
permissions granted on that web page.
159+
160+
7. By copying, installing or otherwise using the software, Licensee
161+
agrees to be bound by the terms and conditions of this License
162+
Agreement.
163+
164+
165+
CNRI LICENSE AGREEMENT FOR PYTHON 1.6.1
166+
---------------------------------------
167+
168+
1. This LICENSE AGREEMENT is between the Corporation for National
169+
Research Initiatives, having an office at 1895 Preston White Drive,
170+
Reston, VA 20191 ("CNRI"), and the Individual or Organization
171+
("Licensee") accessing and otherwise using Python 1.6.1 software in
172+
source or binary form and its associated documentation.
173+
174+
2. Subject to the terms and conditions of this License Agreement, CNRI
175+
hereby grants Licensee a nonexclusive, royalty-free, world-wide
176+
license to reproduce, analyze, test, perform and/or display publicly,
177+
prepare derivative works, distribute, and otherwise use Python 1.6.1
178+
alone or in any derivative version, provided, however, that CNRI's
179+
License Agreement and CNRI's notice of copyright, i.e., "Copyright (c)
180+
1995-2001 Corporation for National Research Initiatives; All Rights
181+
Reserved" are retained in Python 1.6.1 alone or in any derivative
182+
version prepared by Licensee. Alternately, in lieu of CNRI's License
183+
Agreement, Licensee may substitute the following text (omitting the
184+
quotes): "Python 1.6.1 is made available subject to the terms and
185+
conditions in CNRI's License Agreement. This Agreement together with
186+
Python 1.6.1 may be located on the Internet using the following
187+
unique, persistent identifier (known as a handle): 1895.22/1013. This
188+
Agreement may also be obtained from a proxy server on the Internet
189+
using the following URL: http://hdl.handle.net/1895.22/1013".
190+
191+
3. In the event Licensee prepares a derivative work that is based on
192+
or incorporates Python 1.6.1 or any part thereof, and wants to make
193+
the derivative work available to others as provided herein, then
194+
Licensee hereby agrees to include in any such work a brief summary of
195+
the changes made to Python 1.6.1.
196+
197+
4. CNRI is making Python 1.6.1 available to Licensee on an "AS IS"
198+
basis. CNRI MAKES NO REPRESENTATIONS OR WARRANTIES, EXPRESS OR
199+
IMPLIED. BY WAY OF EXAMPLE, BUT NOT LIMITATION, CNRI MAKES NO AND
200+
DISCLAIMS ANY REPRESENTATION OR WARRANTY OF MERCHANTABILITY OR FITNESS
201+
FOR ANY PARTICULAR PURPOSE OR THAT THE USE OF PYTHON 1.6.1 WILL NOT
202+
INFRINGE ANY THIRD PARTY RIGHTS.
203+
204+
5. CNRI SHALL NOT BE LIABLE TO LICENSEE OR ANY OTHER USERS OF PYTHON
205+
1.6.1 FOR ANY INCIDENTAL, SPECIAL, OR CONSEQUENTIAL DAMAGES OR LOSS AS
206+
A RESULT OF MODIFYING, DISTRIBUTING, OR OTHERWISE USING PYTHON 1.6.1,
207+
OR ANY DERIVATIVE THEREOF, EVEN IF ADVISED OF THE POSSIBILITY THEREOF.
208+
209+
6. This License Agreement will automatically terminate upon a material
210+
breach of its terms and conditions.
211+
212+
7. This License Agreement shall be governed by the federal
213+
intellectual property law of the United States, including without
214+
limitation the federal copyright law, and, to the extent such
215+
U.S. federal law does not apply, by the law of the Commonwealth of
216+
Virginia, excluding Virginia's conflict of law provisions.
217+
Notwithstanding the foregoing, with regard to derivative works based
218+
on Python 1.6.1 that incorporate non-separable material that was
219+
previously distributed under the GNU General Public License (GPL), the
220+
law of the Commonwealth of Virginia shall govern this License
221+
Agreement only as to issues arising under or with respect to
222+
Paragraphs 4, 5, and 7 of this License Agreement. Nothing in this
223+
License Agreement shall be deemed to create any relationship of
224+
agency, partnership, or joint venture between CNRI and Licensee. This
225+
License Agreement does not grant permission to use CNRI trademarks or
226+
trade name in a trademark sense to endorse or promote products or
227+
services of Licensee, or any third party.
228+
229+
8. By clicking on the "ACCEPT" button where indicated, or by copying,
230+
installing or otherwise using Python 1.6.1, Licensee agrees to be
231+
bound by the terms and conditions of this License Agreement.
232+
233+
ACCEPT
234+
235+
236+
CWI LICENSE AGREEMENT FOR PYTHON 0.9.0 THROUGH 1.2
237+
--------------------------------------------------
238+
239+
Copyright (c) 1991 - 1995, Stichting Mathematisch Centrum Amsterdam,
240+
The Netherlands. All rights reserved.
241+
242+
Permission to use, copy, modify, and distribute this software and its
243+
documentation for any purpose and without fee is hereby granted,
244+
provided that the above copyright notice appear in all copies and that
245+
both that copyright notice and this permission notice appear in
246+
supporting documentation, and that the name of Stichting Mathematisch
247+
Centrum or CWI not be used in advertising or publicity pertaining to
248+
distribution of the software without specific, written prior
249+
permission.
250+
251+
STICHTING MATHEMATISCH CENTRUM DISCLAIMS ALL WARRANTIES WITH REGARD TO
252+
THIS SOFTWARE, INCLUDING ALL IMPLIED WARRANTIES OF MERCHANTABILITY AND
253+
FITNESS, IN NO EVENT SHALL STICHTING MATHEMATISCH CENTRUM BE LIABLE
254+
FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
255+
WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN
256+
ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT
257+
OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.

‎python/extractor/Makefile

Lines changed: 61 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,61 @@
1+
.PHONY: all
2+
.DEFAULT: all
3+
all:
4+
5+
OS = $(shell uname)
6+
7+
GIT_ROOT = $(shell git rev-parse --show-toplevel)
8+
9+
TOKENIZER_FILE = semmle/python/parser/tokenizer.py
10+
TOKENIZER_DEPS = tokenizer_generator/state_transition.txt tokenizer_generator/tokenizer_template.py
11+
# Must use the same Python version as on jenkins, since output differs per version.
12+
# However, output is unstable on Python 3.5 (which jenkins uses)
13+
TOKENIZER_CMD = python3 -m tokenizer_generator.gen_state_machine $(TOKENIZER_DEPS)
14+
15+
.PHONY: tokenizer
16+
tokenizer: $(TOKENIZER_FILE)
17+
18+
$(TOKENIZER_FILE): $(TOKENIZER_DEPS)
19+
$(TOKENIZER_CMD) > $@
20+
21+
22+
MASTER_FILE = semmle/python/master.py
23+
24+
DBSCHEME_FILE = $(GIT_ROOT)/ql/python/ql/lib/semmlecode.python.dbscheme
25+
26+
.PHONY: dbscheme
27+
dbscheme: $(MASTER_FILE)
28+
python3 -m semmle.dbscheme_gen $(DBSCHEME_FILE)
29+
30+
AST_GENERATED_DIR = $(GIT_ROOT)/ql/python/ql/lib/semmle/python/
31+
AST_GENERATED_FILE = $(AST_GENERATED_DIR)AstGenerated.qll
32+
33+
.PHONY: ast
34+
ast: $(MASTER_FILE)
35+
python3 -m semmle.query_gen $(AST_GENERATED_DIR)
36+
$(GIT_ROOT)/target/intree/codeql/codeql query format --in-place $(AST_GENERATED_FILE)
37+
38+
################################################################################
39+
# Tests
40+
################################################################################
41+
42+
.PHONY: test-all
43+
test-all: test-3
44+
45+
.PHONY: test-3
46+
test-3: pytest-3 test-tokenizer
47+
48+
.PHONY: test-tokenizer
49+
test-tokenizer: SHELL:=/bin/bash
50+
test-tokenizer:
51+
@echo Not running test-tokenizer as jenkins uses Python 3.5
52+
# TODO: Enable again once we run Python > 3.5 on Jenkins
53+
# diff -u $(TOKENIZER_FILE) <($(TOKENIZER_CMD))
54+
55+
.PHONY: pytest-3
56+
pytest-3:
57+
poetry run pytest
58+
59+
.PHONY: pytest-3-deprecation-error
60+
pytest-3-deprecation-error:
61+
PYTHONWARNINGS='error::DeprecationWarning' poetry run pytest

0 commit comments

Comments
 (0)