Skip to content

Commit f5ef364

Browse files
timfeltimfel
committed
[GR-30033] [GR-30257] Support hash namespace.
PullRequest: graalpython/1701
2 parents 80a88bb + 4c047e6 commit f5ef364

File tree

16 files changed

+624
-668
lines changed

16 files changed

+624
-668
lines changed

‎CHANGELOG.md

Lines changed: 4 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -3,6 +3,10 @@
33
This changelog summarizes major changes between GraalVM versions of the Python
44
language runtime. The main focus is on user-observable behavior of the engine.
55

6+
## Version 21.2.0
7+
8+
* Support the `dict` type properly in interop using the new hash interop messages.
9+
610
## Version 21.1.0
711

812
* Support multi-threading with a global interpreter lock by default.

‎docs/user/Interoperability.md

Lines changed: 58 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,7 @@
11
# Interoperability
22

3+
## The Polyglot API
4+
35
Since GraalVM supports several other programming languages including JavaScript, R,
46
Ruby, and those that compile to LLVM bitcode, it also provides a Python API to interact with them.
57
In fact, GraalVM uses this API internally to execute Python C extensions using the GraalVM LLVM runtime.
@@ -123,7 +125,7 @@ draw(result)
123125
time.sleep(10)
124126
```
125127

126-
## Java Interoperability
128+
## The Java Host Interop API
127129

128130
Finally, to interoperate with Java (only when running on the JVM), you can use the `java` module:
129131
```python
@@ -181,3 +183,58 @@ print(java.instanceof(my_list, ArrayList))
181183
```
182184

183185
See [Polyglot Programming](https://www.graalvm.org/docs/reference-manual/polyglot-programming/) and [Embed Languages](https://www.graalvm.org/reference-manual/embed-languages/) for more information about interoperability with other programming languages.
186+
187+
## The Behaviour of Types
188+
189+
The interop protocol defines different "types" which can overlap in all kinds of
190+
ways and have restrictions on how they can interact with Python.
191+
192+
### Interop Types to Python
193+
194+
Most importantly and upfront - all foreign objects passing into Python have the
195+
Python type `foreign`. There is no emulation of i.e., objects that are interop
196+
booleans to have the Python type `bool`. This is because interop types can
197+
overlap in ways that the Python builtin types cannot, and it would not be clear
198+
what should take precendence. Instead, the `foreign` type defines all of the
199+
Python special methods for type conversion that are used throughout the
200+
interpreter (methods like `__add__`, `__int__`, `__str__`, `__getitem__` etc)
201+
and these try to do the right thing based on the interop type (or raise an
202+
exception.)
203+
204+
Types not listed in the below table have no special interpretation in Python
205+
right now.
206+
207+
| Interop type | Python interpretation |
208+
|:-------------|:---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
209+
| Null | It is like None. Important to know: interop null values are equal, but not identical! This was done because JavaScript defines two "null-like" values; `undefined` and `null`, which are *not* identical |
210+
| Boolean | Behaves like Python booleans, including the fact that in Python, all booleans are also integers (1 and 0 for true and false, respectively) |
211+
| Number | Behaves like Python numbers. Python only has one integral and one floating point type, but it cares about the ranges in some places such as typed arrays. |
212+
| String | Behaves like Python strings. |
213+
| Buffer | Buffers are also a concept in Python's native API (albeit a bit different). Interop buffers are treated like Python buffers in some places (like `memoryview`) to avoid copies of data. |
214+
| Array | Arrays can be used with subscript access like Python lists, with integers and slices as indices. |
215+
| Hash | Hashes can be used with subscript access like Python dicts, with any hashable kind of object as key. "Hashable" follows Python semantics, generally all interop types with identity are deemed "hashable". Note that if an interop object is both Array and Hash, the behavior of the subscript access is undefined. |
216+
| Members | Members can be read using normal Python ~.~ notation or the `getattr` etc functions. |
217+
| Iterable | Iterables are treated like Python objects with an `__iter__` method, that is, they can be used in loops and other places that accept Python iterables. |
218+
| Iterator | Iterators are treated like Python objects with a `__next__` method. |
219+
| Exception | Interop exceptions can be caught in generic except clauses. |
220+
| MetaObject | Interop meta objects can be used in subtype and isinstance checks |
221+
| Executable | Executable objects can be executed as functions, but never with keyword arguments. |
222+
| Instantiable | Instantiable objects behave like executable objects (similar to how Python treats this) |
223+
224+
### Python to Interop Types
225+
226+
| Interop type | Python interpretation |
227+
|:-------------|:----------------------------------------------------------------------------------------------------------------------------------|
228+
| Null | Only `None`. |
229+
| Boolean | Only subtypes of Python `bool`. Note that in contrast to Python semantics, Python `bool` is *never* also an interop number. |
230+
| Number | Only subtypes of `int` and `float`. |
231+
| String | Only subtypes of `str`. |
232+
| Array | Any object with a `__getitem__` and a `__len__`, but not if it also has `keys`, `values`, and `items` (like `dict` does.) |
233+
| Hash | Only subtypes of `dict`. |
234+
| Members | Any Python object. Note that the rules for readable/writable are a bit ad-hoc, since checking that is not part of the Python MOP. |
235+
| Iterable | Anything that has an `__iter__` method or a `__getitem__` method. |
236+
| Iterator | Anything with a `__next__` method. |
237+
| Exception | Any Python `BaseException` subtype. |
238+
| MetaObject | Any Python `type`. |
239+
| Executable | Anything with a `__call__` method. |
240+
| Instantiable | Any Python `type`. |

‎graalpython/com.oracle.graal.python.tck/src/com/oracle/graal/python/tck/PythonProvider.java

Lines changed: 69 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -54,6 +54,7 @@
5454
import static org.graalvm.polyglot.tck.TypeDescriptor.TIME;
5555
import static org.graalvm.polyglot.tck.TypeDescriptor.TIME_ZONE;
5656
import static org.graalvm.polyglot.tck.TypeDescriptor.array;
57+
import static org.graalvm.polyglot.tck.TypeDescriptor.hash;
5758
import static org.graalvm.polyglot.tck.TypeDescriptor.iterable;
5859
import static org.graalvm.polyglot.tck.TypeDescriptor.iterator;
5960
import static org.graalvm.polyglot.tck.TypeDescriptor.executable;
@@ -116,7 +117,7 @@ private static final TypeDescriptor tuple(TypeDescriptor componentType) {
116117
}
117118

118119
private static final TypeDescriptor dict(TypeDescriptor keyType, @SuppressWarnings("unused") TypeDescriptor valueType) {
119-
return intersection(OBJECT, iterable(keyType));
120+
return intersection(OBJECT, iterable(keyType), hash(keyType, valueType));
120121
}
121122

122123
private static final TypeDescriptor set(TypeDescriptor componentType) {
@@ -231,7 +232,7 @@ public Collection<? extends Snippet> createExpressions(Context context) {
231232

232233
addExpressionSnippet(context, snippets, "/", "lambda x, y: x / y", NUMBER, PDivByZeroVerifier.INSTANCE, union(BOOLEAN, NUMBER), union(BOOLEAN, NUMBER));
233234

234-
// addExpressionSnippet(context, snippets, "list-from-foreign", "lambda x: list(x)", array(ANY), union(STRING, iterable(ANY), iterator(ANY), array(ANY)));
235+
addExpressionSnippet(context, snippets, "list-from-foreign", "lambda x: list(x)", array(ANY), union(STRING, iterable(ANY), iterator(ANY), array(ANY), hash(ANY, ANY)));
235236

236237
addExpressionSnippet(context, snippets, "==", "lambda x, y: x == y", BOOLEAN, ANY, ANY);
237238
addExpressionSnippet(context, snippets, "!=", "lambda x, y: x != y", BOOLEAN, ANY, ANY);
@@ -242,6 +243,10 @@ public Collection<? extends Snippet> createExpressions(Context context) {
242243

243244
addExpressionSnippet(context, snippets, "isinstance", "lambda x, y: isinstance(x, y)", BOOLEAN, ANY, META_OBJECT);
244245
addExpressionSnippet(context, snippets, "issubclass", "lambda x, y: issubclass(x, y)", BOOLEAN, META_OBJECT, META_OBJECT);
246+
247+
addExpressionSnippet(context, snippets, "[]", "lambda x, y: x[y]", ANY, GetItemVerifier.INSTANCE, union(array(ANY), STRING, hash(ANY, ANY)), ANY);
248+
addExpressionSnippet(context, snippets, "[a:b]", "lambda x: x[:]", union(STRING, array(ANY)), union(STRING, array(ANY)));
249+
245250
// @formatter:on
246251
return snippets;
247252
}
@@ -266,10 +271,10 @@ public Collection<? extends Snippet> createStatements(Context context) {
266271
" return False\n\n" +
267272
"gen_if", BOOLEAN, ANY);
268273

269-
// addStatementSnippet(context, snippets, "for", "def gen_for(l):\n" +
270-
// " for x in l:\n" +
271-
// " return x\n\n" +
272-
// "gen_for", ANY, union(array(ANY), iterable(ANY), iterator(ANY), STRING));
274+
addStatementSnippet(context, snippets, "for", "def gen_for(l):\n" +
275+
" for x in l:\n" +
276+
" return x\n\n" +
277+
"gen_for", ANY, union(array(ANY), iterable(ANY), iterator(ANY), STRING, hash(ANY, ANY)));
273278

274279
// any exception honours the finally block, but non-exception cannot be raised
275280
addStatementSnippet(context, snippets, "try-finally", "def gen_tryfinally(exc):\n" +
@@ -440,6 +445,64 @@ public void accept(SnippetRun snippetRun) throws PolyglotException {
440445
private static final MulVerifier INSTANCE = new MulVerifier();
441446
}
442447

448+
private static class GetItemVerifier extends PResultVerifier {
449+
private static final String[] UNHASHABLE_TYPES = new String[]{"list", "dict", "bytearray", "set"};
450+
451+
public void accept(SnippetRun snippetRun) throws PolyglotException {
452+
List<? extends Value> parameters = snippetRun.getParameters();
453+
assert parameters.size() == 2;
454+
455+
Value par0 = parameters.get(0);
456+
Value par1 = parameters.get(1);
457+
458+
long len = -1;
459+
460+
if (par0.hasArrayElements()) {
461+
len = par0.getArraySize();
462+
} else if (par0.isString()) {
463+
len = par0.asString().length();
464+
}
465+
if (len >= 0) {
466+
int idx;
467+
if (par1.isBoolean()) {
468+
idx = par1.asBoolean() ? 1 : 0;
469+
} else if (par1.isNumber() && par1.fitsInInt()) {
470+
idx = par1.asInt();
471+
} else {
472+
assert snippetRun.getException() != null;
473+
return;
474+
}
475+
if ((idx >= 0 && len > idx) || (len - idx >= 0 && len > len - idx)) {
476+
assert snippetRun.getException() == null : snippetRun.getException().toString();
477+
} else {
478+
assert snippetRun.getException() != null;
479+
}
480+
} else if (par0.hasHashEntries()) {
481+
if (par1.getMetaObject() != null) {
482+
String metaName = par1.getMetaObject().getMetaQualifiedName();
483+
for (String s : UNHASHABLE_TYPES) {
484+
if (metaName.equals(s)) {
485+
// those don't work, but that's expected
486+
assert snippetRun.getException() != null;
487+
return;
488+
}
489+
}
490+
}
491+
Value v = par0.getHashValueOrDefault(par1, PythonProvider.class.getName());
492+
if (v.isString() && v.asString().equals(PythonProvider.class.getName())) {
493+
assert snippetRun.getException() != null;
494+
} else {
495+
assert snippetRun.getException() == null : snippetRun.getException().toString();
496+
}
497+
} else {
498+
// argument type error, rethrow
499+
throw snippetRun.getException();
500+
}
501+
}
502+
503+
private static final GetItemVerifier INSTANCE = new GetItemVerifier();
504+
}
505+
443506
/**
444507
* Only accepts exact matches of types.
445508
*/

‎graalpython/com.oracle.graal.python.test/src/com/oracle/graal/python/test/interop/JavaInteropTest.java

Lines changed: 4 additions & 10 deletions
Original file line numberDiff line numberDiff line change
@@ -50,7 +50,6 @@
5050
import java.io.IOException;
5151
import java.io.UnsupportedEncodingException;
5252
import java.util.Arrays;
53-
import java.util.List;
5453

5554
import org.graalvm.polyglot.Context;
5655
import org.graalvm.polyglot.Context.Builder;
@@ -300,22 +299,17 @@ public void accessSuitePy() throws IOException {
300299
"}",
301300
"suite.py").build();
302301
Value suite = context.eval(suitePy);
303-
304-
Value listConverter = context.eval("python", "list");
305-
Value libraries = suite.getMember("libraries");
302+
Value libraries = suite.getHashValue("libraries");
306303
assertNotNull("libraries found", libraries);
307-
final List<Object> suiteKeys = Arrays.asList(listConverter.execute(suite.invokeMember("keys")).as(Object[].class));
308-
assertTrue("Libraries found among keys: " + suiteKeys, suiteKeys.contains("libraries"));
309-
310304
Value dacapo = null;
311-
for (Object k : listConverter.execute(libraries.invokeMember("keys")).as(List.class)) {
305+
for (Object k : libraries.getHashKeysIterator().as(Iterable.class)) {
312306
System.err.println("k " + k);
313307
if ("DACAPO".equals(k)) {
314-
dacapo = libraries.getMember((String) k);
308+
dacapo = libraries.getHashValue(k);
315309
}
316310
}
317311
assertNotNull("Dacapo found", dacapo);
318-
assertEquals("'e39957904b7e79caf4fa54f30e8e4ee74d4e9e37'", dacapo.getMember("sha1").toString());
312+
assertEquals("'e39957904b7e79caf4fa54f30e8e4ee74d4e9e37'", dacapo.getHashValue("sha1").toString());
319313
}
320314

321315
@ExportLibrary(InteropLibrary.class)

0 commit comments

Comments
 (0)