Skip to content

Commit 03a7676

Browse files
bioballstackoverflow
bioball
and
stackoverflow
authored
Implement power assertions (#1384)
This adds power assertions to Pkl! This implements the SPICE described in apple/pkl-evolution#29 This follows the power assertions style of reporting also found in Groovy, Kotlin, and others. * Literal values are not emitted in the diagram * Stdlib constructors of literals like `List(1, 2)` are also considered literals Power assertions are added to: * Failing type constraints * Failing test facts Power assertions are implemented as a truffle instrument to observe execution. When an assertion fails, the instrument is created and the assertion is run again to observe facts. This incurs runtime overhead to collect facts, but has no impact on code in the non-error case. --------- Co-authored-by: Islon Scherer <[email protected]>
1 parent 3cd294b commit 03a7676

File tree

107 files changed

+2133
-290
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

107 files changed

+2133
-290
lines changed

‎docs/modules/release-notes/pages/0.31.adoc‎

Lines changed: 83 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -20,7 +20,89 @@ To get started, follow xref:pkl-cli:index.adoc#installation[Installation].#
2020

2121
News you don't want to miss.
2222

23-
=== XXX
23+
=== Power Assertions
24+
25+
Pkl's test output and error output has been improved with power assertions (https://github.com/apple/pkl/pull/1384[#1384])!
26+
27+
Pkl has two places that are effectively assertions.
28+
These are:
29+
30+
* Type constraint expressions
31+
* Test facts
32+
33+
Currently, when these assertions fail, the error message prints the assertion's source section.
34+
However, this information can sometimes be lacking.
35+
It tells you what the mechanics of the assertion is, but doesn't tell you _why_ the assertion is failing.
36+
37+
For example, here is the current error output of a failing typecheck.
38+
39+
[source,text]
40+
----
41+
–– Pkl Error ––
42+
Type constraint `name.endsWith(lastName)` violated.
43+
Value: new Person { name = "Bub Johnson" }
44+
45+
7 | passenger: Person(name.endsWith(lastName)) = new { name = "Bub Johnson" }
46+
----
47+
48+
Just from this error message, we don't know what the last name is supposed to be.
49+
What is `name` supposed to end with?
50+
We just know it's some property called `lastName` but, we don't know what it _is_.
51+
52+
In Pkl 0.31, the error output becomes:
53+
54+
[source,text]
55+
----
56+
–– Pkl Error ––
57+
Type constraint `name.endsWith(lastName)` violated.
58+
Value: new Person { name = "Bub Johnson" }
59+
60+
name.endsWith(lastName)
61+
│ │ │
62+
│ false "Smith"
63+
"Bub Johnson"
64+
65+
7 | passenger: Person(name.endsWith(lastName)) = new { name = "Bub Johnson" }
66+
----
67+
68+
Now, we know what the expecation actually is.
69+
70+
This type of diagram is also added to test facts.
71+
When tests fail, Pkl emits a diagram of the expression, and the values produced.
72+
73+
For example, given the following test:
74+
75+
.math.pkl
76+
[source,pkl]
77+
----
78+
amends "pkl:test"
79+
80+
facts {
81+
local function add(a: Int, b: Int) = a * b
82+
local function divide(a: Int, b: Int) = a % b
83+
["math"] {
84+
add(3, 4) == 7
85+
divide(8, 2) == 4
86+
}
87+
}
88+
----
89+
90+
The error output now includes a power assertion diagram, which helps explain why the test failed.
91+
92+
[source,text]
93+
----
94+
module math
95+
facts
96+
✘ math
97+
add(3, 4) == 7 (math.pkl:9)
98+
│ │
99+
12 false
100+
divide(8, 2) == 4 (math.pkl:10)
101+
│ │
102+
0 false
103+
104+
0.0% tests pass [1/1 failed], 0.0% asserts pass [2/2 failed]
105+
----
24106

25107
== Noteworthy [small]#🎶#
26108

‎pkl-cli/src/test/kotlin/org/pkl/cli/CliTestRunnerTest.kt‎

Lines changed: 36 additions & 25 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright © 2024-2025 Apple Inc. and the Pkl project authors. All rights reserved.
2+
* Copyright © 2024-2026 Apple Inc. and the Pkl project authors. All rights reserved.
33
*
44
* Licensed under the Apache License, Version 2.0 (the "License");
55
* you may not use this file except in compliance with the License.
@@ -81,7 +81,7 @@ class CliTestRunnerTest {
8181
facts {
8282
["fail"] {
8383
4 == 9
84-
"foo" != "bar"
84+
1 == 5
8585
}
8686
}
8787
"""
@@ -101,8 +101,13 @@ class CliTestRunnerTest {
101101
facts
102102
✘ fail
103103
4 == 9 (/tempDir/test.pkl, line xx)
104+
│
105+
false
106+
1 == 5 (/tempDir/test.pkl, line xx)
107+
│
108+
false
104109
105-
0.0% tests pass [1/1 failed], 50.0% asserts pass [1/2 failed]
110+
0.0% tests pass [1/1 failed], 0.0% asserts pass [2/2 failed]
106111
107112
"""
108113
.trimIndent()
@@ -283,12 +288,14 @@ class CliTestRunnerTest {
283288
<testsuite name="test" tests="2" failures="1">
284289
<testcase classname="test.facts" name="foo"></testcase>
285290
<testcase classname="test.facts" name="bar">
286-
<failure message="Fact Failure">5 == 9 (/tempDir/test.pkl, line xx)</failure>
291+
<failure message="Fact Failure">5 == 9 (/tempDir/test.pkl, line xx)
292+
│
293+
false</failure>
287294
</testcase>
288295
<system-err><![CDATA[9 = 9
289296
]]></system-err>
290297
</testsuite>
291-
298+
292299
"""
293300
.trimIndent()
294301
)
@@ -481,26 +488,30 @@ class CliTestRunnerTest {
481488
assertThat(junitReport)
482489
.isEqualTo(
483490
"""
484-
<?xml version="1.0" encoding="UTF-8"?>
485-
<testsuites name="pkl-tests" tests="5" failures="3">
486-
<testsuite name="test1" tests="2" failures="1">
487-
<testcase classname="test1.facts" name="foo"></testcase>
488-
<testcase classname="test1.facts" name="bar">
489-
<failure message="Fact Failure">5 == 9 (/tempDir/test1.pkl, line xx)</failure>
490-
</testcase>
491-
</testsuite>
492-
<testsuite name="test2" tests="3" failures="2">
493-
<testcase classname="test2.facts" name="xxx">
494-
<failure message="Fact Failure">false (/tempDir/test2.pkl, line xx)</failure>
495-
</testcase>
496-
<testcase classname="test2.facts" name="yyy">
497-
<failure message="Fact Failure">false (/tempDir/test2.pkl, line xx)</failure>
498-
</testcase>
499-
<testcase classname="test2.facts" name="zzz"></testcase>
500-
</testsuite>
501-
</testsuites>
502-
503-
"""
491+
<?xml version="1.0" encoding="UTF-8"?>
492+
<testsuites name="pkl-tests" tests="5" failures="3">
493+
<testsuite name="test1" tests="2" failures="1">
494+
<testcase classname="test1.facts" name="foo"></testcase>
495+
<testcase classname="test1.facts" name="bar">
496+
<failure message="Fact Failure">5 == 9 (/tempDir/test1.pkl, line xx)
497+
│
498+
false</failure>
499+
</testcase>
500+
</testsuite>
501+
<testsuite name="test2" tests="3" failures="2">
502+
<testcase classname="test2.facts" name="xxx">
503+
<failure message="Fact Failure">false (/tempDir/test2.pkl, line xx)
504+
</failure>
505+
</testcase>
506+
<testcase classname="test2.facts" name="yyy">
507+
<failure message="Fact Failure">false (/tempDir/test2.pkl, line xx)
508+
</failure>
509+
</testcase>
510+
<testcase classname="test2.facts" name="zzz"></testcase>
511+
</testsuite>
512+
</testsuites>
513+
514+
"""
504515
.trimIndent()
505516
)
506517
}

‎pkl-core/src/main/java/org/pkl/core/ast/ExpressionNode.java‎

Lines changed: 23 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
2+
* Copyright © 2024-2026 Apple Inc. and the Pkl project authors. All rights reserved.
33
*
44
* Licensed under the Apache License, Version 2.0 (the "License");
55
* you may not use this file except in compliance with the License.
@@ -16,12 +16,18 @@
1616
package org.pkl.core.ast;
1717

1818
import com.oracle.truffle.api.frame.VirtualFrame;
19+
import com.oracle.truffle.api.instrumentation.GenerateWrapper;
20+
import com.oracle.truffle.api.instrumentation.InstrumentableNode;
21+
import com.oracle.truffle.api.instrumentation.ProbeNode;
22+
import com.oracle.truffle.api.instrumentation.Tag;
1923
import com.oracle.truffle.api.nodes.UnexpectedResultException;
2024
import com.oracle.truffle.api.source.SourceSection;
25+
import org.pkl.core.runtime.PklTags;
2126
import org.pkl.core.runtime.VmTypesGen;
2227
import org.pkl.core.runtime.VmUtils;
2328

24-
public abstract class ExpressionNode extends PklNode {
29+
@GenerateWrapper
30+
public abstract class ExpressionNode extends PklNode implements InstrumentableNode {
2531
protected ExpressionNode(SourceSection sourceSection) {
2632
super(sourceSection);
2733
}
@@ -43,4 +49,19 @@ public double executeFloat(VirtualFrame frame) throws UnexpectedResultException
4349
public boolean executeBoolean(VirtualFrame frame) throws UnexpectedResultException {
4450
return VmTypesGen.expectBoolean(executeGeneric(frame));
4551
}
52+
53+
@Override
54+
public boolean hasTag(Class<? extends Tag> tag) {
55+
return tag == PklTags.Expression.class;
56+
}
57+
58+
@Override
59+
public boolean isInstrumentable() {
60+
return true;
61+
}
62+
63+
@Override
64+
public WrapperNode createWrapper(ProbeNode probe) {
65+
return new ExpressionNodeWrapper(this, probe);
66+
}
4667
}

‎pkl-core/src/main/java/org/pkl/core/ast/PklNode.java‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
2+
* Copyright © 2024-2026 Apple Inc. and the Pkl project authors. All rights reserved.
33
*
44
* Licensed under the Apache License, Version 2.0 (the "License");
55
* you may not use this file except in compliance with the License.
@@ -17,6 +17,7 @@
1717

1818
import com.oracle.truffle.api.CompilerDirectives.TruffleBoundary;
1919
import com.oracle.truffle.api.dsl.TypeSystemReference;
20+
import com.oracle.truffle.api.instrumentation.InstrumentableNode.WrapperNode;
2021
import com.oracle.truffle.api.nodes.Node;
2122
import com.oracle.truffle.api.nodes.NodeInfo;
2223
import com.oracle.truffle.api.source.SourceSection;
@@ -38,7 +39,10 @@ protected PklNode() {
3839
}
3940

4041
@Override
41-
public SourceSection getSourceSection() {
42+
public final SourceSection getSourceSection() {
43+
if (this instanceof WrapperNode wrapperNode) {
44+
return wrapperNode.getDelegateNode().getSourceSection();
45+
}
4246
return sourceSection;
4347
}
4448

‎pkl-core/src/main/java/org/pkl/core/ast/builder/AstBuilder.java‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -2813,7 +2813,7 @@ private URI resolveImport(String importUri, StringConstant ctx) {
28132813
throw exceptionBuilder()
28142814
.evalError(e.getMessage(), e.getMessageArguments())
28152815
.withCause(e.getCause())
2816-
.withHint(e.getHint())
2816+
.withHintBuilder(e.getHintBuilder())
28172817
.withSourceSection(createSourceSection(ctx))
28182818
.build();
28192819
} catch (ExternalReaderProcessException e) {

‎pkl-core/src/main/java/org/pkl/core/ast/expression/binary/ComparatorNode.java‎

Lines changed: 4 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
2+
* Copyright © 2024-2026 Apple Inc. and the Pkl project authors. All rights reserved.
33
*
44
* Licensed under the Apache License, Version 2.0 (the "License");
55
* you may not use this file except in compliance with the License.
@@ -15,12 +15,13 @@
1515
*/
1616
package org.pkl.core.ast.expression.binary;
1717

18+
import com.oracle.truffle.api.frame.VirtualFrame;
1819
import com.oracle.truffle.api.source.SourceSection;
1920

2021
public abstract class ComparatorNode extends BinaryExpressionNode {
21-
public ComparatorNode(SourceSection sourceSection) {
22+
protected ComparatorNode(SourceSection sourceSection) {
2223
super(sourceSection);
2324
}
2425

25-
public abstract boolean executeWith(Object left, Object right);
26+
public abstract boolean executeWith(VirtualFrame frame, Object left, Object right);
2627
}

‎pkl-core/src/main/java/org/pkl/core/ast/expression/binary/GreaterThanNode.java‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
2+
* Copyright © 2024-2026 Apple Inc. and the Pkl project authors. All rights reserved.
33
*
44
* Licensed under the Apache License, Version 2.0 (the "License");
55
* you may not use this file except in compliance with the License.
@@ -17,12 +17,15 @@
1717

1818
import com.oracle.truffle.api.CompilerDirectives.TruffleBoundary;
1919
import com.oracle.truffle.api.dsl.Specialization;
20+
import com.oracle.truffle.api.instrumentation.GenerateWrapper;
21+
import com.oracle.truffle.api.instrumentation.ProbeNode;
2022
import com.oracle.truffle.api.nodes.NodeInfo;
2123
import com.oracle.truffle.api.source.SourceSection;
2224
import org.pkl.core.runtime.VmDataSize;
2325
import org.pkl.core.runtime.VmDuration;
2426

2527
@NodeInfo(shortName = ">")
28+
@GenerateWrapper
2629
public abstract class GreaterThanNode extends ComparatorNode {
2730
protected GreaterThanNode(SourceSection sourceSection) {
2831
super(sourceSection);
@@ -63,4 +66,9 @@ protected boolean eval(VmDuration left, VmDuration right) {
6366
protected boolean eval(VmDataSize left, VmDataSize right) {
6467
return left.compareTo(right) > 0;
6568
}
69+
70+
@Override
71+
public WrapperNode createWrapper(ProbeNode probe) {
72+
return new GreaterThanNodeWrapper(sourceSection, this, probe);
73+
}
6674
}

‎pkl-core/src/main/java/org/pkl/core/ast/expression/binary/LessThanNode.java‎

Lines changed: 9 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright © 2024 Apple Inc. and the Pkl project authors. All rights reserved.
2+
* Copyright © 2024-2026 Apple Inc. and the Pkl project authors. All rights reserved.
33
*
44
* Licensed under the Apache License, Version 2.0 (the "License");
55
* you may not use this file except in compliance with the License.
@@ -17,12 +17,15 @@
1717

1818
import com.oracle.truffle.api.CompilerDirectives.TruffleBoundary;
1919
import com.oracle.truffle.api.dsl.Specialization;
20+
import com.oracle.truffle.api.instrumentation.GenerateWrapper;
21+
import com.oracle.truffle.api.instrumentation.ProbeNode;
2022
import com.oracle.truffle.api.nodes.NodeInfo;
2123
import com.oracle.truffle.api.source.SourceSection;
2224
import org.pkl.core.runtime.VmDataSize;
2325
import org.pkl.core.runtime.VmDuration;
2426

2527
@NodeInfo(shortName = "<")
28+
@GenerateWrapper
2629
public abstract class LessThanNode extends ComparatorNode {
2730
protected LessThanNode(SourceSection sourceSection) {
2831
super(sourceSection);
@@ -63,4 +66,9 @@ protected boolean eval(VmDuration left, VmDuration right) {
6366
protected boolean eval(VmDataSize left, VmDataSize right) {
6467
return left.compareTo(right) < 0;
6568
}
69+
70+
@Override
71+
public WrapperNode createWrapper(ProbeNode probe) {
72+
return new LessThanNodeWrapper(sourceSection, this, probe);
73+
}
6674
}

‎pkl-core/src/main/java/org/pkl/core/ast/expression/member/InvokeMethodVirtualNode.java‎

Lines changed: 10 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,5 +1,5 @@
11
/*
2-
* Copyright © 2024-2025 Apple Inc. and the Pkl project authors. All rights reserved.
2+
* Copyright © 2024-2026 Apple Inc. and the Pkl project authors. All rights reserved.
33
*
44
* Licensed under the Apache License, Version 2.0 (the "License");
55
* you may not use this file except in compliance with the License.
@@ -23,6 +23,8 @@
2323
import com.oracle.truffle.api.dsl.NodeChild;
2424
import com.oracle.truffle.api.dsl.Specialization;
2525
import com.oracle.truffle.api.frame.VirtualFrame;
26+
import com.oracle.truffle.api.instrumentation.GenerateWrapper;
27+
import com.oracle.truffle.api.instrumentation.ProbeNode;
2628
import com.oracle.truffle.api.nodes.DirectCallNode;
2729
import com.oracle.truffle.api.nodes.ExplodeLoop;
2830
import com.oracle.truffle.api.nodes.IndirectCallNode;
@@ -40,6 +42,7 @@
4042
@ImportStatic(Identifier.class)
4143
@NodeChild(value = "receiverNode", type = ExpressionNode.class)
4244
@NodeChild(value = "receiverClassNode", type = GetClassNode.class, executeWith = "receiverNode")
45+
@GenerateWrapper
4346
public abstract class InvokeMethodVirtualNode extends ExpressionNode {
4447
protected final Identifier methodName;
4548
@Children private final ExpressionNode[] argumentNodes;
@@ -173,6 +176,12 @@ protected ClassMethod resolveMethod(VmClass receiverClass) {
173176
.build();
174177
}
175178

179+
@Override
180+
public WrapperNode createWrapper(ProbeNode probe) {
181+
return new InvokeMethodVirtualNodeWrapper(
182+
sourceSection, methodName, argumentNodes, lookupMode, needsConst, this, probe);
183+
}
184+
176185
private void checkConst(ClassMethod method) {
177186
if (needsConst && !method.isConst()) {
178187
CompilerDirectives.transferToInterpreter();

0 commit comments

Comments
 (0)