Skip to content

Commit 6b82f19

Browse files
pjhartoutpjhartout
authored
v1.0.1 - add benchmark results and description update to docs (#41)
* add benchmark results and description update to docs * transpose table, remove VUN * Update version to 1.0.1, remove redundant section from README, and enhance benchmarking table in documentation
1 parent f59e96b commit 6b82f19

File tree

3 files changed

+129
-197
lines changed

3 files changed

+129
-197
lines changed

‎README.md‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -172,3 +172,69 @@ for metric in tqdm(metrics):
172172
generated,
173173
)
174174
```
175+
## Example Benchmark
176+
177+
The following results mirror the tables from our paper. Bold indicates best, and underlined indicates second-best. Values are multiplied by 100 for legibility. Standard deviations are obtained with subsampling using `StandardPGDInterval` and `MoleculePGDInterval`. Specific parameters are discussed in the paper.
178+
179+
<table>
180+
<thead>
181+
<tr>
182+
<th>Method</th>
183+
<th style="text-align:right;">Planar-L</th>
184+
<th style="text-align:right;">Lobster-L</th>
185+
<th style="text-align:right;">SBM-L</th>
186+
<th style="text-align:right;">Proteins</th>
187+
<th style="text-align:right;">Guacamol</th>
188+
<th style="text-align:right;">Moses</th>
189+
</tr>
190+
</thead>
191+
<tbody>
192+
<tr>
193+
<td>AutoGraph</td>
194+
<td style="text-align:right;"><strong>34.0 ± 1.8</strong></td>
195+
<td style="text-align:right;"><u>18.0 ± 1.6</u></td>
196+
<td style="text-align:right;"><strong>5.6 ± 1.5</strong></td>
197+
<td style="text-align:right;"><strong>67.7 ± 7.4</strong></td>
198+
<td style="text-align:right;"><u>22.9 ± 0.5</u></td>
199+
<td style="text-align:right;"><strong>29.6 ± 0.4</strong></td>
200+
</tr>
201+
<tr>
202+
<td>AutoGraph*</td>
203+
<td style="text-align:right;">—</td>
204+
<td style="text-align:right;">—</td>
205+
<td style="text-align:right;">—</td>
206+
<td style="text-align:right;">—</td>
207+
<td style="text-align:right;"><strong>10.4 ± 1.2</strong></td>
208+
<td style="text-align:right;">—</td>
209+
</tr>
210+
<tr>
211+
<td>DiGress</td>
212+
<td style="text-align:right;">45.2 ± 1.8</td>
213+
<td style="text-align:right;"><strong>3.2 ± 2.6</strong></td>
214+
<td style="text-align:right;"><u>17.4 ± 2.3</u></td>
215+
<td style="text-align:right;">88.1 ± 3.1</td>
216+
<td style="text-align:right;">32.7 ± 0.5</td>
217+
<td style="text-align:right;"><u>33.4 ± 0.5</u></td>
218+
</tr>
219+
<tr>
220+
<td>GRAN</td>
221+
<td style="text-align:right;">99.7 ± 0.2</td>
222+
<td style="text-align:right;">85.4 ± 0.5</td>
223+
<td style="text-align:right;">69.1 ± 1.4</td>
224+
<td style="text-align:right;">89.7 ± 2.7</td>
225+
<td style="text-align:right;">—</td>
226+
<td style="text-align:right;">—</td>
227+
</tr>
228+
<tr>
229+
<td>ESGG</td>
230+
<td style="text-align:right;"><u>45.0 ± 1.4</u></td>
231+
<td style="text-align:right;">69.9 ± 0.6</td>
232+
<td style="text-align:right;">99.4 ± 0.2</td>
233+
<td style="text-align:right;"><u>79.2 ± 4.3</u></td>
234+
<td style="text-align:right;">—</td>
235+
<td style="text-align:right;">—</td>
236+
</tr>
237+
</tbody>
238+
</table>
239+
240+
<sub>* AutoGraph* denotes a variant that leverages additional training heuristics as described in the paper.</sub>

‎docs/index.md‎

Lines changed: 62 additions & 196 deletions
Original file line numberDiff line numberDiff line change
@@ -92,201 +92,67 @@ PGD and its motivation are described in more detail in the paper and API docs.
9292

9393
### Benchmarking snapshot
9494

95-
The table below shows an example benchmark generated with this library across multiple datasets and models. Values illustrate typical outputs from the implemented metrics (VUN, PGD, and PGD subscores). For completeness, this library and our paper also implements and provides various MMD estimates on the datasets below.
96-
97-
<style>
98-
table {
99-
font-size: 85%;
100-
border-collapse: collapse;
101-
}
102-
th, td {
103-
text-align: center;
104-
padding: 4px 6px;
105-
border: 1px solid #ddd;
106-
}
107-
th:first-child, td:first-child {
108-
text-align: left;
109-
}
110-
th:nth-child(2), td:nth-child(2) {
111-
text-align: left;
112-
}
113-
</style>
95+
The table below shows an example benchmark generated with this library across multiple datasets and models. Values illustrate typical outputs from the newly proposed PolyGraph Discrepancy. For completeness, this library and our paper also implements and provides various MMD estimates on the datasets below. Values are scaled by 100 for legibility and subsampling is used to obtain standard deviations (using `StandardPGDInterval` and `MoleculePGDInterval`). More details are provided in our paper.
11496

11597
<table>
116-
<thead>
117-
<tr>
118-
<th rowspan="2">Dataset</th>
119-
<th rowspan="2">Model</th>
120-
<th rowspan="2">VUN (↑)</th>
121-
<th rowspan="2">PGD (↓)</th>
122-
<th colspan="6">PGD subscores</th>
123-
</tr>
124-
<tr>
125-
<th>Clust. (↓)</th>
126-
<th>Deg. (↓)</th>
127-
<th>GIN (↓)</th>
128-
<th>Orb5. (↓)</th>
129-
<th>Orb4. (↓)</th>
130-
<th>Eig. (↓)</th>
131-
</tr>
132-
</thead>
133-
<tbody>
134-
135-
<!-- Planar-L -->
136-
<tr><td rowspan="4"><b>Planar-L</b></td>
137-
<td>AutoGraph</td>
138-
<td><i>85.1</i></td>
139-
<td><b>34.0 ± 1.8</b></td>
140-
<td><b>7.0 ± 2.9</b></td>
141-
<td><b>7.8 ± 3.2</b></td>
142-
<td><b>8.8 ± 3.0</b></td>
143-
<td><b>34.0 ± 1.8</b></td>
144-
<td><b>28.5 ± 1.5</b></td>
145-
<td><b>26.9 ± 2.3</b></td></tr>
146-
<tr><td>DiGress</td>
147-
<td>80.1</td>
148-
<td>45.2 ± 1.8</td>
149-
<td>24.8 ± 2.0</td>
150-
<td>23.3 ± 1.2</td>
151-
<td><i>29.0 ± 1.1</i></td>
152-
<td>45.2 ± 1.8</td>
153-
<td><i>40.3 ± 1.8</i></td>
154-
<td>39.4 ± 2.0</td></tr>
155-
<tr><td>GRAN</td>
156-
<td>1.6</td>
157-
<td>99.7 ± 0.2</td>
158-
<td>99.3 ± 0.2</td>
159-
<td>98.3 ± 0.3</td>
160-
<td>98.3 ± 0.3</td>
161-
<td>99.7 ± 0.1</td>
162-
<td>99.2 ± 0.2</td>
163-
<td>98.5 ± 0.4</td></tr>
164-
<tr><td>ESGG</td>
165-
<td><b>93.9</b></td>
166-
<td><i>45.0 ± 1.4</i></td>
167-
<td><i>10.9 ± 3.2</i></td>
168-
<td><i>21.7 ± 3.0</i></td>
169-
<td>32.9 ± 2.2</td>
170-
<td><i>45.0 ± 1.4</i></td>
171-
<td>42.8 ± 1.9</td>
172-
<td><i>29.6 ± 1.6</i></td></tr>
173-
174-
<!-- Lobster-L -->
175-
<tr><td rowspan="4"><b>Lobster-L</b></td>
176-
<td>AutoGraph</td>
177-
<td><i>83.1</i></td>
178-
<td><i>18.0 ± 1.6</i></td>
179-
<td>4.2 ± 1.9</td>
180-
<td><i>12.1 ± 1.6</i></td>
181-
<td><i>14.8 ± 1.5</td>
182-
<td><i>18.0 ± 1.6</i></td>
183-
<td><i>16.1 ± 1.6</i></td>
184-
<td><i>13.0 ± 1.1</i></td></tr>
185-
<tr><td>DiGress</td>
186-
<td><b>91.4</b></td>
187-
<td><b>3.2 ± 2.6</b></td>
188-
<td><i>2.0 ± 1.3</i></td>
189-
<td><b>1.2 ± 1.5</b></td>
190-
<td><b>2.3 ± 2.0</b></td>
191-
<td><b>3.0 ± 3.1</b></td>
192-
<td><b>4.5 ± 2.3</b></td>
193-
<td><b>1.3 ± 1.1</b></td></tr>
194-
<tr><td>GRAN</td>
195-
<td>41.3</td>
196-
<td>85.4 ± 0.5</td>
197-
<td>20.8 ± 1.1</td>
198-
<td>77.1 ± 1.2</td>
199-
<td>79.8 ± 0.6</td>
200-
<td>85.4 ± 0.5</td>
201-
<td>85.0 ± 0.6</td>
202-
<td>69.8 ± 1.2</td></tr>
203-
<tr><td>ESGG</td>
204-
<td>70.9</td>
205-
<td>69.9 ± 0.6</td>
206-
<td><b>0.0 ± 0.0</b></td>
207-
<td>63.4 ± 1.1</td>
208-
<td>66.8 ± 1.0</td>
209-
<td>69.9 ± 0.6</td>
210-
<td>66.0 ± 0.6</td>
211-
<td>51.7 ± 1.8</td></tr>
212-
213-
<!-- SBM-L -->
214-
<tr><td rowspan="4"><b>SBM-L</b></td>
215-
<td>AutoGraph</td>
216-
<td><b>85.6</b></td>
217-
<td><b>5.6 ± 1.5</b></td>
218-
<td><b>0.3 ± 0.6</b></td>
219-
<td><b>6.2 ± 1.4</b></td>
220-
<td><b>6.3 ± 1.3</td>
221-
<td><b>3.2 ± 2.2</td>
222-
<td><b>4.4 ± 2.0</td>
223-
<td><b>2.5 ± 2.2</td></tr>
224-
<tr><td>DiGress</td>
225-
<td><i>73.0</i></td>
226-
<td><i>17.4 ± 2.3</td>
227-
<td><i>5.7 ± 2.8</td>
228-
<td><i>8.2 ± 3.3</td>
229-
<td><i>13.8 ± 1.7</td>
230-
<td><i>17.4 ± 2.3</td>
231-
<td><i>14.8 ± 2.5</td>
232-
<td><i>8.7 ± 3.0</td></tr>
233-
<tr><td>GRAN</td>
234-
<td>21.4</td>
235-
<td>69.1 ± 1.4</td>
236-
<td>50.2 ± 1.9</td>
237-
<td>58.6 ± 1.4</td>
238-
<td>69.1 ± 1.4</td>
239-
<td>65.7 ± 1.3</td>
240-
<td>62.8 ± 1.3</td>
241-
<td>55.9 ± 1.5</td></tr>
242-
<tr><td>ESGG</td>
243-
<td>10.4</td>
244-
<td>99.4 ± 0.2</td>
245-
<td>97.9 ± 0.5</td>
246-
<td>97.5 ± 0.6</td>
247-
<td>98.3 ± 0.4</td>
248-
<td>96.8 ± 0.4</td>
249-
<td>89.2 ± 0.7</td>
250-
<td>99.4 ± 0.2</td></tr>
251-
252-
<!-- Proteins -->
253-
<tr><td rowspan="4"><b>Proteins</b></td>
254-
<td>AutoGraph</td>
255-
<td>–</td>
256-
<td><b>67.7 ± 7.4</b></td>
257-
<td><i>47.7 ± 5.7</i></td>
258-
<td><i>31.5 ± 8.5</td>
259-
<td><i>45.3 ± 5.1</td>
260-
<td><b>67.7 ± 7.4</td>
261-
<td><b>47.4 ± 7.0</td>
262-
<td>53.2 ± 6.9</td></tr>
263-
<tr><td>DiGress</td>
264-
<td>–</td>
265-
<td>88.1 ± 3.1</td>
266-
<td><b>36.1 ± 4.3</b></td>
267-
<td><b>29.2 ± 5.0</b></td>
268-
<td><b>23.2 ± 5.3</b></td>
269-
<td>88.1 ± 3.1</td>
270-
<td><i>60.8 ± 3.6</i></td>
271-
<td><b>23.4 ± 11.8</b></td></tr>
272-
<tr><td>GRAN</td>
273-
<td>–</td>
274-
<td>89.7 ± 2.7</td>
275-
<td>86.0 ± 2.0</td>
276-
<td>70.6 ± 3.1</td>
277-
<td>71.5 ± 3.0</td>
278-
<td>90.4 ± 2.4</td>
279-
<td>84.4 ± 3.3</td>
280-
<td>76.7 ± 4.7</td></tr>
281-
<tr><td>ESGG</td>
282-
<td>–</td>
283-
<td><i>79.2 ± 4.3</td>
284-
<td>58.2 ± 3.6</td>
285-
<td>54.0 ± 3.6</td>
286-
<td>57.4 ± 4.1</td>
287-
<td><i>80.2 ± 3.1</td>
288-
<td>72.5 ± 3.0</td>
289-
<td><i>24.3 ± 11.0</td></tr>
290-
291-
</tbody>
292-
</table>
98+
<thead>
99+
<tr>
100+
<th>Method</th>
101+
<th style="text-align:right;">Planar-L</th>
102+
<th style="text-align:right;">Lobster-L</th>
103+
<th style="text-align:right;">SBM-L</th>
104+
<th style="text-align:right;">Proteins</th>
105+
<th style="text-align:right;">Guacamol</th>
106+
<th style="text-align:right;">Moses</th>
107+
</tr>
108+
</thead>
109+
<tbody>
110+
<tr>
111+
<td>AutoGraph</td>
112+
<td style="text-align:right;"><strong>34.0 ± 1.8</strong></td>
113+
<td style="text-align:right;"><u>18.0 ± 1.6</u></td>
114+
<td style="text-align:right;"><strong>5.6 ± 1.5</strong></td>
115+
<td style="text-align:right;"><strong>67.7 ± 7.4</strong></td>
116+
<td style="text-align:right;"><u>22.9 ± 0.5</u></td>
117+
<td style="text-align:right;"><strong>29.6 ± 0.4</strong></td>
118+
</tr>
119+
<tr>
120+
<td>AutoGraph*</td>
121+
<td style="text-align:right;">—</td>
122+
<td style="text-align:right;">—</td>
123+
<td style="text-align:right;">—</td>
124+
<td style="text-align:right;">—</td>
125+
<td style="text-align:right;"><strong>10.4 ± 1.2</strong></td>
126+
<td style="text-align:right;">—</td>
127+
</tr>
128+
<tr>
129+
<td>DiGress</td>
130+
<td style="text-align:right;">45.2 ± 1.8</td>
131+
<td style="text-align:right;"><strong>3.2 ± 2.6</strong></td>
132+
<td style="text-align:right;"><u>17.4 ± 2.3</u></td>
133+
<td style="text-align:right;">88.1 ± 3.1</td>
134+
<td style="text-align:right;">32.7 ± 0.5</td>
135+
<td style="text-align:right;"><u>33.4 ± 0.5</u></td>
136+
</tr>
137+
<tr>
138+
<td>GRAN</td>
139+
<td style="text-align:right;">99.7 ± 0.2</td>
140+
<td style="text-align:right;">85.4 ± 0.5</td>
141+
<td style="text-align:right;">69.1 ± 1.4</td>
142+
<td style="text-align:right;">89.7 ± 2.7</td>
143+
<td style="text-align:right;">—</td>
144+
<td style="text-align:right;">—</td>
145+
</tr>
146+
<tr>
147+
<td>ESGG</td>
148+
<td style="text-align:right;"><u>45.0 ± 1.4</u></td>
149+
<td style="text-align:right;">69.9 ± 0.6</td>
150+
<td style="text-align:right;">99.4 ± 0.2</td>
151+
<td style="text-align:right;"><u>79.2 ± 4.3</u></td>
152+
<td style="text-align:right;">—</td>
153+
<td style="text-align:right;">—</td>
154+
</tr>
155+
</tbody>
156+
</table>
157+
158+
<sub>* AutoGraph* denotes a variant that leverages additional training heuristics as described in the paper.</sub>

‎pyproject.toml‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@ build-backend = "setuptools.build_meta"
44

55
[project]
66
name = "polygraph-benchmark"
7-
version = "1.0.0"
7+
version = "1.0.1"
88
description = "Evaluation benchmarks for graph generative models"
99
readme = "README.md"
1010
authors = [

0 commit comments

Comments
 (0)