Skip to content

Commit 3407f0d

Browse files
author
Marcin Kardas
committed
Add notebooks
1 parent eda6a36 commit 3407f0d

File tree

4 files changed

+1403
-0
lines changed

4 files changed

+1403
-0
lines changed

‎notebooks/datasets.ipynb

Lines changed: 238 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,238 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"metadata": {},
6+
"source": [
7+
"# Datasets\n",
8+
"\n",
9+
"We published four datasets for training and evaluating extraction of performance results from machine learning papers. In this notebook we describe the format and show how to use our python API to conveniently work with the datasets. Due to the licensing the datasets consists of metadata and annotations, but do not include papers and data extracted from them. However, we made special effort in our extraction pipeline to get reproducible results."
10+
]
11+
},
12+
{
13+
"cell_type": "markdown",
14+
"metadata": {},
15+
"source": [
16+
"Simple functions to load the datasets"
17+
]
18+
},
19+
{
20+
"cell_type": "code",
21+
"execution_count": 1,
22+
"metadata": {},
23+
"outputs": [],
24+
"source": [
25+
"from axcell.helpers.datasets import read_arxiv_papers\n",
26+
"from pathlib import Path\n",
27+
"\n",
28+
"V1_URL = 'https://github.com/paperswithcode/axcell/releases/download/v1.0/'\n",
29+
"ARXIV_PAPERS_URL = V1_URL + 'arxiv-papers.csv.xz'\n",
30+
"SEGMENTED_TABLES_URL = V1_URL + 'segmented-tables.json.xz'\n",
31+
"PWC_LEADERBOARDS_URL = V1_URL + 'pwc-leaderboards.json.xz'"
32+
]
33+
},
34+
{
35+
"cell_type": "markdown",
36+
"metadata": {},
37+
"source": [
38+
"## ArxivPapers\n",
39+
"\n",
40+
"**ArxivPapers** dataset is a corpus of over 100,000 scientific papers related to machine learning. In our work we use the corpus for self-supervised training of ULMFiT langauge model (see the lm_training notebook) and for extraction of common abbreviations. The dataset is a CSV file with one row per paper and the following fields:\n",
41+
"* arxiv_id - arXiv identifier with version\n",
42+
"* archive_size - the file size in bytes of the e-print archive\n",
43+
"* sha256 - SHA-256 hash of the e-print archive\n",
44+
"* title - paper's title\n",
45+
"* status - the text and tables extraction status for this paper, one of:\n",
46+
" + success,\n",
47+
" + no-tex - LaTeX source is unavailable,\n",
48+
" + processing-error - extraction issues,\n",
49+
" + withdrawn - the paper is withdrawn from arXiv\n",
50+
"* sections - number of extracted sections and subsections\n",
51+
"* tables - number of extracted tables"
52+
]
53+
},
54+
{
55+
"cell_type": "code",
56+
"execution_count": 2,
57+
"metadata": {},
58+
"outputs": [
59+
{
60+
"name": "stdout",
61+
"output_type": "stream",
62+
"text": [
63+
"Number of papers: 104710\n",
64+
"└── with LaTeX source: 93811\n",
65+
"Number of extracted tables: 277946\n"
66+
]
67+
}
68+
],
69+
"source": [
70+
"arxiv_papers = read_arxiv_papers(ARXIV_PAPERS_URL)\n",
71+
"\n",
72+
"print(f'Number of papers: {len(arxiv_papers):8}')\n",
73+
"print(f'└── with LaTeX source: {(~arxiv_papers.status.isin([\"no-tex\", \"withdrawn\"])).sum():8}')\n",
74+
"print(f'Number of extracted tables: {arxiv_papers.tables.sum():8}')"
75+
]
76+
},
77+
{
78+
"cell_type": "markdown",
79+
"metadata": {},
80+
"source": [
81+
"The arXiv id can be used to generate links to e-prints. Please read https://arxiv.org/help/bulk_data and play nice."
82+
]
83+
},
84+
{
85+
"cell_type": "code",
86+
"execution_count": 3,
87+
"metadata": {},
88+
"outputs": [
89+
{
90+
"data": {
91+
"text/plain": [
92+
"104705 http://export.arxiv.org/e-print/2002.08204v1\n",
93+
"104706 http://export.arxiv.org/e-print/2002.08253v1\n",
94+
"104707 http://export.arxiv.org/e-print/2002.08264v1\n",
95+
"104708 http://export.arxiv.org/e-print/2002.08301v1\n",
96+
"104709 http://export.arxiv.org/e-print/2002.08325v1\n",
97+
"dtype: object"
98+
]
99+
},
100+
"execution_count": 3,
101+
"metadata": {},
102+
"output_type": "execute_result"
103+
}
104+
],
105+
"source": [
106+
"def get_eprint_link(paper):\n",
107+
" return f'http://export.arxiv.org/e-print/{paper.arxiv_id}'\n",
108+
"\n",
109+
"links = arxiv_papers.apply(get_eprint_link, axis=1)\n",
110+
"links.tail()"
111+
]
112+
},
113+
{
114+
"cell_type": "markdown",
115+
"metadata": {},
116+
"source": [
117+
"## SegmentedTables & LinkedResults\n",
118+
"\n",
119+
"The **SegmentedTables** dataset contains annotations of almost 2,000 tables. The dataset is a JSON array with one item per paper and the following fields:\n",
120+
"* arxiv_id - arXiv identifier with version. The version can be different than in **ArxivTables**,\n",
121+
"* sha256 - SHA-256 hash of the e-print archive\n",
122+
"* fold - one of 11 folds, f.e., img_class or speech_rec. Each paper has exactly one fold, even if it's related to more than one task,\n",
123+
"* tables - array of tables annotations\n",
124+
" + index - 0-based index of tables extracted from paper,\n",
125+
" + leaderboard - a boolean denoting if this table is a leaderboard table,\n",
126+
" + ablation - a boolean denoting if this table is an ablation table (a table can be both a leaderboard and an ablation table),\n",
127+
" + dataset_text - datasets mentioned in table's caption, not normalized\n",
128+
" + segmentation - for leaderboard tables, a 2D array (list of lists) with one label per cell\n",
129+
"\n",
130+
"Additionally we annotated part of the tables with performance results, called simply the **LinkedResults** dataset. Each table contains a 'records' array with items containing:\n",
131+
"* task, dataset, metric - task, dataset and metric names normalized across all papers from the **LinkedResults** dataset,\n",
132+
"* value - normalized metric value,\n",
133+
"* model - model name,\n",
134+
"* row, column - 0-based cell location with this result."
135+
]
136+
},
137+
{
138+
"cell_type": "code",
139+
"execution_count": 4,
140+
"metadata": {},
141+
"outputs": [
142+
{
143+
"name": "stdout",
144+
"output_type": "stream",
145+
"text": [
146+
"Number of papers: 352\n",
147+
"Number of tables: 1994\n",
148+
"├── leaderboards: 796\n",
149+
"└── ablations: 468\n",
150+
"Linked results: 1591\n"
151+
]
152+
}
153+
],
154+
"source": [
155+
"from axcell.helpers.datasets import read_tables_annotations\n",
156+
"\n",
157+
"segmented_tables_annotations = read_tables_annotations(SEGMENTED_TABLES_URL)\n",
158+
"\n",
159+
"leaderboards = (segmented_tables_annotations.tables.apply(\n",
160+
" lambda tables: len([t for t in tables if t['leaderboard']])\n",
161+
").sum())\n",
162+
"ablations = (segmented_tables_annotations.tables.apply(\n",
163+
" lambda tables: len([t for t in tables if t['ablation']])\n",
164+
").sum())\n",
165+
"records = (segmented_tables_annotations.tables.apply(\n",
166+
" lambda tables: sum([len(t['records']) for t in tables])\n",
167+
").sum())\n",
168+
"\n",
169+
"print(f'Number of papers: {len(segmented_tables_annotations):8}')\n",
170+
"print(f'Number of tables: {segmented_tables_annotations.tables.apply(len).sum():8}')\n",
171+
"print(f'├── leaderboards: {leaderboards:8}')\n",
172+
"print(f'└── ablations: {ablations:8}')\n",
173+
"print(f'Linked results: {records:8}')"
174+
]
175+
},
176+
{
177+
"cell_type": "markdown",
178+
"metadata": {},
179+
"source": [
180+
"## PWCLeaderboards\n",
181+
"\n",
182+
"The **PWCLeaderboards** dataset is similar in structure to the **LinkedResults** dataset. It's a JSON array with one item per paper, containing:\n",
183+
"* arxiv_id - arXiv identifier with version. The version corresponds to the version in **ArxivTables**,\n",
184+
"* tables\n",
185+
" + index - 0-based table index\n",
186+
" + records - as in **LinkedResults**"
187+
]
188+
},
189+
{
190+
"cell_type": "code",
191+
"execution_count": 5,
192+
"metadata": {},
193+
"outputs": [
194+
{
195+
"name": "stdout",
196+
"output_type": "stream",
197+
"text": [
198+
"Number of papers: 731\n",
199+
"Number of tables: 1278\n",
200+
"Linked results: 5393\n"
201+
]
202+
}
203+
],
204+
"source": [
205+
"pwc_leaderboards = read_tables_annotations(PWC_LEADERBOARDS_URL)\n",
206+
"\n",
207+
"records = (pwc_leaderboards.tables.apply(\n",
208+
" lambda tables: sum([len(t['records']) for t in tables])\n",
209+
").sum())\n",
210+
"\n",
211+
"print(f'Number of papers: {len(pwc_leaderboards):8}')\n",
212+
"print(f'Number of tables: {pwc_leaderboards.tables.apply(len).sum():8}')\n",
213+
"print(f'Linked results: {records:8}')"
214+
]
215+
}
216+
],
217+
"metadata": {
218+
"kernelspec": {
219+
"display_name": "Python 3",
220+
"language": "python",
221+
"name": "python3"
222+
},
223+
"language_info": {
224+
"codemirror_mode": {
225+
"name": "ipython",
226+
"version": 3
227+
},
228+
"file_extension": ".py",
229+
"mimetype": "text/x-python",
230+
"name": "python",
231+
"nbconvert_exporter": "python",
232+
"pygments_lexer": "ipython3",
233+
"version": "3.6.8"
234+
}
235+
},
236+
"nbformat": 4,
237+
"nbformat_minor": 2
238+
}

0 commit comments

Comments
 (0)