Skip to content

Commit 6ea21f3

Browse files
nuestnuest
committed
add event recipe
1 parent b097ab3 commit 6ea21f3

File tree

7 files changed

+338
-0
lines changed

7 files changed

+338
-0
lines changed

‎.gitignore‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -2,3 +2,9 @@
22
\.sass-cache
33

44
_site/
5+
6+
guide/recipe.pdf
7+
8+
guide/temp.pdf
9+
10+
guide/CODECHECK-event-recipe.pdf

‎assets/codecheck.css‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -120,6 +120,10 @@ blockquote p:first-of-type {
120120
ol {
121121
max-width: 80%;
122122
}
123+
124+
ul {
125+
max-width: 80%;
126+
}
123127
}
124128

125129
@media screen and (min-width: 1200px) {
@@ -130,6 +134,10 @@ blockquote p:first-of-type {
130134
ol {
131135
max-width: 60%;
132136
}
137+
138+
ul {
139+
max-width: 60%;
140+
}
133141
}
134142

135143
/* based on https://milanaryal.com/adding-hover-anchor-links-to-header-on-github-pages-using-jekyll/ */

‎get-involved.md‎

Lines changed: 8 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -16,6 +16,7 @@ Please open an issue on the [codecheckers](https://github.com/codecheckers/codec
1616
You want to share the code underlying your research paper? **Congratulations** for the decision to practice Open Science and reproducible research!
1717

1818
There are different ways for you to get involved, for example (a) submit a manuscript to a journal conducting CODECHECKs, (b) suggest a CODECHECK as part of your cover letter, or (c) submit your workflow to the [community review process](/guide/community-workflow) right away if you have published a preprint or submit to a journal with an open review.
19+
If you want to reference the CODECHECK project in your manuscript, please use the citation provided at the end of this page.
1920

2021
## As a reviewer or editor
2122

@@ -43,6 +44,13 @@ We would be happy to list your journal of conference on our website after we hav
4344
CODECHECK is a community effort, and your help is welcome across all tasks that the team faces.
4445
If you want to write code to assisst codecheckers to conduct more effective reviews or to streamline the CODECHECK review processes, or if you want to help presenting and sharing CODECHECK's vision and educate others on code executability checks - then _please get in touch with the [CODECHECK team](/partners/) (see contact details below)_.
4546

47+
## As an event organiser
48+
49+
You would like to (co-)organise a CODECHECK event, such as a workshop or a codechecking exercise, to bring together codecheckers, authors, and editors?
50+
You want to introduce reproducibility checking into your community, maybe expand an existing event or series, such as a conference?
51+
_We got you covered!_
52+
Please check out our [event recipe](/guide/event-recipe) for a detailed guide on how to run a CODECHECK event.
53+
4654
## As a curious person or someone with a question - get in touch!
4755

4856
Contact the PIs and seniour codecheckers at [[email protected]](mailto:[email protected]).

‎guide/cover-recipe.odt‎

66.5 KB
Binary file not shown.

‎guide/cover-recipe.pdf‎

21.8 KB
Binary file not shown.

‎guide/event-recipe.md‎

Lines changed: 314 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,314 @@
1+
---
2+
layout: page
3+
title: "CODECHECK Workshops - A recipe for organizers"
4+
permalink: guide/event-recipe
5+
# first version created from .docx file by Frank with:
6+
# pandoc CodeCheckNL_workshop_recipe.docx -t markdown -o CodeCheckNL_workshop_recipe.md
7+
#
8+
# CLI commands to turn this page into a nice PDF (using https://pdfcpu.io/getting_started/install):
9+
# wkhtmltopdf --margin-bottom 2cm --margin-top 3cm --margin-left 2cm --margin-right 2cm http://localhost:4000/guide/event-recipe temp.pdf
10+
# pdfcpu stamp add -mode image -- "../logo/codecheck_logo.png" "scalef:0.1, rot: 0, pos:tr, off: -20 -20" temp.pdf temp.pdf
11+
# pdfcpu merge recipe.pdf cover-recipe.pdf temp.pdf
12+
# pdfcpu stamp add -mode text -- "Page %p of %P \n https://doi.org/10.5281/zenodo.15423186" "fontname:Helvetica, scale:0.4 abs, pos:bc, margins: 0 0 20 0, rot:0" recipe.pdf CODECHECK-event-recipe.pdf
13+
---
14+
15+
This document contains all the lessons learned, do's and don'ts of the four workshops conducted as part of the [CODECHECKing goes NL (CHECK-NL) project](https://codecheck.org.uk/nl/).
16+
Its aim is to support everyone who wants to try a CODECHECK workshop and build/support local communities for reproducibility checks.
17+
18+
> **Cite this document**
19+
>
20+
> Ostermann, F., Gawehns, D., Momin, A., Sharma, S., Sun, J., Belliard, F., Eglen, S., & Nüst, D. (2025). CODECHECK Workshops - A recipe for organizers. CODECHECK Community on Zenodo. <https://doi.org/10.5281/zenodo.15423186>
21+
22+
## Workshop ingredients
23+
24+
- At least 2 enthusiastic organizers (one local and one experienced codechecker)
25+
- At least a dozen participants
26+
- Roughly 1 code submission per 4 participants
27+
- A location with flexible set-up
28+
- Up to a day for cooking
29+
30+
## How to prepare
31+
32+
### 1. Decide on your aims
33+
34+
a. Who is your audience: classroom (students, early career research training), research group, general or domain specific university audience? Part of a course or a curriculum, or broader target audience (university, or domain)?
35+
36+
b. Can you offer certificates for attendence/graduate school credits?
37+
38+
c. Introduction to concepts of reproducibility and codechecking necessary, or advanced tips and skills?
39+
40+
d. One-shot or community-building?
41+
42+
### 2. Check your available resources
43+
44+
a. Who can take care of logistics?
45+
46+
b. Which rooms are available?
47+
48+
c. Is there catering you can offer?
49+
50+
d. Can you offer travel grants?
51+
52+
### 3. Decide on the flavour
53+
54+
Based on your responses, there are different types or flavours of workshops possible:
55+
56+
- A _full-day workshop_ with a broad (domain) scope and inclusive participation requirements (this is the type the project did).
57+
- A more _targeted workshop_, probably also shorter event (addressing specific challenges or career stages of participants).
58+
- Whatever you are willing to try out and _experiment_!
59+
60+
Below, we show the recipe for a full-day workshop (the top-most
61+
flavour), assuming that you have secured the resources for a room that
62+
can comfortably hold at least 20 participants, and there are funds for
63+
at least a shared sandwich lunch.
64+
65+
### 4. Preparation timeline
66+
67+
**As early as possible:** contact/recruit a local organizer (in case
68+
that isn't you yourself), who knows the venue and how to secure access
69+
to the room and catering (yes, this seems very logical, but still
70+
important to recall), and together with them, decide on a date.
71+
72+
Ideally **three months before** the workshop: Create the workshop
73+
organizing committee. While it is possible to do everything yourself,
74+
you probably shouldn't. In our experience, you want to cover the
75+
following roles:
76+
77+
- Local organizer: as mentioned above, knows the venue and can arrange access etc.
78+
- Communications lead: creates the registration environment, publishes the advertisements, communicates with (prospective) participants
79+
- Content lead: knows how to do a codecheck (has at least one published) and is comfortable presenting the concept and ideas behind it
80+
81+
Further, why not reach out to the [CODECHECK team](/get-involved/) to invite a
82+
(remote) guest speaker and possible participation in discussion
83+
(especially if you feel more comfortable organising than taking the full
84+
content lead). We have done live CODECHECKing as part of a workshop. But
85+
this is entirely up to you! We are also happy to host a simple website
86+
for the workshop.
87+
88+
**Two months** before the workshop (earlier if you plan to do it during
89+
a particularly busy phase of the academic year): publish the
90+
registration page and the call for contributions. We used Eventbrite
91+
(see [Codecheck - Leiden
92+
University](https://www.universiteitleiden.nl/en/events/2025/02/codecheckatlucdh) for an example) for the registrations and an additional
93+
webpage with information on the call (e.g., [CODECHECK-NL Workshop for Digital Humanities on the CODECHECK website](https://codecheck.org.uk/nl/workshop4.html)). The details on the
94+
registration depend very much on the type of workshop you want to run.
95+
At the very least, the registration page should contain
96+
97+
- Directions to the venue
98+
- **Contact details** of the organizers; we used an new and dedicated
99+
Gmail address for this throughout the project; the experience was
100+
not without hassle because multiple people needed access to it; for
101+
a one-time workshop, we suggest to use the email addresses of the
102+
two organizer roles: the communication lead for any questions about
103+
logistics and organization, and the content lead for code
104+
submissions and questions about, well, content and codechecking
105+
- Tentative schedule
106+
- Information on catering
107+
- What to bring, e.g., their own laptop with admin rights (!) so they
108+
can set up any software they want to check - this is not the case
109+
for all employees
110+
- recall that from an AGILE workshop)
111+
- Expected outcomes (codechecks? Of participants or others?)
112+
- Options for travel grants or other certificates
113+
- Implications of no-show
114+
115+
On the latter, we did not have any. This proved to be easier in
116+
implementation, but might have led to an increased number of no-shows,
117+
which led to too much food waste. At least clearly point out that you
118+
will have excess food if people register but do not show up.
119+
120+
Be clear on the **expectations and roles** of the participants: What will
121+
they be working on? Can they just sit-in and listen (usually not), do
122+
they have to submit code to be checked to participate (no, but highly
123+
encouraged to do so!), can they skip part of the workshop (depends)?
124+
125+
For our workshops we had two roles: participants would either submit
126+
code to be checked, or join as codecheckers who would work on the code
127+
to be checked. The former would still be expected to attend the
128+
workshop, to enable a smoother process and increase chances of success.
129+
130+
If your aim is to generate codechecks (in addition to general
131+
community-building), depending on the domain or context, you might have
132+
difficulties getting submissions to be codechecked. Be very clear about
133+
the conditions, requirements, and purpose of these submissions. In our
134+
case, these were:
135+
136+
- Code creates the output of some published study (preprint, journal
137+
paper, citable report, etc.). No work-in-progress in its early
138+
stages (because the certificate needs to relate to a citable
139+
object).
140+
- All data is available and open.
141+
- At least one author of the work to be codechecked is present during the break-out groups.
142+
143+
But don't forget to point out the rewards:
144+
145+
- Improved code
146+
- Citable, linkable CODECHECK certificate
147+
- Depending on the publication, a Codecheck badge
148+
149+
Continue advertising, tagging relevant people etc., until you have
150+
exceeded the desired number of participants. If you do not get
151+
sufficient code submissions, start targeted recruiting in your network.
152+
153+
As a rule of thumb, about 3 to 4 participants per submission are a good
154+
ratio. A workshop codecheck can be done by a pair of participants or a
155+
group of 5-6, but the fun and the effectiveness are reduced.
156+
157+
**About one month** before the workshop, the deadline for submissions
158+
should close. Depending on your registrations so far, you might want to
159+
continue advertising or stop. If you haven't done so during the
160+
submission window in a rolling review, now is the time to check the
161+
submissions whether they are suitable for a codecheck. If you are in
162+
doubt, don't hesitate to reach out to us for advice! Have a look at
163+
codecheck.org.uk/nl, however please note that the project Gmail address
164+
is NOT monitored anymore! Please use our regular work e-mail addressess.
165+
166+
**About one to two weeks** before the workshop, send out a pre-workshop
167+
survey to learn more about the participants and possibly tailor the
168+
workshop accordingly. \[add on survey\]
169+
170+
## How to serve
171+
172+
For the workshop day itself, we recommend that the local organizer has
173+
the usual materials ready: textile-friendly **stickers and a marker** so
174+
that everyone can write down and wear their names; **outlets** and
175+
expansion cables for chargers; a **list of registered participants**;
176+
and if budget allows, some (laptop) **stickers** for the collectors
177+
among the participants.
178+
179+
A **typical workshop schedule** looked like this (in square brackets
180+
which role leads this part):
181+
182+
09:30 Open doors, welcome with coffee
183+
184+
10:00 Official start with round of introductions, information on
185+
schedule \[communications lead\]
186+
187+
10:15 Introduction to codechecking \[content lead\]
188+
189+
10:45 Live CODECHECK demo with Q&A on process \[content lead\]
190+
191+
11:15 Create breakout groups \[communications lead\]
192+
193+
11:30 Codechecks begin in breakout groups \[content lead acts as
194+
rotating facilitator\]
195+
196+
12:15 Lunch break
197+
198+
13:15 Codechecks in breakout groups continue \[content lead acts as
199+
rotating facilitator\]
200+
201+
15:15 Coffee break
202+
203+
15:30 Short reflection from the groups, discussion w/editors
204+
205+
16:30 Closing
206+
207+
We suggest to have a **collaborative document** (e.g., on HackMD) where
208+
information can be shared, including participants' names, affiliations,
209+
e-mail, and group.
210+
211+
An important aspect to consider is **how to form groups** for the
212+
break-out sessions. In our experience, several approaches can work, as
213+
long as you prepare them. The pre-workshop surveys can be helpful to
214+
decide. The two extremes are obviously to either assign all participants
215+
to a group or to have them choose. Our suggestion is to come up with an
216+
"ideal" group split based on the survey, and then let participants
217+
choose freely. If the groups become too unbalanced, use the "ideal"
218+
split as guidance to suggest balancing, e.g., ask all participants to
219+
reconsider joining a certain group, or ask specific participants to
220+
join. In any case, we suggest that at least one of the author(s) of a
221+
work (if present) joins the group codechecking it.
222+
223+
Some criteria for an "ideal" group composition:
224+
225+
- Familiarity with the computing environment/language
226+
- Expertise in the method/domain
227+
- In-group diversity with respect to computing environments (Windows, Linux, Mac, hardware performance) and overall expertise
228+
229+
If a group finishes early because the submitted code works flawlessly on
230+
most computing environments, it can be handy to have "spare" CODECHECKs,
231+
either from open issues from the CODECHECK website or internal from your
232+
organization.
233+
234+
In our experience, the set-up should enable participants to assume
235+
"ownership" of "their" codechecking quickly. We have not observed a
236+
single group being disinterested in tackling their problem in a
237+
collegial way. However, this engagement can decline quickly once the
238+
problem is solved, i.e., the code has been successfully checked. Then
239+
the nitty-gritty details of writing a short report and submitting it can
240+
be tricky to motivate. We recommend to emphasize that the report is
241+
written as much as possible during the CODECHECK itself, so that very
242+
little remains to be done after the completed CODECHECK.
243+
244+
For an overview of an adapted CODECHECK workflow, see the Appendix of
245+
this recipe.
246+
247+
During the **wrap-up**, each group should briefly present how their
248+
CODECHECK went. What problems have they encountered, where they
249+
successful? Then, a general discussion on the workshop and Codecheck in
250+
general can be helpful to exchange further views and experiences. We
251+
used Mentimeter as supporting tool during these discussion, but this is
252+
not a strong recommendation.
253+
254+
**Follow-up/left-overs**
255+
256+
As mentioned above, it is important to complete the CODECHECK as much as
257+
possible during the workshop. Otherwise, the organizers might end up
258+
chasing participants with e-mails to ensure that the CODECHECK is
259+
completed and finally registered on the Codecheck website. The content
260+
lead should be in charge here.
261+
262+
Lastly, once all the housekeeping is done, we suggest to write a nice
263+
blog post about the event and share it widely (at the very least with
264+
the participants) and advertise the successful CODECHECKs. Again we
265+
would be happy to crosspost it on the CODECHECK website.
266+
267+
**Questions?**
268+
269+
Don't hesitate to contact the [CODECHECK team](/team/) and also [people from the
270+
Codecheck-NL project](/nl/). Although the project has finished, we are all
271+
open science enthusiasts and will continue to be part of the community.
272+
Hope to have encouraged you to go for it -- Success!
273+
274+
## Appendix: Codecheck workflow during workshop
275+
276+
1. **Authors** create a pre-producible workflow: all data and code,
277+
plus a readme file detailing the content, a manifest file detailing
278+
the output ([CODECHECK configuration file specification](https://codecheck.org.uk/spec/config/1.0/)), and a
279+
license file; this is ideally bundled in a single repository or
280+
archive file and accompanied by a (pre-published) paper
281+
282+
2. **Authors** send their request for a CODECHECK to the advertised contact e-mail address
283+
284+
3. The **workshop team** accepts the request for the workshop or advises to follow the normal community workflow
285+
286+
4. During the workshop, **codecheckers** download materials or clone a repository
287+
288+
5. The workshop **codecheckers** create a new directory in their
289+
working environment where all new files go, and start documenting
290+
the ongoing CODECHECK; exact form of codechecking procedure can vary
291+
greatly; there are some optional tools, such as an R package that
292+
includes an Rmd template for the report and automates some steps,
293+
and templates for word processors to use
294+
295+
6. During a CODECHECK, the workshop **codecheckers** can ask the
296+
**authors** (if present at the workshop) in case of encountered
297+
problems, keeping in mind the general Codecheck philosophy
298+
(especially "the codechecker records but does not fix" -- unless it
299+
is a very trivial bug like pathnames)
300+
301+
7. The **codecheckers** summarize the process and outcome in a report
302+
and bundle it with all input and output files; this workshop
303+
CODECHECK bundle is then shared with the **workshop team** via email
304+
or repository; the report should at least contain the information on
305+
_who_ checked _what_ and _how_ with _which outcomes_; document for
306+
future self and other researchers; have a look at the available
307+
reports; most contain also optional information (compare [CODECHECK community workflow guide](https://codecheck.org.uk/guide/community-workflow#codecheck-steps))
308+
309+
8. The **workshop team** checks the bundle and report, and together
310+
with the workshop codecheckers, revise where necessary; once ready,
311+
either the **workshop team** or a corresponding **codechecker**
312+
upload the file on Zenodo or OSF.
313+
314+
9. The **workshop team** adds the new CODECHECK to the register in a new issue using this link: <https://github.com/codecheckers/register/issues/new?template=new-community-codecheck.md>.

‎nl/index.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,6 +28,8 @@ For updates, watch this space or follow us on the channels listed below!
2828

2929
### Latest News
3030

31+
- Our **event recipe** for running a local CODECHECK event is now published [on Zenodo](https://doi.org/10.5281/zenodo.15423186) and on the CODECHECK website [here](/guide/event-recipe)
32+
3133
- Join us at an **additional tutorial at AGILE 2025 conference**, see more information on the [event page](/nl/agilegis-2025.html).
3234

3335
- The final workshop of the CHECK-NL project took place on **14 February** at Leiden University. More details on our [workshop page](/nl/workshop4.html).

0 commit comments

Comments
 (0)