Replicated, Reproduced and functional?

[jon-bell]

Currently, there is some confusion on the "Results Replicated" and "Results Reproduced" badges. The process for reviewing, reporting, and retroactive badge awarding needs to be clarified. FSE 2018 and 2019, ICSE 2019 and 2020 do not award the "Functional" badge. Is this a good practice? Should we generally move away from this badge?

[sbaltes]

Are there any cases where "Results Replicated" was actually awarded?
I think reproduction is possible during the review process (but may be time consuming).
However, replication would require reviewers to execute an analysis, for example, using a new dataset. I don't think that is possible during artifact evaluation.

[jon-bell]

I'm not sure. I think that a related question is: what exactly are the minimum criteria for meeting the reproduced or functional bars?

[wagnerst]

I saw that happening once. If the paper was available as a pre-print, it could be already replicated.
But maybe we need ways to apply for that badge even later on.

[jkrinke]

I just had an email from Tim Menzies about the ICSE2020 track and which made me think about the Replication / Reproduction Badges. Good replications and reproductions need a lot of work and we as a community should support the effort, for example, by R&R tracks. If such tracks become more widespread, then awarding the badges become straight forward: the authors of a replicated paper receive the badge when the replicating paper appeared (in an R&R track).

[jon-bell]

@jkrinke What about the model where: Author A writes an ICSE2020 paper. Author A publishes an artifact that allows for replication of this paper. Artifact evaluation committee independently replicates the paper result using the provided artifact, awards a badge?

[jkrinke]

A proper replication is too much work for an assigned replicator. A good replication will often lead to a paper on its own and it is the replication that should be honoured, not the replicated work. A committee will only be able to do the most simple work - does the provided artefact run? But that is not the point of a replication, as the provided artefact may not be what the paper describes. Thus, a proper replication would have to re-implement the approach by the description given in the paper. I believe the act or replication is of a great value to the community and we usually learn something else during the replication. See, for example, the programming language dispute at the moment.

I had my fair share of artefacts that were not what the paper promised, but figuring that out usually took weeks of investigation.

[jon-bell]

@jkrinke In my interpretation, you are describing a reproduction (re-implementing the approach), based on ACM's definitions:

Replicated: Available + main results of the paper have been obtained in a subsequent study by a person or team other than the authors, using, in part, artifacts provided by the author.

Reproduced: Available + the main results of the paper have been independently obtained in a subsequent study by a person or team other than the authors, without the use of author-supplied artifacts.

How would you distinguish between a replication and a reproduction?

[jkrinke]

True, I was under the impression that Claerbout's terminology is used.

I have now read the ACM's definition and it requires a peer-reviewed publication reporting on the replication or reproduction.