"Artifact Available" badge guidelines

[jon-bell]

Currently, the publisher will apply the ACM "Artifact Available" badge to the camera ready of a paper if the authors submit a DOI for their artifact. This is in contrast to the rest of the badges that are given when the artifact evaluation committee evaluates an artifact. The publisher might check that the DOI is valid, but certainly will not check the contents of that artifact (e.g. it might be empty). Should there be a review process at all for "Artifact Available?" ICSE 2020 has one, ISSTA 2019 did not.

[sbaltes]

Related question: Is an artifact available when it is provided in a proprietary format that requires commercial software to open/execute it?

[piskachev]

Related question: Is an artifact available when it is provided in a proprietary format that requires commercial software to open/execute it?

I would say no. In my opinion "Artifact Available" should conform to open science policy. At least in EU this is important.

[dgraziotin]

As much as I am on the open side of things, I would also include freely available proprietary software as long as that particular version of proprietary software is made available for free in perpetuity (e.g.: archived and licensed). We need to take into account that a lot of research in SE is with companies.

[sbaltes]

What about, for example, coded qualitative data provided as proprietary MAXQDA files?
There is a free trial version of MAXQDA, would that be open enough?
(BTW: I'm guilty of publishing data in that format).