Consider a "noindex" key

[libremente]

In the last couple of months we faced situations where the publiccode.yml:

• is a "placeholder" filled with Lorem Ipsum;
• is under construction (not totally filled);
• has been copied from another repo and the information contained inside are not related to the project;
• is a test repo;
• other possibilities.

As such, I wonder if it could be feasible to add a decentralized mechanism to tell the spiders not to index such a file.
I was thinking at something like a boolean "noindex" key which simply states whether the repo should be indexed or not yet.
What do you think about this?
I think that acting in a centralized way from the spider perspective (e.g., using something like a blacklist) is harder since it requires the intervention of the spider's responsibles who are external from the software publishers group, it does not scale, it is very error prone and it requires a communication channel to be established at least twice.

The main risk is of course that someone could leave the value to "true" and such a software would never be indexed.

What do you think?

[bfabio]

Here's my two cents. As I see it, either the file is willingly incomplete or its contents are wrong.

In the first case (fe. placeholder, under construction, test repo) the developer doesn't want the file to be crawled yet, and they should be able to express that, but in the second case (fe. copied from another repo or plain wrong info) the file should just be fixed.

I'd KISS and just hint in the documentation to rename the file to something like publiccode.yaml.wip or whatever if it's not ready to be crawled.

[libremente]

In the first case (fe. placeholder, under construction, test repo) the developer doesn't want the file to be crawled yet, and they should be able to express that, but in the second case (fe. copied from another repo or plain wrong info) the file should just be fixed.

Yeah, I discarded the filename change in the first place since that always implies changing all the places where that file is used (I can think of a CI calling the validator already in use in some repos).
Furthermore, by keeping the publiccode.yml name consistent and adding the proposed key a crawler could also keep track of all the non-yet-indexed repos and that could be valuable (e.g., we could feed the issue-opener tool with such a list and automatically open a reminder issue).
But again I'm just thinking aloud and I haven't made up my mind yet, thanks for your cents 😄

[bfabio]

issue-opener

I didn't know such a tool was in the works, nice!

What about using a robots.txt file in the repo's root directory? It's a well known format and we could reuse the same solution in similar circumstances, both present (projects_settings.yml?) and future.

User-agent: Golang_italia_backend_bot
Disallow: /publiccode.yml