Metrics for community partnership landing pages to track health

[lwasser]

website PR with the actual website updates for the screenshots below are here. but please leave comments here about metrics!

Astropy and other affiliated packages partnerships -

Landing page with community partner buttons

First - the landing page with new community specific buttons at the top leading to a community partner page

Search feature to find astropy affiliated packages on our main python-packages landing page.

Then we have the landing page where someone might search for astropy to find related packages.

Community landing page

I envision something like this (styles can be fixed) that listed only the affiliated packages. A few notes

1. i created a placeholder graphic at the top for astropy - we can change it it's just a mockup.
2. We should discuss in more detail what metrics we want to track here. I was thinking it would be nice to be able to sort packages by varying metrics.
3. Based upon what we decide above, we then could create a xml feed that shared data associated with a select group of packages to populate another website. we'd need to agree on what items would be most useful to share in that feed.

What metrics would be most useful for us to collect?

A starting list includes:

• pony factor (calculated in the scientific python devstats)

• of contributors (over time??)

• last commit date
• last release date and value
• published on conda-forge: boolean
• something like average pr's merged a year?
• monthly pr's opened?

repo level metrics that pyos wishes to track

• build system / tool used (repo review?)
• using pyproject toml
• others

[rebeccaringuette]

Suggest adding date of last package install/download as a measure of how often the community is using the package.

[pllim]

"last package/install download" is also affected by automated CI, so it depends on if you count "CI" as community or not.

[rebeccaringuette]

Isn't automated CI triggered by a person or action, such as a user-generated PR? If there is a way to separate automated installations/downloads from others, then both could be used separately. Otherwise, the behavior of the metric over time would be a good indicator of activeness/usefulness, even if the package isn't being updated.

[lwasser]

thank you both for this. i was thinking that looking at average downloads a month might be a good representation of ongoing use. it would be easy to track conda-forge and pip and that way we could view overall volume of downloads over time (knowing that some of these are CI related). Snyk does this with weekly downlaods.

[pllim]

Back to the original post, I am not sure if I find one button per health category useful. I was envisioning an at-a-glance table where you see all the metrics for multiple packages all at once; something like http://www.astropy.org/astropy-dashboard/status.html but not the same metrics.

But then again, perhaps there would be too many things to fit in a single table. So, the most important ones for users would be in table, but if you click on "more" for a package, it would give you everything?

For example, if I am browsing as a user, I want to know what is the latest release and when, and where to find the package and doc. And maybe if CI is passing. The pony factors and number of contributors would be secondary, but these details would be useful for, say, reviewers.

Everything in "devstats" (e.g., https://devstats.scientific-python.org/_generated/astropy/index.html) is secondary.

Everything in "repo-stats" (e.g., https://scientific-python.github.io/repo-review/?repo=astropy%2Fastropy&branch=main) is also secondary.

The primary stuff I mentioned above could be pulled directly from places like PyPI and CI provider, though different packages may use different CI providers.

[lwasser]

@pllim i'm following up on this. i'm sorry i somehow totally missed this comment. what you are suggesting in many ways (from a web dev perspective) is easier to implement but also in some ways harder.

The harder part is how would we know what badges we want? as you have seen with astropy the badges go out of date. so say someone moves from github actions for tests to gitlab. how would we know that? similarly docs - they could be using readthedocs or they could have a github action that is only building mkdocs. so how would we automate capturing that information is my question.

creating that table would be easy with the data we have now from a web dev perspective. but the data part is harder.

some things like pypi downloads, conda downloads - no problem. i can grab that info. similarly last commit date i am already grabbing. let's revisit this convo so i can pull together a proof of concept

[lwasser]

So I wanted to follow up on this as we now have a metrics dashboard.

Here is a page that has some metrics broken out by package. We could easily create (or open an issue to create) a page that is astropy-specific. @pllim, I'm curious about your take on this, given your vision for what it might look like. We do have the ability to create tables that you can sort, like this

[lwasser]

If we decide that metrics page works, then we can link to it on the astropy landing page here as well and move this issue to the metrics repo for further discussion.

[pllim]

@lwasser thanks for the update! Looking good, but I have some follow-up questions:

• Astropy page will only have metrics for the Astropy packages, I assume? But I don't see how to filter them out in the current implementation at https://www.pyopensci.org/metrics/peer-review/accepted-packages.html#detailed-pyopensci-metrics
• I would prefer to get rid of forks_count and stargazers_count, and instead use that real estate for things like "last commit date", "last release/tag date", and possibly "open PRs count" unless "open issues count" also include PRs since PR technically is subclass of Issue on GH?
• How do you pull these stats for a small number of packages not on GitHub?