Statistics ideas for master's thesis

[jonkri]

I'm about to do a master's thesis in Software Engineering. I would like to apply (Bayesian) statistics and, ideally, conduct some kind of experiment. I posted a message on Haskell-Cafe about it yesterday. I have also asked the Hackage administrator to see if I could have access to the Hackage metadata.

I was wondering if you have any suggestions for statistical questions that I could look into that would be of interest from a PVP point of view, for example some kind of analysis related to dependencies or breakages.

Thanks!

[hasufell]

I think it would be interesting to know:

• how many maintainers violate PVP (e.g. missed a major bump despite API breaking changes)... also mind the corner case of re-exporting other packages API, which is a disaster in its own
• how many maintainers do lazy major bumps although there was no API breakage (I'm looking at you @michaelpj 😁)
• come up with some vague estimations about man-hours spent on updating one's package for one dependency (major bump) and then calculate the total amount of man-hours wasted in the entire ecosystem per, say, year (bonus points if you include GHC)
• what are the most common bump patterns (for both major and minor)
• what do people use the 4th and 5th etc. version components for

All the things I proposed kinda require to also have an understanding of the API of the package, not just the metadata.

I'm not sure that's within your scope. But it can be done statically.

[jonkri]

Very interesting! Thank you, @hasufell!

I wonder what would be a good way of determining the API of packages. 🤔 Could GHCi's :browse/:browse! command suffice, perhaps? Or would I need to dig deeper, perhaps getting into parsing .hi files?

[ulysses4ever]

@jonkri at the Cabal project, we are looking into API checking to ensure PVP based on https://github.com/Kleidukos/print-api This package is in early development, so be warned :-) There are downsides to it (but they're probably inherent to any tool based on GHC API), which you can read about here: haskell/cabal#10259

[jonkri]

@ulysses4ever: Thanks for letting me know about print-api!

Since I'm interested in analyzing API changes over time, I wonder how far back print-api and Cabal could go.

For example, do you think it would be possible to use modern Cabal to build old packages such as OpenGL-2.1 from 2006 (assuming C headers could be provided for FFI)?

$ cabal update hackage.haskell.org,2006-11-02T14:21:52Z
$ cabal get OpenGL-2.1
$ cd OpenGL-2.1
$ cabal install
Warning: Requested index-state 2006-11-02T14:21:52Z is newer than
'hackage.haskell.org'! Falling back to older state (2006-11-02T14:21:40Z).
Error: cabal: Could not resolve dependencies:
[__0] trying: OpenGL-2.1 (user goal)
[__1] next goal: OpenGL:setup.Cabal (dependency of OpenGL)
[__1] rejecting: OpenGL:setup.Cabal-3.10.3.0/installed-3.10.3.0 (conflict:
OpenGL => OpenGL:setup.Cabal>=1.0 && <1.25)
[__1] fail (backjumping, conflict set: OpenGL, OpenGL:setup.Cabal)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: OpenGL, OpenGL:setup.Cabal

I'm not sure what “OpenGL:setup.Cabal>=1.0 && <1.25” means. Is that a constraint on the version of Cabal?

[fgaz]

Yes, that's a constraint on the version of Cabal (the library) used to build the setup script (Setup.hs). It isn't specified in the OpenGL .cabal file, so it defaults to <1.25.

You can use a newer cabal-the tool to build packages that have a setup script requiring an older Cabal up to a point (depending on GHC). If you use a newer index-state (probably because of the OpenGL revision, but I'm not sure how exactly), you get a better error that includes this line:

constraint from minimum version of Cabal used by Setup.hs requires >=3.12

So that package is past the cabal-install+GHC → Cabal compatibility window.

The compatibility window is determined by a common lower bound of 1.20 plus a bound based on the GHC version you are using.

  -- GHC 8.2   needs  Cabal >= 2.0
  -- GHC 8.0   needs  Cabal >= 1.24

So you might be able to build the package by using GHC<8.2.

...or you could try to allow a newer Cabal with --allow-newer=OpenGL.setup:Cabal.

[gbaz]

There is some prior art on some statistics in this old IFL paper. The field is sometimes called "empirical software engineering" or more specifically "mining software repositories" -- would be nice to have continued work in this regard:

https://ifl2014.github.io/submissions/ifl2014_submission_14.pdf

There is also a paper on stackage which is interesting as well: https://arxiv.org/abs/2310.10887

[ulysses4ever]

@jonkri software archeology is hard in general, and Haskell doesn't make it much easier. For one particular package, you could probably put some effort and build it way back: you'll probably have to set up older GHCs, as noted above, and for some of those you'll need an older GLIBC, and, for that, perhaps, an older OS altogether. Having native dependencies (like with OpenGL) makes it harder. Finally, doing it at scale (e.g. tens or hundreds or thousands of packages) --- I doubt it. Again, this is a hard problem in general. Depending on your goals, you may be better off by picking a more syntactic approach that wouldn't require you to compile code before you can analyze it.

[jonkri]

Thanks, @gbaz and @ulysses4ever!

In order to identify API changes, do you think it would make sense to crawl the Haddock pages on Hackage, or Hoogle databases, and identify the package APIs from there? 🤔

I'm currently running the following naive script to collect print-api types for all LTS 22.40 packages (including previous versions) that build successfully with GHC 9.6.6 from a haskell:9.6.6 Docker image. $1 is the package and $2 is the version.

#!/bin/bash
set -e

echo "Fetching upload time for $1-$2..."
curl -s https://hackage.haskell.org/package/$1-$2/upload-time > /out/$1-$2-upload-time
sed -e '$a\' -i /out/$1-$2-upload-time

upload_time=$(cat /out/$1-$2-upload-time)

echo "Downloading Cabal index for $upload_time..."
cabal update hackage.haskell.org,$upload_time

echo "Extracting $1-$2..."
mkdir /package
tar -C /package -f package.tar.gz --strip-components=1 -xz
cd package

echo "Building $1-$2..."
cabal build --write-ghc-environment-files=always

echo "Extracting API from $1-$2..."
print-api -p $1 > /out/$1-$2-api

So far I've basically gotten through all packages starting with an upper case letter and gotten around 300 print-api specifications.

Edit: Mentioned Hoogle databases.

[jonkri]

Looking at the Hackage package index now (in parallell to what's being discussed above). Is there anything from the Cabal metadata that you would be interested in knowing? For example, would it be interesting to know to what extent (upper or lower) bounds are used in build-depends?

[jonkri]

@gbaz: Thanks again for the papers, they were an interesting read! Regarding the first paper, did you have anything specific in mind when you said it would be nice to have continued work in this regard?

[jonkri]

@hasufell:

come up with some vague estimations about man-hours spent on updating one's package for one dependency (major bump) and then calculate the total amount of man-hours wasted in the entire ecosystem per, say, year (bonus points if you include GHC)

Perhaps this isn't what you meant, but I'm wondering if work related to breakages really should be seen as waste. I'm thinking that packages breaking to some extent can be seen as a natural evolution of a healthy and innovative package ecosystem, and that it's a balance. Either way, I guess it could be useful to have an estimate of the time it takes to fix a breakage. I guess the most straight-forward way to measure this would be to measure the time it takes until a package which fixes the breakage is released.

[jonkri]

Here are the research questions I've considered have so far:

• How frequently are major and minor versions bumped?
• What are the fourth and fifth version components used for?
• To what extent, and how, are dependency version bounds (both lower and upper) used?
• To what extent, and how, are lock files used (checked into version control)? How up to date are the lock files? How often are lock files updated?
• To what extent, and why, are major versions bumped (unnecessarily, against the PVP) when there are no breaking changes?
• To what extent, and why, are minor versions bumped (incorrectly, against the PVP) when there are breaking changes?
• How could a tool for automatically detecting versioning problems (e.g, a Cabal or Stack feature, or a GitHub Actions extension) work?
• How long are packages remaining outdated when there are breaking changes in dependencies?
• How much work is required to adopt to breaking changes in dependencies?
• What effect does re-exporting other package APIs have on breakages?

Edit: “Broken” replaced with “outdated” and “fix” replaced with “adopt to”.