Retire the "Eclipse Foundation" IP service

[waynebeaton]

The Eclipse Dash License Tool uses a service on eclipse.org to find licence information from IPLab. I have to retire that service.

Rather than attempt to create an alternative service in its place, I'd like to explore having the tool just query directly against IPLab.

IP review results are structured in a well-defined manner, so using the GitLab API to query the file hierarchy and pull contents directly from the repository should be straightforward.

The wrinkle being that use of the GitLab API is rate-limited which means that there is a high probability that use of the tool will require a token. IMHO, this is a good thing. There is also some potential that, for legitimate use with large dependency lists, we might hit rate limits. This will require some careful testing.

[HannesWell]

The wrinkle being that use of the GitLab API is rate-limited which means that there is a high probability that use of the tool will require a token.

Most users of the license-check workflow already use a Gitlab API token, e.g.:
https://github.com/eclipse-platform/eclipse.platform.releng.aggregator/blob/f6fcbd64566301a5e726b888475ea0116ea2ff38/.github/workflows/licensecheck.yml#L25-L26

so this could then just continue to be used without changes, couldn't it?

[gomes89]

The wrinkle being that use of the GitLab API is rate-limited which means that there is a high probability that use of the tool will require a token. IMHO, this is a good thing. There is also some potential that, for legitimate use with large dependency lists, we might hit rate limits. This will require some careful testing.

Depending on the current settings, it may work without a token in most scenarios. A single run of the Eclipse Dash License Tool would involve 1 + N API calls: one request to the Projects API and N requests to the Repository files API (where N is the number of files to be pulled).

The default rate limits for unauthenticated API calls are as follows:

• General limit per IP: 3600/hour
• Projects API: 400/minute
• Repository files API:
  • Disabled for files <= 10MB
  • 5 calls per minute per object per project for files > 10MB

[waynebeaton]

@gomes89 I did a little experimenting, at it seems that the default rate limits as you've described are in place.

I cobbled together a quick proof of concept using the 1+N scenario that you describe. It obviously takes longer to run than the current implementation which batches requests, but it works quite well.

There are some things that we can consider to optimise, like using the repository API (treestream) to get an index of what's in the repository so that we can void calls that we know will result in 404. We could try doing some local caching with a lightweight DB or something... But let's keep it simple for now and worry about that later.

@HannesWell yes: it should just work for folks who already have a token. I'm thinking that the requirement to have a token is a good thing as it will force users to create an Eclipse account and bind them to our terms of use.