[WIP] Rankings Backend (Go)

Rankings Project Structure

• Frontend (React/ViteJS):

  • Live: https://rankings.polinetwork.org/
  • Repo: https://github.com/polinetworkorg/Rankings

• Backend: (newer is first)

  • Go: this repo (WIP)
  • C#: https://github.com/PoliNetworkOrg/GraduatorieScriptCSharp (stable)
  • Py: https://github.com/PoliNetworkOrg/GraduatorieScript (deprecated)

• Data: https://github.com/PoliNetworkOrg/RankingsDati

Usage

Note

The following instructions should be stable, but if they are not working anymore, please open an issue.

There are 4 total commands (ATTOW):

• scraper: perform scraping of rankings html files and school manifests against Polimi website
• parser: perform parsing of raw html files into custom data shapes, output as JSON
• playground: for testing purpose only, especially useful when dealing with JSON encoding/decoding. Do not expect this package to last forever.
• migrate: made to convert old html folder structure to the new one. See 2b99e43, 3f57469, 9008f83 commits for more details

The most common scenario is the following:

1. Run scraper

   go run ./cmd/scraper -d ../RankingsDati/data

2. Run parser

   go run ./cmd/parser -d ../RankingsDati/data

Important

If you don’t provide the -d (--data-dir) argument to either the scraper or parser commands, they will default to using the temporary folder ./tmp.
To understand why we are passing a data folder from another repository, check the C# README.
Note that for the purpose of using this script, it is possible to use a folder inside this project (e.g. ./data), but it is not recommended.

You can change the log level with the LOG_LEVEL env variable (debug/info/warn/error). Example:

LOG_LEVEL=error go run ./cmd/parser