Recurring tasks are managed by Sidekiq via the sidekiq-cron plugin. See config/schedule.yml.
- ArticleImporter: Imports articles based on mediawiki page id or title.
- UploadImporter: Imports files uploaded to Wikimedia Commons.
- UserImporter: Imports user data.
Imported data is persisted in the database.
- AverageViewsImporter: Updates average view counts for articles.
- RatingImporter: Updates article ratings.
Imported data is persisted in the database.
- CategoryImporter: Imports article titles from a specified category.
- PagePileApi: Imports article titles from a pile id.
- PetScanApi: Imports article titles from a pet scan id.
- TransclusionImporter: Imports article titles that transclude a given page.
Imported data is persisted in the database.
Since the data-rearchitecture deployment, the Dashboard no longer stores data for each individual wiki revision in the database. Instead, it collects data for revisions within a certain timeframe and stores only the aggregate statistics for each time period (what we call timeslices).
- RevisionDataManager: Fetches revisions and corresponding scores (it invokes the RevisionScoreImporter).
- RevisionScoreImporter: Fetches revision scoring data from Lift Wing and reference-counter APIs.
- WMFLabsTools: Source for all revision and article data except view counts
- MediaWiki API: Data about uploads to Wikimedia Commons, revision metadata, and user information. All or nearly all data from WMFLabsTools could be reimplemented to pull directly from MediaWiki, and we need to use MediaWiki directly when it is important to fetch up-to-date data, since the replica database used by WMFLabsTools may have replication lag.