feat: add a cron tasks to update gitlab repo list

[frank-zsy]

This pull request introduces a scheduled task to fetch and update GitLab repository metadata in the database, along with configuration changes to support GitLab integration. The main changes are grouped into GitLab integration and database/data pipeline updates.

GitLab integration:

• Added a new gitlab section to the configuration (config.ts) to store the GitLab API token and URL, enabling secure and configurable access to the GitLab API.

Database and data pipeline:

• Implemented a new cron task (updateGitlabRepos.ts) that:
  • Periodically fetches GitLab repository data using the GitLab API.
  • Ensures the gitlab_repo_list table exists in ClickHouse and is correctly structured.
  • Parses and saves repository metadata, including topics, tags, activity timestamps, and statistics.
  • Handles incremental updates by querying for the latest activity timestamp and fetching only new or updated repositories.

[Copilot]

Pull request overview

This pull request adds a scheduled task to periodically fetch and update GitLab repository metadata in ClickHouse, enabling the OpenDigger platform to track GitLab repositories alongside GitHub repositories.

Changes:

• Added GitLab API configuration (token and apiUrl) to config.ts
• Implemented a new cron task updateGitlabRepos.ts that fetches GitLab repository data via the GitLab API and stores it in ClickHouse
• The task performs incremental updates by tracking the latest last_activity_at timestamp

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 15 comments.

File	Description
src/config.ts	Adds GitLab configuration section with token and apiUrl fields to support GitLab API access
src/cron/tasks/updateGitlabRepos.ts	Implements cron task to fetch GitLab repository data, create ClickHouse table, and incrementally update repository metadata

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.