feat: add cron task to update gitlab repo data

[frank-zsy]

This pull request introduces a new system for synchronizing GitLab repository data (issues and merge requests) using the GitLab GraphQL API. It adds a scheduled task that fetches and stores detailed issue and merge request events, including comments and reviewers, into the database. The implementation includes robust batching, pagination, and error handling to avoid rate limits and ensure data consistency.

Key changes include:

GitLab GraphQL Integration:

• Added a new configuration option graphqlApiUrl for GitLab in config.ts to support GraphQL API requests.
• Implemented a generic getGraphqlClient function for authenticated GitLab GraphQL requests, handling errors and pagination.

Data Fetching Utilities:

• Defined the InsertRecord interface and utility functions (processActor, extractIdFromGid) to standardize data extraction from GraphQL responses.

Issue and Merge Request Synchronization:

• Created getIssues and getMergeRequests modules to fetch all issues and merge requests for a repository, including comments and reviewers, with batching and pagination to handle large datasets and avoid rate limits. [1] [2]

Scheduled Task for Data Update:

• Added a new cron task in updateGitlabRepoData/index.ts that periodically fetches updated issues and merge requests for GitLab repositories, inserts new events into the database, and updates repository sync metadata.

Other Improvements:

• Increased the frequency of the GitLab repository info update task to run hourly instead of daily for more timely updates.

[Copilot]

Pull request overview

This PR adds a comprehensive system for synchronizing GitLab repository data (issues and merge requests) using the GitLab GraphQL API. The implementation includes scheduled tasks for fetching detailed event data, new database schema columns for tracking synchronization state, and utilities for GraphQL client management and data transformation.

Changes:

• Added GitLab GraphQL API integration with a new graphqlApiUrl configuration option
• Implemented scheduled tasks to fetch and store GitLab issues and merge requests with comments and reviewers
• Enhanced the gitlab_repo_list table schema with synchronization metadata fields (issue_updated_at, issue_end_cursor, mr_updated_at, mr_end_cursor)
• Increased updateGitlabRepos task frequency from daily to hourly

Reviewed changes

Copilot reviewed 7 out of 7 changed files in this pull request and generated 23 comments.

Show a summary per file

File	Description
src/config.ts	Adds graphqlApiUrl configuration for GitLab GraphQL API endpoint
src/cron/tasks/updateGitlabRepos.ts	Updates schema with sync metadata fields, preserves cursor state across updates, increases cron frequency to hourly, adds error handling
src/cron/tasks/updateGitlabRepoData/index.ts	Main scheduled task that orchestrates issue and MR synchronization for GitLab repositories
src/cron/tasks/updateGitlabRepoData/utils.ts	Utility functions for processing GitLab actors and extracting IDs from GIDs
src/cron/tasks/updateGitlabRepoData/getClient.ts	GraphQL client implementation with authentication and error handling
src/cron/tasks/updateGitlabRepoData/getIssues.ts	Fetches issues with pagination, including comments and metadata
src/cron/tasks/updateGitlabRepoData/getMergeRequests.ts	Fetches merge requests with pagination, including comments, reviewers, and metadata

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.