You have exceeded a secondary rate limit.

[alampros]

I was really looking forward to using this (thank you!) - we have a ton of repos and managing their actions is almost a full time job. Unfortunately, we have SO many repos that I couldn't even try GAMA out.

When it attempts to load the "Repository" tab, it shows a bunch of "You have exceeded a secondary rate limit." errors in the bottom pane.

For reference, I have access to around ~450 repos split amongst 4 organizations.

I hope I'll have some time soon to put together a PR, but if there's anything I can do to help (test a build?) in the meantime, let me know.

Thanks again!

[github-actions]

🤖 AI Issue Assistant

Thank you for bringing this issue to our attention. The GitHub API rate limit is a common challenge when working with large numbers of repositories. Here's a comprehensive analysis and potential solutions:

Root Cause Analysis

The current implementation of ListRepositories in the GitHub repository method is not optimized for handling a large number of repositories. Specifically:

1. The method attempts to fetch all repositories in a single request
2. It uses concurrent workers, which can trigger secondary rate limits
3. No pagination or intelligent fetching strategy is implemented

Recommended Solutions

1. Implement Proper Pagination

func (r *Repo) ListRepositories(ctx context.Context, limit int, page int, sort domain.SortBy) ([]GithubRepository, error) {
    var allRepositories []GithubRepository
    
    for currentPage := 1; ; currentPage++ {
        repositories, err := r.fetchRepositoriesPage(ctx, limit, currentPage, sort)
        if err != nil {
            return nil, err
        }
        
        if len(repositories) == 0 {
            break
        }
        
        allRepositories = append(allRepositories, repositories...)
        
        // Optional: Break if we've reached desired total or max pages
        if len(allRepositories) >= limit || len(repositories) < limit {
            break
        }
    }
    
    return allRepositories, nil
}

func (r *Repo) fetchRepositoriesPage(ctx context.Context, limit int, page int, sort domain.SortBy) ([]GithubRepository, error) {
    var repositories []GithubRepository
    err := r.do(ctx, nil, &repositories, requestOptions{
        method: http.MethodGet,
        paths:  []string{"user", "repos"},
        queryParams: map[string]string{
            "visibility": "all",
            "per_page":   strconv.Itoa(limit),
            "page":       strconv.Itoa(page),
            "sort":       sort.String(),
            "direction":  "desc",
        },
    })
    return repositories, err
}

2. Implement Rate Limit Handling

func (r *Repo) ListRepositories(ctx context.Context, limit int, page int, sort domain.SortBy) ([]GithubRepository, error) {
    var allRepositories []GithubRepository
    var retryCount = 0
    const maxRetries = 3
    
    for currentPage := 1; ; currentPage++ {
        repositories, err := r.fetchRepositoriesPage(ctx, limit, currentPage, sort)
        
        if err != nil {
            // Check for rate limit error
            if isRateLimitError(err) {
                if retryCount < maxRetries {
                    waitTime := calculateBackoffTime(retryCount)
                    time.Sleep(waitTime)
                    retryCount++
                    continue
                }
                return nil, fmt.Errorf("rate limit exceeded after %d retries", maxRetries)
            }
            return nil, err
        }
        
        // Reset retry count on successful request
        retryCount = 0
        
        // Pagination logic remains the same
        if len(repositories) == 0 {
            break
        }
        
        allRepositories = append(allRepositories, repositories...)
    }
    
    return allRepositories, nil
}

func isRateLimitError(err error) bool {
    // Implement GitHub-specific rate limit error detection
    return strings.Contains(err.Error(), "secondary rate limit")
}

func calculateBackoffTime(retryCount int) time.Duration {
    return time.Duration(math.Pow(2, float64(retryCount))) * time.Second
}

Additional Recommendations

1. Configurable Repository Fetching

   • Add configuration options to control:
     • Maximum number of repositories to fetch
     • Pagination strategy
     • Timeout settings

2. Incremental Loading

   • Implement a lazy loading mechanism for repositories
   • Show initial set of repositories quickly
   • Allow user to trigger additional loading

3. Caching Mechanism

   • Implement local caching of repository lists
   • Reduce API calls on subsequent runs

Testing & Validation

1. Test with various GitHub account configurations
2. Implement robust error handling
3. Add logging for rate limit and pagination events

Next Steps

1. Modify the current implementation
2. Add comprehensive error handling
3. Implement pagination and rate limit management
4. Add configuration options

Contribution Welcome

We welcome your contribution! If you're interested in helping, you can:

• Test the proposed solutions
• Provide feedback on implementation
• Submit a pull request with improvements

Would you be willing to help test a modified version of the repository listing method?

---

This analysis was performed by Issue Assistant. If you have any questions, please contact the repository maintainers.

[github-actions]

🏷️ AI Label Assistant

I've added the following labels to this issue:

• help wanted
• bug

Explanation:
This issue describes a technical problem with rate limiting when attempting to load repositories, which clearly indicates a bug in the application's functionality. The user is experiencing a specific error that prevents them from using the tool effectively. The 'help wanted' label is suggested because the user is offering to assist and seems open to providing more information or testing, which could help resolve the issue.

---

This label analysis was performed by Issue Assistant. If you have any questions, please contact the repository maintainers.

[canack]

Hello, thank you for your interest in GAMA.

In GAMA's current structure, our method for fetching repositories from GitHub is not very efficient. Any improvement in this area would be excellent 🙏

Actually, there's no other mechanism that needs to be tested besides this. Perhaps for future processes, we can build a cache mechanism for this repository tab. 🍀

[indranilvyasFlipp]

I would lIke to work on this, I tried using Gama and I too have multiple GitHub orgs that have way more than 100 repositories, paging repositories or use of GitHub App to by pass PAT rate limits will be awesome

[canack]

Thank you for feedbacks. ❤️

I'll make fixes and improvements about on it in shortest time 💯