feat: implement recursive CRD discovery and robust GitHub URL parsing

[muhammadolammi]

Description

This PR fixes #881
This PR introduces a more resilient approach to model generation by implementing recursive tree walking and intelligent URL routing.

Testing Performed
A comprehensive test suite was added to package_test.go c:

[x] Standard git:// protocol with specific roots.

[x] GitHub Browser URLs containing /tree/ and /blob/.

[x] GitHub Release Tag URLs (e.g., /releases/tag/v1.0).

[x] Root-level discovery with auto-branch resolution.

[x] Handling of "messy" URLs with multiple slashes and trailing separators.

Notes for Reviewers
This PR depends on #885

Signed commits

• Yes, I signed my commits.

[gemini-code-assist]

Summary of Changes

Hello @muhammadolammi, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the model generation process by introducing a more resilient and intelligent approach to handling GitHub source URLs. It allows for recursive CRD discovery by supporting diverse URL formats, including browser and release links, and automatically resolves default branches when not explicitly provided. These changes, coupled with improved zip extraction security, contribute to a more robust and flexible system for fetching and processing CRDs.

Highlights

• Robust GitHub URL Parsing: The system now intelligently parses various GitHub URL formats, including standard git:// protocols, GitHub browser URLs (/tree/, /blob/), and GitHub release tag URLs (/releases/tag/). This ensures more flexible and accurate source identification.
• Automatic Default Branch Resolution: When a GitHub URL is provided without a specific branch, the system can now automatically resolve and fetch the default branch of the repository, improving usability and reducing manual specification.
• Enhanced Zip Extraction Security and Reliability: The zip extraction utility has been improved to prevent 'Zip Slip' vulnerabilities, ensuring files are extracted only within the intended directory. It also now skips macOS metadata files (._*, __MACOSX) for cleaner extractions.
• Comprehensive Test Coverage: New and expanded test suites have been added for GitHub URL parsing, default branch retrieval, and zip extraction, covering a wide range of scenarios and edge cases to ensure the robustness of the new features.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces a more robust mechanism for parsing various GitHub URL formats and implements recursive discovery for CRDs. A significant improvement is the rewrite of the ExtractZip function, which now correctly handles resource cleanup, prevents Zip Slip security vulnerabilities, and extracts files to the correct destination. The accompanying tests are comprehensive. My feedback focuses on a critical edge case in the new URL parsing logic that could lead to a panic, a minor typo, and a suggestion to improve test hermeticity by mocking network calls.

[gemini-code-assist]

This test makes real network calls to the GitHub API when a URL with only an owner and repo is provided (e.g., git://github.com/meshery/meshkit). This makes the test non-hermetic, slow, and potentially flaky, depending on network conditions and the state of the remote repository (e.g., if the default branch changes).

Unit tests should be self-contained and not rely on external services. The call to GetDefaultBranchFromGitHub within extractRepoDetailsFromSourceURL should be mocked.

One way to achieve this is to introduce a function variable in git_repo.go that can be replaced during tests.

In generators/github/git_repo.go:

// Add this variable at the package level
var getDefaultBranch = GetDefaultBranchFromGitHub

// In extractRepoDetailsFromSourceURL, change the call to use this variable
// ...
b, err := getDefaultBranch("https://api.github.com", owner, repo, http.DefaultClient)
// ...

Then, in generators/github/git_repo_test.go, you can mock it:

func TestExtractRepoDetailsFromSourceURL(t *testing.T) {
    // Mock the function
    oldGetDefaultBranch := getDefaultBranch
    getDefaultBranch = func(baseUrl, owner, repo string, client *http.Client) (string, error) {
        // For this test, we can assume the default branch is always "master"
        // to match the test expectations.
        if owner == "meshery" && repo == "meshkit" {
            return "master", nil
        }
        return "", fmt.Errorf("unexpected repo: %s/%s", owner, repo)
    }
    // Restore the original function after the test
    defer func() { getDefaultBranch = oldGetDefaultBranch }()

    // ... rest of the test code
}

This change will make your tests fast, reliable, and independent of external services.

[leecalcote]

@Yorisoft, would you like to give this PR a review?

[leecalcote]

@parthivsaikia, and you?

[Yorisoft]

@muhammadolammi
Gave the changes a look. Nothing immediately pops out as incorrect or needing fixing 👍

I would suggest less comments if possible: code should be self explaining.

As I'm still learning how things work I can't give a garuntee that this works as intended. I will give it another pass to validate. Hopefully you get others to review too.

[leecalcote]

This is a good item to add to the weekly Meshery Dev meeting agenda. You can add this item in the doc, attend, and present it. Meeting details at https://meshery.io/calendar.