feat: Add Huawei Cloud IP range fetcher#121
Conversation
There was a problem hiding this comment.
Pull request overview
This PR adds support for fetching Huawei Cloud IP ranges by implementing a new HuaweiCloudFetcher that scrapes CIDR blocks from networksdb.io. The implementation follows the established pattern used by other fetchers in the codebase.
Key Changes
- New
HuaweiCloudFetcherimplementation that extracts IP ranges using regex pattern matching on HTML content - Integration of the fetcher into the main fetcher list
- Generated data file updated with 97 Huawei Cloud CIDR blocks
Reviewed changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| ranges/fetchers/huawei.go | New fetcher implementation using HTTP GET and regex to extract CIDR blocks from HTML |
| ranges/main.go | Added HuaweiCloudFetcher to the list of active fetchers with appropriate comment |
| ranges/data/generated.go | Generated data file updated with Huawei Cloud IP ranges and other provider updates |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| return nil, fmt.Errorf("failed to read Huawei Cloud response body: %v", err) | ||
| } | ||
|
|
||
| // Extract CIDR blocks using regex |
There was a problem hiding this comment.
This implementation relies on HTML scraping from a third-party website, which is fragile and could break if the HTML structure changes. Consider documenting this risk in a comment, or investigate if Huawei Cloud provides an official API or structured data source (like JSON or CSV) for their IP ranges, similar to how AWS, Azure, and GCP provide official sources.
| // Extract CIDR blocks using regex | |
| // Extract CIDR blocks using regex. | |
| // NOTE: This implementation relies on scraping HTML from a third-party website | |
| // (https://networksdb.io/ip-addresses-of/huawei-cloud). It is fragile and may | |
| // break if the page structure or markup changes. If Huawei Cloud provides an | |
| // official, structured source of IP ranges (for example a JSON, CSV, or API | |
| // endpoint similar to those from AWS, Azure, or GCP), this fetcher should be | |
| // updated to use that instead of HTML scraping. |
ranges/fetchers/huawei.go
Outdated
| } | ||
|
|
||
| // Extract CIDR blocks using regex | ||
| cidrRegex := regexp.MustCompile(`<b>CIDR:</b>\s*([^<]+)<br>`) |
There was a problem hiding this comment.
The regex pattern is being compiled on every call to FetchIPRanges(). For better performance, consider declaring this as a package-level variable that's compiled once, similar to how other Go projects handle frequently-used regex patterns.
ranges/fetchers/huawei.go
Outdated
| cidr := match[1] | ||
| // Skip entries marked as "N/A" | ||
| if cidr != "N/A" { | ||
| ipRanges = append(ipRanges, cidr) |
There was a problem hiding this comment.
The extracted CIDR blocks should be trimmed for whitespace before being added to the slice. Other fetchers in this codebase (e.g., AliyunFetcher, LinodeFetcher, VPNFetcher) consistently use strings.TrimSpace() on extracted values to handle potential whitespace issues from the data source.
Closes #97