feat(agent): add daydaymap search engine support#711
feat(agent): add daydaymap search engine support#711dogancanbakir merged 6 commits intoprojectdiscovery:devfrom
Conversation
Bumps [golang.org/x/crypto](https://github.com/golang/crypto) from 0.32.0 to 0.45.0. - [Commits](golang/crypto@v0.32.0...v0.45.0) --- updated-dependencies: - dependency-name: golang.org/x/crypto dependency-version: 0.45.0 dependency-type: indirect ... Signed-off-by: dependabot[bot] <support@github.com>
…bot/go_modules/golang.org/x/crypto-0.45.0 chore(deps): bump golang.org/x/crypto from 0.32.0 to 0.45.0
Add daydaymap as a new supported search engine with complete integration: - Add daydaymap agent implementation in sources/agent/daydaymap/ - Integrate daydaymap CLI option (-ddm/--daydaymap) in runner options - Add daydaymap API key configuration in provider and keys - Register daydaymap in available engines list - Support DAYDAYMAP_API_KEY environment variable The implementation follows the existing agent pattern and maintains consistency with other search engine integrations (greynoise, driftnet, etc).
|
Important Review skippedAuto reviews are disabled on base/target branches other than the default branch. Please check the settings in the CodeRabbit UI or the You can disable this status message by setting the WalkthroughA new Daydaymap data source agent is integrated into the uncover tool, adding CLI flag support, API client implementation with pagination, request/response structures, and key management configuration to enable querying the Daydaymap API alongside existing sources. Changes
Sequence Diagram(s)sequenceDiagram
participant CLI as CLI / Runner
participant Agent as Daydaymap Agent
participant API as Daydaymap API
CLI->>Agent: Query(session, query)<br/>with API key & search term
activate Agent
Agent->>Agent: Encode search term<br/>in base64
Agent->>Agent: Build DaydaymapRequest<br/>(page, page_size, keyword)
loop Pagination (until limit or empty)
Agent->>API: POST with JSON body<br/>+ api-key header
activate API
API-->>Agent: DaydaymapResponse
deactivate API
Agent->>Agent: Decode response<br/>validate Code field
Agent->>Agent: Extract IP, port, domain,<br/>service → construct URL
Agent->>Agent: Populate Result<br/>(IP, Port, Host, Url, Raw)
Agent-->>CLI: Stream Result<br/>via results channel
Agent->>Agent: Check: total results<br/>reached limit or<br/>page empty?
end
Agent-->>CLI: Close results channel
deactivate Agent
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~20 minutes
Poem
Pre-merge checks and finishing touches❌ Failed checks (1 warning)
✅ Passed checks (2 passed)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🧹 Nitpick comments (1)
sources/agent/daydaymap/daydaymap.go (1)
147-156: Consider using domain in URL when available.The URL is constructed using
result.IP, but whenresult.Host(domain) is available, it may be more useful to include it in the URL for better readability and context.// Extract title for URL construction if service, ok := daydaymapResult["service"]; ok && service != nil { serviceStr := fmt.Sprint(service) if serviceStr == "https" || serviceStr == "http" { - result.Url = fmt.Sprintf("%s://%s", serviceStr, result.IP) + host := result.IP + if result.Host != "" { + host = result.Host + } + result.Url = fmt.Sprintf("%s://%s", serviceStr, host) if result.Port > 0 && result.Port != 80 && result.Port != 443 { result.Url = fmt.Sprintf("%s:%d", result.Url, result.Port) } } }
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (7)
runner/options.go(7 hunks)sources/agent/daydaymap/daydaymap.go(1 hunks)sources/agent/daydaymap/request.go(1 hunks)sources/agent/daydaymap/response.go(1 hunks)sources/keys.go(2 hunks)sources/provider.go(4 hunks)uncover.go(3 hunks)
🧰 Additional context used
🧬 Code graph analysis (4)
uncover.go (2)
sources/agent/daydaymap/daydaymap.go (1)
Agent(20-20)sources/agent.go (1)
Agent(8-11)
sources/agent/daydaymap/request.go (1)
sources/agent/fofa/fofa.go (1)
Fields(25-25)
sources/agent/daydaymap/response.go (1)
sources/agent/hunterhow/response.go (1)
List(8-12)
sources/agent/daydaymap/daydaymap.go (7)
sources/agent.go (1)
Query(3-6)sources/session.go (1)
Session(39-44)sources/keys.go (1)
Keys(3-24)uncover.go (1)
New(58-116)sources/agent/daydaymap/request.go (1)
DaydaymapRequest(4-10)sources/util.go (1)
NewHTTPRequest(10-17)sources/agent/daydaymap/response.go (1)
DaydaymapResponse(4-8)
🔇 Additional comments (8)
sources/keys.go (1)
23-23: LGTM!The
Daydaymapfield addition and its inclusion inEmpty()follow the established pattern for other provider keys.Also applies to: 45-46
sources/provider.go (1)
40-40: LGTM!Daydaymap provider integration follows the established patterns for struct field, key retrieval, environment variable loading, and key presence checking.
Also applies to: 127-129, 162-162, 202-202
sources/agent/daydaymap/request.go (1)
1-10: LGTM!The request structure is clean with appropriate JSON tags and optional field handling via
omitempty.sources/agent/daydaymap/response.go (1)
1-20: LGTM!Using
map[string]interface{}forDaydaymapResultprovides flexibility for varying API response fields. The agent correctly extracts the needed fields (ip,port,domain,service) with type assertions in the query method.uncover.go (1)
13-13: LGTM!Daydaymap agent is properly imported, registered in the switch statement, and included in
AllAgents().Also applies to: 96-97, 202-202
runner/options.go (1)
66-66: LGTM!Complete CLI integration for Daydaymap: field definition, flag registration with
-ddmalias, engine help text, validation checks, and query aggregation all follow established patterns.Also applies to: 78-78, 100-100, 177-178, 251-252, 280-281, 326-326
sources/agent/daydaymap/daydaymap.go (2)
26-80: LGTM on pagination logic.The Query method correctly handles pagination with proper termination conditions: user limit, total results, empty page, and API maximum. The goroutine properly closes the results channel on exit.
36-37: Verify base64 encoding requirement with Daydaymap API documentation.The implementation uses base64 encoding for query keywords, but this deviates from standard API practices where query parameters are typically URL-encoded. Confirm this matches the Daydaymap API specification before deployment, as public documentation is not publicly available. If base64 is not required, replace with standard URL encoding (percent-encoding).
| func (agent *Agent) query(URL string, session *sources.Session, daydaymapRequest *DaydaymapRequest, results chan sources.Result) *DaydaymapResponse { | ||
| resp, err := agent.queryURL(session, URL, daydaymapRequest) | ||
| if err != nil { | ||
| results <- sources.Result{Source: agent.Name(), Error: err} | ||
| return nil | ||
| } | ||
|
|
||
| daydaymapResponse := &DaydaymapResponse{} | ||
| if err := json.NewDecoder(resp.Body).Decode(daydaymapResponse); err != nil { | ||
| results <- sources.Result{Source: agent.Name(), Error: err} | ||
| return nil | ||
| } |
There was a problem hiding this comment.
Missing resp.Body.Close() causes resource leak.
The HTTP response body is decoded but never closed, which will leak connections and eventually exhaust resources under load.
Apply this diff to fix the resource leak:
func (agent *Agent) query(URL string, session *sources.Session, daydaymapRequest *DaydaymapRequest, results chan sources.Result) *DaydaymapResponse {
resp, err := agent.queryURL(session, URL, daydaymapRequest)
if err != nil {
results <- sources.Result{Source: agent.Name(), Error: err}
return nil
}
+ defer resp.Body.Close()
daydaymapResponse := &DaydaymapResponse{}
if err := json.NewDecoder(resp.Body).Decode(daydaymapResponse); err != nil {
results <- sources.Result{Source: agent.Name(), Error: err}
return nil
}🤖 Prompt for AI Agents
In sources/agent/daydaymap/daydaymap.go around lines 101 to 112, the HTTP
response body from agent.queryURL is decoded but never closed which leaks
connections; after confirming resp is non-nil and err is nil, immediately defer
resp.Body.Close() so the body is always closed whether JSON decoding succeeds or
fails, ensuring resources are released even when sending errors to the results
channel.
Mzack9999
left a comment
Mzack9999
left a comment
There was a problem hiding this comment.
lgtm!
% go run . -e daydaymap -q "domain=\"projectdiscovery.io\"" -pc provider-config.yaml -v
...
projectdiscovery.io
[INF] Current uncover version v1.2.1 (latest)
[daydaymap] 76.76.21.21:443
...3 participants
Add daydaymap as a new supported search engine with complete integration:
The implementation follows the existing agent pattern and maintains consistency with other search engine integrations (greynoise, driftnet, etc).
Summary by CodeRabbit
✏️ Tip: You can customize this high-level summary in your review settings.