Enhance pagination and search functionality for nodes in JSON endpoints#734
Enhance pagination and search functionality for nodes in JSON endpoints#734javuto merged 5 commits intojmpsec:mainfrom
Conversation
There was a problem hiding this comment.
Pull Request Overview
This PR implements server-side pagination and search functionality for node management endpoints to improve performance with large datasets. The implementation includes paginated node retrieval, counting methods for accurate pagination metadata, and client-side ordering parameter forwarding.
- Added server-side pagination with LIMIT/OFFSET queries instead of loading all nodes into memory
- Implemented separate counting methods for total and filtered node counts
- Updated frontend to pass DataTables ordering parameters to the backend
Reviewed Changes
Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
| pkg/nodes/nodes.go | Added paginated retrieval methods with ordering and counting functionality |
| cmd/admin/handlers/json-nodes.go | Updated handler to use paginated queries and process ordering parameters |
| cmd/admin/templates/table.html | Modified frontend to forward DataTables ordering parameters to server |
Comments suppressed due to low confidence (2)
cmd/admin/handlers/json-nodes.go:1
- The
orderByparameter is directly concatenated into the SQL ORDER BY clause without validation. This could potentially lead to SQL injection if themapDTColumnToDBfunction doesn't properly sanitize all possible inputs. Consider using a whitelist validation or GORM's safe ordering methods.
package handlers
cmd/admin/handlers/json-nodes.go:1
- The
orderByparameter is directly concatenated into the SQL ORDER BY clause without validation. This could potentially lead to SQL injection if themapDTColumnToDBfunction doesn't properly sanitize all possible inputs. Consider using a whitelist validation or GORM's safe ordering methods.
package handlers
Tip: Customize your code reviews with copilot-instructions.md. Create the file or learn how to get started.
| if limit <= 0 { // safety default | ||
| limit = 25 | ||
| } | ||
| if limit > 500 { // cap to avoid abuse | ||
| limit = 500 | ||
| } | ||
| if offset < 0 { | ||
| offset = 0 | ||
| } |
There was a problem hiding this comment.
The limit validation logic is duplicated between GetByEnvPage and SearchByEnvPage methods. Consider extracting this into a helper function to avoid code duplication.
| } | ||
| var nodes []OsqueryNode | ||
| likeTerm := "%" + term + "%" | ||
| query := n.DB.Where("environment = ? AND (uuid LIKE ? OR hostname LIKE ? OR localname LIKE ? OR ip_address LIKE ? OR username LIKE ? OR osquery_user LIKE ? OR platform LIKE ? OR osquery_version LIKE ?)", env, likeTerm, likeTerm, likeTerm, likeTerm, likeTerm, likeTerm, likeTerm, likeTerm) |
There was a problem hiding this comment.
The complex search query with multiple LIKE conditions is duplicated between SearchByEnvPage and CountSearchByEnv. Consider extracting this query building logic into a helper method to ensure consistency and maintainability.
| // CountSearchByEnv counts matching nodes for a search term with target filters | ||
| func (n *NodeManager) CountSearchByEnv(env, term, target string, hours int64) (int64, error) { | ||
| likeTerm := "%" + term + "%" | ||
| query := n.DB.Model(&OsqueryNode{}).Where("environment = ? AND (uuid LIKE ? OR hostname LIKE ? OR localname LIKE ? OR ip_address LIKE ? OR username LIKE ? OR osquery_user LIKE ? OR platform LIKE ? OR osquery_version LIKE ?)", env, likeTerm, likeTerm, likeTerm, likeTerm, likeTerm, likeTerm, likeTerm, likeTerm) |
There was a problem hiding this comment.
The complex search query with multiple LIKE conditions is duplicated between SearchByEnvPage and CountSearchByEnv. Consider extracting this query building logic into a helper method to ensure consistency and maintainability.
javuto
added
osctrl-admin
#635
This PR:
For now, the query speed is about 10,000× faster than before with 10K nodes, and we can clearly see changes in both the database queries and the request URLs