Add management APIs to replace shell scripts

[kanghengliu]

Summary

The gateway/registry is still administered through shell helpers like cli/service_mgmt.sh, cli/agent_mgmt.sh, and cli/user_mgmt.sh. Each script shells out to uv run commands, curls the Keycloak admin API, and even docker execs into containers to mutate scopes.yml or FAISS metadata. We need first-class, authenticated management APIs so operators, automation, and the frontend can manage the registry remotely instead of SSH-ing somewhere to run Bash.

Pain points

• cli/service_mgmt.sh is the only way to add/delete services, toggle status, and manage Keycloak group wiring for scopes (see the built-in commands listed at the top of the script). It assumes local Docker names like mcp-gateway-registry-auth-server-1 and reaches into container filesystems.
• cli/agent_mgmt.sh simply wraps uv run python cli/agent_mgmt.py … for register/list/update/toggle/test/search, so every integration must carry .oauth-tokens/ingress.json and the right env vars to succeed.
• cli/user_mgmt.sh handles M2M + human users by grabbing a Keycloak admin token directly and issuing curl calls; it stores secrets under .oauth-tokens and requires the admin password in KEYCLOAK_ADMIN_PASSWORD.
• There is no authenticated HTTP API we can point CI, the React frontend, or partner automation to for these management flows, which makes everything brittle and manual.

Proposal

1. Design a Management API surface that mirrors the behaviors of the scripts (service lifecycle, scopes/group attachments, FAISS metadata refresh, agent CRUD/test/search, user + group operations).
2. Implement the API inside the registry (or a companion management service) with proper authN/Z so we can issue scoped tokens instead of admin passwords.
3. Provide a Go/Python client (or extend cli/mcp_client.py) that targets the new endpoints, keeping feature parity.
4. Update docs (e.g., docs/service-management.md, docs/cli.md) and deprecate the Bash scripts once the API ships.
5. Keep shell helpers only as thin wrappers around the HTTP API (or remove them entirely) after the migration.

Acceptance criteria

• Every command advertised in the existing scripts has an equivalent HTTP endpoint with schema + validation documented.
• CI/CD and frontend features can use the HTTP API without docker exec or direct Keycloak admin calls.
• Secrets/tokens are managed via scoped service accounts, not shared admin credentials written to disk.
• Documentation and examples default to the API workflow; scripts are optional.