Fix/array of objects and cloud docs check (#166)

* fix: Correctly render array-of-objects fields as empty arrays

Array-of-objects fields like client_certs[] were incorrectly rendered as
flat objects with all child properties expanded, creating invalid
configurations that mixed mutually exclusive options.

Changes:
- buildConfigYaml.js: Check field.kind === 'array' to detect array-of-objects
- renderObjectField.js: Same check for nested array-of-objects fields
- Now renders 'client_certs: []' instead of expanded object structure

This fixes 81 occurrences of client_certs plus other array-of-objects
fields like tools[], roles[], sasl[], etc.

Impact: 105 files changed in generated docs, removing 966 lines of
incorrect expanded structures.

Resolves issue where examples showed both inline (cert/key) and
file-based (cert_file/key_file) options together, violating the
documented 'either/or' constraint.

* feat: Check cloud-docs repo for missing cloud-supported connectors

Added logic to verify that all cloud-supported connectors (inCloud + cloudOnly)
have corresponding documentation pages in the cloud-docs repository.

Changes:
- Build set of cloud-supported connectors from binary analysis
- Use GitHub API to check if connector pages exist in cloud-docs
- Report missing connectors with their paths
- Runs automatically during draft-missing workflow (can be disabled)

The check uses GitHub API (via Octokit) to avoid cloning the cloud-docs repo.
Respects VBOT_GITHUB_API_TOKEN or GITHUB_TOKEN environment variables.

Also improved cloud-only connector detection:
- Cloud-only connectors now only check the cloud-only directory
- Regular connectors check pages and partials (not cloud-only)

This helps identify connectors that are available in Cloud but missing
documentation pages in the cloud-docs repository.

* fix: Surface non-404 errors in cloud-docs check

Update cloud-docs validation to record and report non-404 API errors
(auth, rate-limit, network) instead of silently ignoring them.

- Add cloudDocsErrors array to track non-404 failures
- Capture error status and message for each failure
- Report inconclusive check with error details when failures occur
- Only show success message when check completes without errors
- Provide troubleshooting guidance for common error causes

* fix: Preserve cloudOnly/requiresCgo flags and optimize octokit init

- Preserve cloudOnly and requiresCgo flags when building validConnectors
  from dataObj so cloud-only connectors are properly detected
- Move Octokit initialization outside the loop to reuse connection
  instead of creating a new instance for each connector check

* feat: Add raw URL fallback for cloud-docs validation

When GitHub API fails with auth/rate-limit errors, fall back to
checking raw.githubusercontent.com URLs via HTTP HEAD request.

Flow:
1. Try official API with authentication
2. If 404 -> file missing (expected)
3. If 403/401/rate-limit -> try raw URL fallback
   - 200 -> file exists (success)
   - 404 -> file missing (confirmed)
   - Other -> record error with both API and fallback details

This allows validation to work even without GITHUB_TOKEN or when
rate-limited, while still preferring the official API when available.

* fix: Correct cloud-docs path and enable check without binary analysis

- Fix cloud-docs path from modules/components/pages to
  modules/develop/pages/connect/components
- Fix plural type handling (inputs not inputss)
- Enable cloud-docs validation without binary analysis by checking
  all non-deprecated connectors when binary data unavailable
- Reduces false positives from 282 to 64 actual missing connectors

* feat: Auto-load .env files and fix cloud-docs validation

Changes:
- Add dotenv package for automatic .env file loading
- Load .env files at startup in both CLI entry points
- Fix cloud-docs validation to only check actual connector types

Cloud-docs validation improvements:
- Filter out config types (config/*) - internal schemas only
- Filter out Bloblang functions/methods - documented on reference pages
- Filter out rate-limits - documented differently
- Only check: inputs, outputs, processors, caches, buffers, scanners, metrics, tracers

Result: Validation now correctly reports ~7 missing connectors instead of 252

Benefits:
- No more manual export of GITHUB_TOKEN needed
- Binary analysis works automatically with .env file
- Cloud-docs validation is accurate for connector coverage

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* docs: Auto-update CLI reference documentation (PR #166)

* fix: Skip deprecated connectors in cloud-docs validation

Deprecated connectors shouldn't be flagged as missing from cloud-docs.
This filter reduces false positives (e.g., pg_stream which is deprecated).

Co-Authored-By: Claude Sonnet 4.5 <noreply@anthropic.com>

* docs: Auto-update CLI reference documentation (PR #166)

* fix: Downgrade dotenv to v16 to suppress promotional messages

dotenv 17.x includes promotional stderr output that clutters CLI output.
Downgraded to 16.x which provides the same functionality without
the promotional messages.

* docs: Auto-update CLI reference documentation (PR #166)

* refactor: Consolidate Octokit instances into shared client

Creates a centralized Octokit client singleton in cli-utils/octokit-client.js
that is shared across all doc-tools modules. This eliminates redundant
initialization, shares rate limit tracking, and works without authentication
when no GitHub token is set.

Changes:
- Add cli-utils/octokit-client.js as shared singleton
- Update 7 modules to use shared client:
  - tools/get-redpanda-version.js
  - tools/get-console-version.js
  - tools/fetch-from-github.js
  - tools/cloud-regions/generate-cloud-regions.js
  - extensions/generate-rp-connect-info.js
  - tools/redpanda-connect/connector-binary-analyzer.js
  - tools/redpanda-connect/rpcn-connector-docs-handler.js
- Configure auth only when token is available
- Simplify code by removing 38 lines of duplicate initialization

Benefits:
- Single initialization point for all GitHub API access
- Shared rate limit pool across modules
- Works without authentication (60 req/hr) or with token (5000 req/hr)
- Easier to maintain and update configuration

---------

Co-authored-by: Claude Sonnet 4.5 <noreply@anthropic.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>

Author: 3 people

bin/doc-tools-mcp.js

@@ -15,6 +15,9 @@
  * - Telemetry: Usage tracking for adoption metrics
  */
 
+// Load environment variables from .env file if it exists
+require('dotenv').config();
+
 const fs = require('fs');
 const path = require('path');
 const { Server } = require('@modelcontextprotocol/sdk/server/index.js');

bin/doc-tools.js

@@ -2,6 +2,9 @@
 
 'use strict'
 
+// Load environment variables from .env file if it exists
+require('dotenv').config()
+
 const { spawnSync } = require('child_process')
 const os = require('os')
 const { Command, Option } = require('commander')

cli-utils/octokit-client.js

@@ -0,0 +1,36 @@
+'use strict'
+
+const { Octokit } = require('@octokit/rest')
+const { getGitHubToken } = require('./github-token')
+
+/**
+ * Shared Octokit client instance for GitHub API access
+ * Configured with optional authentication and retry logic
+ *
+ * This singleton instance is shared across all doc-tools modules to:
+ * - Avoid redundant initialization
+ * - Share rate limit tracking
+ * - Centralize GitHub API configuration
+ */
+
+// Get authentication token from environment
+const token = getGitHubToken()
+
+// Configure Octokit options
+const octokitOptions = {
+  userAgent: 'redpanda-docs-tools',
+  retry: {
+    enabled: true,
+    retries: 3
+  }
+}
+
+// Only add auth if token is available
+if (token) {
+  octokitOptions.auth = token
+}
+
+// Create singleton instance
+const octokit = new Octokit(octokitOptions)
+
+module.exports = octokit

extensions/generate-rp-connect-info.js

@@ -25,11 +25,9 @@ module.exports.register = function ({ config }) {
   // Use csvpath (legacy) or csvPath
   const localCsvPath = csvpath || null
 
-  async function loadOctokit () {
-    const { Octokit } = await import('@octokit/rest')
-    const { getGitHubToken } = require('../cli-utils/github-token')
-    const token = getGitHubToken()
-    return token ? new Octokit({ auth: token }) : new Octokit()
+  function loadOctokit () {
+    // Use shared Octokit client
+    return require('../cli-utils/octokit-client')
   }
 
   // Use 'on' and return the promise so Antora waits for async completion

package-lock.json

@@ -1,6 +1,6 @@
 {
   "name": "@redpanda-data/docs-extensions-and-macros",
-  "version": "4.13.4",
+  "version": "4.13.5",
   "description": "Antora extensions and macros developed for Redpanda documentation.",
   "keywords": [
     "antora",
@@ -103,6 +103,7 @@
     "chalk": "4.1.2",
     "cheerio": "^1.1.2",
     "commander": "^14.0.0",
+    "dotenv": "^16.6.1",
     "glob": "^11.0.0",
     "gulp": "^4.0.2",
     "gulp-connect": "^5.7.0",

package.json

@@ -70,8 +70,8 @@ function displayClusterType(ct) {
  */
 async function fetchYaml({ owner, repo, path, ref = 'main', token }) {
   try {
-    const { Octokit } = await import('@octokit/rest');
-    const octokit = new Octokit(token ? { auth: token } : {});
+    // Use shared Octokit client
+    const octokit = require('../../cli-utils/octokit-client');
 
     console.log(`[cloud-regions] INFO: Fetching ${owner}/${repo}/${path}@${ref} via GitHub API`);
 

tools/cloud-regions/generate-cloud-regions.js

@@ -1,23 +1,10 @@
 const fs = require('fs');
 const path = require('path');
 
-let octokitInstance = null;
-async function loadOctokit() {
-  if (!octokitInstance) {
-    const { Octokit } = await import('@octokit/rest');
-    octokitInstance = process.env.VBOT_GITHUB_API_TOKEN
-      ? new Octokit({
-          auth: process.env.VBOT_GITHUB_API_TOKEN,
-        })
-      : new Octokit();
-
-    if (!process.env.VBOT_GITHUB_API_TOKEN) {
-      console.info(
-        'No GitHub token found (VBOT_GITHUB_API_TOKEN).'
-      );
-    }
-  }
-  return octokitInstance;
+// Use shared Octokit client
+function loadOctokit() {
+  const octokit = require('../cli-utils/octokit-client');
+  return octokit;
 }
 
 async function saveFile(content, saveDir, filename) {

tools/fetch-from-github.js

@@ -23,13 +23,8 @@ module.exports = async function getConsoleVersion({ beta = false, fromAntora = f
     useBeta = getPrereleaseFromAntora();
   }
 
-  // Initialize GitHub client
-  const { getGitHubToken } = require('../cli-utils/github-token');
-  const { Octokit } = await import('@octokit/rest');
-  const token = getGitHubToken();
-  const octokit = token
-    ? new Octokit({ auth: token })
-    : new Octokit();
+  // Use shared Octokit client
+  const octokit = require('../cli-utils/octokit-client');
 
   // Fetch latest release info
   let data;

tools/get-console-version.js

@@ -19,13 +19,8 @@ module.exports = async function getRedpandaVersion({ beta = false, fromAntora =
     useBeta = getPrereleaseFromAntora();
   }
 
-  // Load Octokit
-  const { getGitHubToken } = require('../cli-utils/github-token');
-  const { Octokit } = await import('@octokit/rest');
-  const token = getGitHubToken();
-  const octokit = token
-    ? new Octokit({ auth: token })
-    : new Octokit();
+  // Use shared Octokit client
+  const octokit = require('../cli-utils/octokit-client');
 
   // Fetch version data
   let data;