protodoc: Pre-parse/validate the protodoc manifest

[phlax]

Signed-off-by: Ryan Northey ryan@synca.io

Commit Message:
Additional Description:

Currently the protodoc_manifest which contains api examples, is validated in every protodoc process

This moves the validation to a separate job and builds a json artefact from the validated data

Risk Level:
Testing:
Docs Changes:
Release Notes:
Platform Specific Features:
[Optional Runtime guard:]
[Optional Fixes #Issue]
[Optional Fixes commit #PR or SHA]
[Optional Deprecated:]
[Optional API Considerations:]

[phlax]

/docs

[repokitteh-read-only]

Docs for this Pull Request will be rendered here:

https://storage.googleapis.com/envoy-pr/21559/docs/index.html

The docs are (re-)rendered each time the CI envoy-presubmit (precheck docs) job completes.

🐱

Caused by: a #21559 (comment) was created by @phlax.

see: more, trace.

[htuch]

Can you indicate the performance improvement on these optimizations? Seems reasonable, yet we could arbtirarily preprocess things and it would be good to focus on the heavy hitters.

[phlax]

thisi s probably the most important optimization that i found (none were huge)

i can get some timings tomorrow - but the point is - moving the validation out of protodoc means that the protodoc environment is leaner (no runfiles, config_validation, etc)

the validation happening here is also pointless - it is validating the same thing repeatedly

even if it doesnt give a massive increase getttng this one out of the way is an important step to the next PR - which mangles all required data for protodoc into a single json object - see here https://github.com/envoyproxy/envoy/pull/21561/files#diff-1e2d0f2d31a35d12f876114dd98bde49a7559427f00329f9b9c33aa8d494d569R86

[phlax]

With the following diff:

diff --git a/tools/api_proto_plugin/traverse.py b/tools/api_proto_plugin/traverse.py
index a27813ffa5..6bf1aee991 100644
--- a/tools/api_proto_plugin/traverse.py
+++ b/tools/api_proto_plugin/traverse.py
@@ -1,7 +1,12 @@
 """FileDescriptorProto traversal for api_proto_plugin framework."""
 
+import logging
+import time
+
 from tools.api_proto_plugin import type_context
 
+logger = logging.getLogger(__name__)
+
 
 def traverse_service(type_context, service_proto, visitor):
     """Traverse a service definition.
@@ -74,6 +79,7 @@ def traverse_file(file_proto, visitor):
     Returns:
         Plugin specific output.
     """
+    start = time.time()
     source_code_info = type_context.SourceCodeInfo(file_proto.name, file_proto.source_code_info)
     package_type_context = type_context.TypeContext(source_code_info, file_proto.package)
     services = [
@@ -91,4 +97,6 @@ def traverse_file(file_proto, visitor):
             package_type_context.extend_enum(index, enum.name, enum.options.deprecated), enum,
             visitor) for index, enum in enumerate(file_proto.enum_type)
     ]
-    return visitor.visit_file(file_proto, package_type_context, services, msgs, enums)
+    result = visitor.visit_file(file_proto, package_type_context, services, msgs, enums)
+    logger.warning(f"VISITED {file_proto.name}: {time.time() - start}s")
+    return result

and ommitting logs for all but the worst offender...

before

VISITED envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto: 10.386241436004639s

after

VISITED envoy/extensions/filters/network/http_connection_manager/v3/http_connection_manager.proto: 0.43071722984313965s

[phlax]

in terms of protodoc itself this opti is huge - but as discussed offline very little of the time generating the api docs is actually spent in protodoc itself so the overall effect is not massive

[phlax]

this can be sped up considerably by making use of the v3_proto_set added in #21579 - i have opened #21607 to work on that

[phlax]

@htuch it would be great to land this one

currently i am rebasing my dev branches on this and the no-aspect branch as between them they make building docs/ running tests etc bearable

a couple of my WIP PRs are waiting on this one, and i also have further optimizations that i know of to this script that i am waiting to add

[htuch]

Feel free to merge when main is unlocked.