feat: microgen - add AST analysis utilities

[chalmerlowe]

Follows: PR #2291 (should be merged after that PR is merged.)

This PR adds the CodeAnalyzer class, which is a node visitor that traverses an AST and extracts structured information about classes, methods, and their arguments.

Inludes:

• generate.py file
• the CodeAnalyzer class
• several helper functions

[chalmerlowe]

For clarity:

1. The GitHub Actions are being used to help ensure that unit tests pass.
   
   
2. The KOKORO tests are failing. This is a known problem and will be dealt with in a separate PR. It should not affect merging into the autogen dev branch.
   
   

[Linchin]

Should we raise an error if self._current_class_info turns out to be false, instead of doing nothing silently?

[chalmerlowe]

The current structure of the code is this:

def visit_Assign(self, node):
    # ... other logic ...
    if self._current_class_info:
        # ... determine attr_name, attr_type ...
        self._add_attribute(attr_name, attr_type)
    self.generic_visit(node)
 
def visit_AnnAssign(self, node):
    # ... other logic ...
    if self._current_class_info:
        # ... determine attr_name, attr_type ...
        self._add_attribute(attr_name, attr_type)
    self.generic_visit(node)

Given this structure:

1. The if self._current_class_info: check inside _add_attribute is redundant. It
   will always be True because the callers guarantee it.
2. There is no need to raise an error for a missing class context within _add_attribute, as the situation is prevented by the calling methods.

I removed the check inside of _add_attribute. This makes _add_attribute cleaner and more focused, relying on the contract established by its callers. I updated the docstring to reflect this assumption.

[Linchin]

This check seems redundant

[chalmerlowe]

Good catch. Removed.

[Linchin]

For my education, why is file_name omitted here?

[chalmerlowe]

Thanks for asking! Seems like a reasonable question.

This design of the list_code_objects() function is to return a well-structured dictionary with clear keys, whether it's analyzing a single file or an entire directory.

This library can be run automatically OR interactively. It also has the ability to be run against a single file OR a directory full of files. Looking at both cases:

1. if os.path.isfile(path) and path.endswith(".py"):

   • This block executes when the user provides a path to a single Python file.
   • process_structure(structure) is called without the file_name argument.
   • Why? Since we are only analyzing one file, any class name found is unique to
     that file. There's no need to disambiguate it with the filename in the output
     keys. The key in process_structure will just be class_info["class_name"].

2. elif os.path.isdir(path):

   • This block executes when the user provides a path to a directory.
   • The code iterates through all .py files within that directory.
   • process_structure(structure, file_name=os.path.basename(file_path)) is called
     for each file.
   • Why? When scanning multiple files, it's possible to encounter classes with the
     same name in different files. To prevent these from clobbering each other in
     the results dictionary and to make the output clear, the file_name is used to
     make the key unique. The key in process_structure becomes
     f"{class_info["class_name"]} (in {file_name})".

In essence: The file_name argument is only provided when processing a directory to
ensure that class names in the output dictionary are unique, even if the same class
name appears in multiple files. When processing a single file, this disambiguation
is not necessary.