Skip to content

Add Klish AAA kvmtest validation with VS DUT#23

Open
devin-ai-integration[bot] wants to merge 6 commits intomasterfrom
devin/1771434202-klish-aaa-kvmtest
Open

Add Klish AAA kvmtest validation with VS DUT#23
devin-ai-integration[bot] wants to merge 6 commits intomasterfrom
devin/1771434202-klish-aaa-kvmtest

Conversation

@devin-ai-integration
Copy link

Why I did it

Validates the Klish AAA CLI commands implemented in a prior session by running KVM-based tests against a Virtual Switch (VS) DUT in the Azure DevOps pipeline. The AAA implementation spans sonic-mgmt-common (transformer), sonic-mgmt-framework (Klish CLI/actioners), and sonic-mgmt (test cases).

Work item tracking
  • Microsoft ADO: N/A

How I did it

  1. Updated submodules to include the AAA feature code:

    • src/sonic-mgmt-common → AAA transformer branch (27c22002) with OpenConfig-to-CONFIG_DB translation for AAA paths
    • src/sonic-mgmt-framework → AAA Klish CLI branch (eabe6c84) with CLI XML definitions, actioners, and renderer templates
  2. Pointed sonic-mgmt repo resource at devin/1771309030-klish-aaa-tests branch which contains tests/klish_aaa/test_klish_aaa.py — a 780-line pytest suite covering:

    • CONFIG_DB CRUD for authentication (failthrough/fallback/debug/trace/login), authorization, and accounting
    • REST API calls (GET/PATCH/DELETE) via the OpenConfig AAA paths
    • Klish CLI commands: aaa authentication, aaa authorization, aaa accounting, and all no variants
    • Full end-to-end configure-and-verify / negate-and-verify workflows
    • Automatic save/restore of AAA config between tests
  3. Added a Test stage (kvmtest_klish_aaa job) that:

    • Depends on BuildVS stage
    • Downloads the VS image artifact, sets up sonic-mgmt container & KVM testbed
    • Runs the Klish AAA test suite against vlab-01 on topology vms-kvm-t0
    • Publishes JUnit XML results and test logs as pipeline artifacts
    • Posts a formatted test results summary as a GitHub PR comment

How to verify it

  1. Merge this PR to trigger the Azure DevOps pipeline
  2. Confirm the BuildVS stage succeeds and produces a sonic-buildimage.vs artifact
  3. Confirm the Test stage kvmtest_klish_aaa job:
    • Successfully boots the VS DUT
    • Runs all Klish AAA tests
    • Publishes test results (check the Azure DevOps "Tests" tab)
  4. Verify the test results comment is posted back to this PR

Which release branch to backport (provide reason below if selected)

  • 202305
  • 202311
  • 202405
  • 202411
  • 202505
  • 202511

N/A - This is a new feature test, not a backport candidate.

Tested branch (Please provide the tested image version)

Description for the changelog

Add KVM-based validation tests for Klish AAA CLI commands using Virtual Switch DUT in Azure DevOps pipeline.

Link to config_db schema for YANG module changes

N/A - No YANG model changes in this PR. The AAA YANG models and transformer were implemented in the submodule branches.


⚠️ Important Review Notes

This PR is NOT ready to merge as-is. It is intended to validate the AAA implementation in a test environment. Before merging:

  1. Submodule branches must be merged first: Both sonic-mgmt-common and sonic-mgmt-framework currently point to unmerged devin/ feature branches. These need to be:

    • Reviewed and merged to their respective master branches
    • Then this PR's submodule SHAs updated to point to merged commits
  2. sonic-mgmt test branch: The pipeline references devin/1771309030-klish-aaa-tests branch. Either:

    • Merge the test file to sonic-mgmt master and update the ref to master, OR
    • Keep the feature branch ref temporarily for validation, then update before final merge
  3. Agent prerequisites: The sonic-build agent pool must have:

    • docker-sonic-mgmt:latest image pre-loaded
    • KVM infrastructure configured
    • /data/sonic-vm and /data/sonic-mgmt paths available
    • Sufficient resources for a 240-minute test run
  4. Testbed configuration: Verify vtestbed.yaml exists in sonic-mgmt and matches the expected format for vms-kvm-t0 topology.

Human Review Checklist

  • Verify submodule SHAs point to correct AAA implementation commits (check commit messages/diffs)
  • Confirm sonic-build agent has docker-sonic-mgmt image and KVM setup
  • Review testbed setup flow: refresh-dut → deploy-mg → test execution
  • Check if 240-minute timeout is sufficient (or too generous)
  • Validate GitHub token extraction logic for PR commenting
  • Confirm vtestbed.yaml format matches what testbed-cli.sh expects
  • Review test coverage: does test_klish_aaa.py actually cover all AAA commands from the original session?

Link to Devin run: https://cisco-demo.devinenterprise.com/sessions/ca4e945629df47588becf9e972c3435a
Requested by: @arthurkkp-cog

A picture of a cute animal (not mandatory but encouraged)

🦦 (Otter - because AAA is all about authentication, authorization, and accounting... and otters are great at keeping things secure!)

- Update sonic-mgmt-common submodule to AAA transformer branch
- Update sonic-mgmt-framework submodule to AAA Klish CLI branch
- Add Test stage with kvmtest_klish_aaa job using VS DUT
- Reference sonic-mgmt branch with Klish AAA test cases
- Pipeline: build VS image, setup testbed, run tests, publish results
- Post test results as GitHub PR comment

Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
@devin-ai-integration
Copy link
Author

🤖 Devin AI Engineer

I'll be helping with this pull request! Here's what you should know:

✅ I will automatically:

  • Address comments on this PR. Add '(aside)' to your comment to have me ignore it.
  • Look at CI failures and help fix them

Note: I can only respond to comments from users who have write access to this repository.

⚙️ Control Options:

  • Disable automatic comment and CI monitoring

devin-ai-integration bot and others added 5 commits February 18, 2026 23:14
…1 parse error

Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
…e whitespace

Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
…build context

Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
…ailures

Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

0 participants