Add kvmtest-t0 integration test stage to CI pipeline#21
Open
devin-ai-integration[bot] wants to merge 16 commits intomasterfrom
Open
Add kvmtest-t0 integration test stage to CI pipeline#21devin-ai-integration[bot] wants to merge 16 commits intomasterfrom
devin-ai-integration[bot] wants to merge 16 commits intomasterfrom
Conversation
Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
Author
🤖 Devin AI EngineerI'll be helping with this pull request! Here's what you should know: ✅ I will automatically:
Note: I can only respond to comments from users who have write access to this repository. ⚙️ Control Options:
|
Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
…path Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
Owner
❌ Build Failed:
|
Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
Owner
❌ Build Failed:
|
Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
Owner
❌ Build Failed:
|
Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
Owner
❌ Build Failed:
|
…sh-dut Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
Owner
❌ Build Failed:
|
…verify sonic-mgmt Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
…comments Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
Owner
❌ Build Failed:
|
…tion, handle oversized comments Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
Owner
❌ Build Failed:
|
Open
4 tasks
Adds a pipeline step to fetch the fixed test_cacl.py from arthur-cog-sonic/sonic-mgmt fork before running kvmtest. The fix adds teardown to restore_test_env fixture so config_db_check passes. Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
![devin-ai-integration[bot]](https://avatars.githubusercontent.com/in/811515?s=60&v=4){kind=small}
Comment on lines
+298
to
+299
| git remote update | ||
| git reset --hard origin/master |
Author
There was a problem hiding this comment.
🔴 git reset --hard origin/master in "Setup testbed" step overwrites the test fix applied by the earlier "Update sonic-mgmt" step
The "Update sonic-mgmt with test fixes" step at azure-pipelines.yml:206 checks out a patched test_cacl.py from the arthur-fork branch into /data/sonic-mgmt. However, the later "Setup testbed" step at azure-pipelines.yml:299 runs git reset --hard origin/master, which resets the entire working tree back to origin/master, completely erasing the file checked out in the earlier step.
Root Cause and Impact
The step ordering is:
- Line 206:
git checkout arthur-fork/devin/1771222385-fix-cacl-config-restore -- tests/generic_config_updater/test_cacl.py - Line 299:
git reset --hard origin/master— this discards ALL uncommitted changes, including the checkout from step 1.
The intent of the "Update sonic-mgmt with test fixes" step is to patch test_cacl.py with a config restore fix before tests run. But because the setup-testbed step does a hard reset afterward, the fix is never actually present when kvmtest.sh executes at line 327. The test_cacl.py config restore fix is silently lost every run.
Impact: The test fix for test_cacl.py is never applied, so kvmtests always run with the unpatched version from origin/master.
Prompt for agents
In azure-pipelines.yml, the "Setup testbed" step (around line 297-299) does `pushd /data/sonic-mgmt`, `git remote update`, then `git reset --hard origin/master`. This erases the test_cacl.py fix that was checked out from arthur-fork in the "Update sonic-mgmt with test fixes" step (line 206). To fix this, either: (1) Move the "Update sonic-mgmt with test fixes" step (lines 193-210) to AFTER the `git reset --hard origin/master` in the "Setup testbed" step (i.e., after line 299 but before line 300), or (2) Re-apply the checkout of test_cacl.py after the git reset in the setup-testbed script. For example, after line 299 add: `git fetch arthur-fork devin/1771222385-fix-cacl-config-restore 2>/dev/null && git checkout arthur-fork/devin/1771222385-fix-cacl-config-restore -- tests/generic_config_updater/test_cacl.py 2>/dev/null || true`.
Was this helpful? React with 👍 or 👎 to provide feedback.
Owner
❌ Build Failed:
|
The previous approach using git fetch from the fork likely failed on the build agent due to network/auth issues. Replace with an inline Python script that patches test_cacl.py directly on the agent filesystem. Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
The previous inline Python code broke Azure Pipelines YAML parser because unindented Python lines violated the YAML literal block scalar indentation rules. Fix by base64-encoding the patch script and decoding it at runtime on the agent. Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
Owner
❌ Build Failed:
|
The patch was being applied before the setup testbed step which runs 'git reset --hard origin/master', reverting the patch. Move patch step to after setup testbed but before test execution. Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
Owner
❌ Build Failed:
|
Add sed patch to fix the stale test_ipv6.py reference in kvmtest.sh that was renamed to test_ip_bgp.py in sonic-net/sonic-mgmt#13650. This was causing 'file or directory not found' errors during kvmtest. Co-Authored-By: Arthur Poon <arthur.poon@windsurf.com>
Owner
❌ Build Failed:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why I did it
Enable KVM-based integration testing (kvmtest-t0) in the CI pipeline for the fork. The Test stage was previously commented out because it depended on the SONiC Elastictest managed service, which is not available in the Cisco-SONiC-PoC Azure DevOps organization. This PR adds a direct KVM test stage that runs on the self-hosted
sonic-buildagent pool, modeled after the existing.azure-pipelines/run-test-template.yml.Work item tracking
How I did it
Added a new
Teststage toazure-pipelines.ymlthat:BuildVSsucceedingsonic-mgmtDocker container if it doesn't exist, using/data/sonic-mgmt/setup-container.shwith a manual fallbackvms6-1containers, cEOS data directories, and stalevlabVMs via libvirt, then verifies the sonic-mgmt container survived cleanup/data/sonic-mgmttoorigin/master, configuresveos_vtb, auto-detects the cEOS image version on the agent and patchesceos.yml, then runstestbed-cli.sh refresh-dutgit reset --hard origin/masterto avoid being reverted):restore_test_envfixture intest_cacl.py, restoringconfig_db.jsonafter the module completes (see sonic-mgmt PR #1)kvmtest.shreference:test_ipv6.pywas renamed totest_ip_bgp.pyupstream (PR #13650) butkvmtest.shwas never updatedkvmtest.shwith t0 topology againstvms-kvm-t0/vlab-01post-failure-to-github.ymltemplateAlso rewrote
post-failure-to-github.ymlto improve error visibility: extracts pytest short test summary sections, uses more precise error patterns, includes environment snapshots, and handles oversized comments by rebuilding with reduced output.Updates since last revision
config_db_checkfailure (confirmed in builds update swss, swss-common, sairedis submodules sonic-net/sonic-buildimage#189, sonic-cfggen with sonicv2 dockers sonic-net/sonic-buildimage#190): Therestore_test_envfixture intest_cacl.pyreloaded config from minigraph at setup but had no teardown to restore the original config. Created sonic-mgmt PR #1 with the fix. The pipeline now patches the file in-place via a base64-encoded script.git reset --hard origin/master, silently reverting the patch. Moved the patch step to run after testbed setup but before test execution.test_ipv6.pymissing file error: The file was renamed totest_ip_bgp.pyin upstream PR #13650 (Oct 2024) butkvmtest.shwasn't updated. Added asedpatch in the pipeline to fix the reference. Also fixed in sonic-mgmt PR #1.test_gnmi_configdb_full_01).How to verify it
update-sonic-mgmt.logforSUCCESS: test_cacl.py patchedandkvmtest.sh patchedconfig_db_checkno longer fails aftertest_cacl.py(confirmed in builds update swss, swss-common, sairedis submodules sonic-net/sonic-buildimage#189, sonic-cfggen with sonicv2 dockers sonic-net/sonic-buildimage#190)Human Review Checklist:
git reset ordering: The patch step now runs AFTER(Fixed in build update swss, swss-common, sairedis submodules sonic-net/sonic-buildimage#189)git reset --hard origin/masterMissing test_ipv6.py: File was renamed to(Fixed in build sonic-cfggen with sonicv2 dockers sonic-net/sonic-buildimage#190 via sed patch)test_ip_bgp.pyupstreamrestore_test_envfixture text, the patch will silently fail (continueOnError: true). Consider merging sonic-mgmt PR #1 to upstream.test_gnmi_configdb_full_01fails in build sonic-cfggen with sonicv2 dockers sonic-net/sonic-buildimage#190. This appears to be an unrelated upstream issue, not caused by this PR.vms6-1: The cleanup step usesgrep 'vms6-1'to find containers. This matches thegroup-nameinvtestbed.yamlforvms-kvm-t0. If the testbed config changes, update the cleanup step accordingly.virsh -c qemu:///system listfails on the agent. The cleanup step's VM removal and kvmdump collection depend on libvirt. Verify libvirtd is running and accessible to theazureuseraccount.sleep 180is a fixed wait for DUT readiness. May need tuning based on actual boot times.Which release branch to backport (provide reason below if selected)
Tested branch (Please provide the tested image version)
test_ipv6.pyfile.test_gnmi_configdb_full_01).Description for the changelog
Add kvmtest-t0 integration test stage to the CI pipeline using direct KVM testing on the self-hosted build agent, with auto-creation of sonic-mgmt container, auto-detection of cEOS image version, cleanup of existing testbed topology, and improved GitHub failure reporting with pytest summary extraction
Link to config_db schema for YANG module changes
N/A
A picture of a cute animal (not mandatory but encouraged)
N/A
Link to Devin run: https://cisco-demo.devinenterprise.com/sessions/ec7f851273cb42029be7c8d2d912ec7b
Requested by: Arthur Poon (@arthurkkp-cog)