[engsys] Add CI check to protect against regneration failures

[xirzec]

Per #35024 (comment) we should invest in some automation that can detect that we have broken the scripts used to regenerate SDKs from spec updates

We could start with this being manually run, but it might make sense to trigger it for all dev-tool updates (or other things that touch common)

[qiaozha]

/cc @wanlwanl to provide your opinion on this.

[jeremymeng]

PR #36821

[qiaozha]

we should prevent the manual commit after generation from service team like this #36943

[qiaozha]

@jeremymeng I wonder if we should skip the copilot review for generated commits?

[jeremymeng]

we should prevent the manual commit after generation from service team like this #36943

@qiaozha Can you provide more details on the problem and requirement for this? Maybe in a new issue?

@jeremymeng I wonder if we should skip the copilot review for generated commits?

We can probably exclude certain paths, but I am not sure whether it is possible to exclude certain commits.

[qiaozha]

We can probably exclude certain paths, but I am not sure whether it is possible to exclude certain commits.

For self service, we cannot assume service team won't edit the auto generated files, but we might be able to tell which commits are from the tooling like maybe from the commit message or maybe the first commit etc.

[maorleger]

Can we have a job that regenerates and applies customization for https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/template/template (which uses https://github.com/Azure/azure-rest-api-specs/blob/main/specification/widget/data-plane/WidgetAnalytics as the spec source)?

That would be a relatively straightforward way to ensure basic codegen is working and can be run from either this repo or azure-rest-api-specs repo. It won't flag potential issues immediately (regenerating feels like an expensive operation for PR level validation) but will flag potential issues within a single day.

we should prevent the manual commit after generation from service team like this #36943

@qiaozha can you help me understand why we want to prevent this? While I understand that service teams should not be making manual changes to things that are auto-generated, preferring changes in TypeSpec, blocking commits to the generated SDK PR seems a very big hammer and something that will cause more pain than it solves for so I'd like to better understand the reasoning

[jeremymeng]

Can we have a job that regenerates and applies customization for https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/template/template (which uses https://github.com/Azure/azure-rest-api-specs/blob/main/specification/widget/data-plane/WidgetAnalytics as the spec source)?

@maorleger that's what I am trying to achieve for dev-tool changes (and probably can be extended to cover other important pieces) in https://github.com/Azure/azure-sdk-for-js/pull/36821/changes#diff-4ae5f2cb02bf0600b66bddc0ff39274b5ef494d5385dc3e06ff6fdefcd4d3dd8R8

[maorleger]

Can we have a job that regenerates and applies customization for https://github.com/Azure/azure-sdk-for-js/tree/main/sdk/template/template (which uses https://github.com/Azure/azure-rest-api-specs/blob/main/specification/widget/data-plane/WidgetAnalytics as the spec source)?

@maorleger that's what I am trying to achieve for dev-tool changes (and probably can be extended to cover other important pieces) in https://github.com/Azure/azure-sdk-for-js/pull/36821/changes#diff-4ae5f2cb02bf0600b66bddc0ff39274b5ef494d5385dc3e06ff6fdefcd4d3dd8R8

Ah nice! I was wondering, would it be simpler to kick off an automation SDK generation nightly from the https://github.com/Azure/azure-rest-api-specs repo using its automation infrastructure, etc? I was looking at the yaml and was wondering if we could leverage all the existing automation that service teams use, with the added benefit that we know we're running the same generation steps that service teams use without any risk of drift...

I know there's the downside of not being able to block PRs that potentially break automation; however, it might be a good compromise and leaves us with less to maintain and maybe flagging potential bad commits daily (which we can investigate / revert easily) is good enough. Just a thought, but either one is cool with me!

[jeremymeng]

Ah nice! I was wondering, would it be simpler to kick off an automation SDK generation nightly from the https://github.com/Azure/azure-rest-api-specs repo using its automation infrastructure, etc?

Agree that nightly from spec repo is simpler but this particular goal is to ensure JS repo changes won't break SDK generation automation, so we need to trigger on JS repo PRs.