Skip to content

Patch to fix workflow errors from HGCalTypes changes#50039

Closed
kpedro88 wants to merge 3 commits intocms-sw:masterfrom
kpedro88:PatchHGCalTypes_161X
Closed

Patch to fix workflow errors from HGCalTypes changes#50039
kpedro88 wants to merge 3 commits intocms-sw:masterfrom
kpedro88:PatchHGCalTypes_161X

Conversation

@kpedro88
Copy link
Contributor

@kpedro88 kpedro88 commented Feb 4, 2026

PR description:

Changes in #49763 led to disruption in Phase 2 workflows, as reported in #49795. It was eventually found that the error could occur even in brand-new MC (consuming no old input files), meaning that it must be the result of an actual inconsistency, not just an incompatibility between releases.

I found one bug in #49763 (inadvertent change in the order of the list of constants), which is fixed here. However, this bug did not turn out to be the cause of the workflow errors.

Only reverting the changes in the values of the constants from #49763 fixes the workflows. I still do not know why this is the case. Every usage of these values that I can find compares them to the actual named constants, not raw/hardcoded versions of the values. But there must be some comparison somewhere that does use raw values and therefore becomes inconsistent with the change. For now, all we can do is revert the change. If the new values were in fact correct, they can be restored at a later time once all usage is understood and corrected.

PR validation:

Ran on the example failing workflow from #49795 (comment). Also ran on a privately produced sample of MC from the same generator fragment that showed the same error. The workflows run successfully with the changes in this PR.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Not a backport, should not need to be backported because this change was only introduced in 16_1_X.

@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 4, 2026

cms-bot internal usage

@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 4, 2026

@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 4, 2026

A new Pull Request was created by @kpedro88 for master.

It involves the following packages:

  • Geometry/HGCalCommonData (geometry)

@Dr15Jones, @bsunanda, @civanch, @cmsbuild, @kpedro88, @makortel, @mdhildreth can you please review it and eventually sign? Thanks.
@bsunanda, @fabiocos, @martinamalberti this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@kpedro88
Copy link
Contributor Author

kpedro88 commented Feb 4, 2026

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 5, 2026

+1

Size: This PR adds an extra 24KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f5b669/51112/summary.html
COMMIT: 70803e9
CMSSW: CMSSW_16_1_X_2026-02-04-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/50039/51112/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • You potentially added 7 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 8678 differences found in the comparisons
  • DQMHistoTests: Total files compared: 52
  • DQMHistoTests: Total histograms compared: 4029600
  • DQMHistoTests: Total failures: 165000
  • DQMHistoTests: Total nulls: 225
  • DQMHistoTests: Total successes: 3864355
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 51 files compared)
  • Checked 222 log files, 193 edm output root files, 52 DQM output files
  • TriggerResults: found differences in 3 / 50 workflows

@bsunanda
Copy link
Contributor

bsunanda commented Feb 5, 2026

@kpedro88 The order was changed to conform with the specification in XML file. Earlier order and the specification did not conform with the ones in the XML files. Please let me try to check these and provide a more stable solution

@kpedro88
Copy link
Contributor Author

kpedro88 commented Feb 5, 2026

hold

@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 5, 2026

Pull request has been put on hold by @kpedro88
They need to issue an unhold command to remove the hold state or L1 can unhold it for all

@cmsbuild cmsbuild added the hold label Feb 5, 2026
@kpedro88
Copy link
Contributor Author

kpedro88 commented Feb 5, 2026

so the list of names is in the wrong order? This was an inconsistency that I noticed but was not the actual cause of workflow failure. Let me revert that commit and see if it fixes the comparison

@kpedro88
Copy link
Contributor Author

kpedro88 commented Feb 5, 2026

please test

@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 5, 2026

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50039/47886

@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 5, 2026

Pull request #50039 was updated. @Dr15Jones, @bsunanda, @civanch, @kpedro88, @makortel, @mdhildreth can you please check and sign again.

@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 5, 2026

+1

Size: This PR adds an extra 24KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f5b669/51113/summary.html
COMMIT: 24fc6c4
CMSSW: CMSSW_16_1_X_2026-02-04-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/50039/51113/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

  • No significant changes to the logs found
  • Reco comparison results: 6756 differences found in the comparisons
  • DQMHistoTests: Total files compared: 52
  • DQMHistoTests: Total histograms compared: 4029600
  • DQMHistoTests: Total failures: 18424
  • DQMHistoTests: Total nulls: 28
  • DQMHistoTests: Total successes: 4011128
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 51 files compared)
  • Checked 222 log files, 193 edm output root files, 52 DQM output files
  • TriggerResults: found differences in 3 / 50 workflows

@smuzaffar
Copy link
Contributor

smuzaffar commented Feb 5, 2026

please test workflows 34551.85,34634.114,34634.21,34634.703,34634.99,34634.9921,34634.999

lets run the workflows which are failing in 16.1.X IB

@AdrianoDee
Copy link
Contributor

A cross-check in #49984 (comment) was successful.

@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 5, 2026

REMINDER @sextonkennedy, @mandrenguyen, @ftenchini: This PR was tested with #49984, please check if they should be merged together

@kpedro88
Copy link
Contributor Author

kpedro88 commented Feb 5, 2026

It looks like I misunderstood #50039 (comment). In any case, this PR seems to be superseded by #50043.

@cmsbuild
Copy link
Contributor

cmsbuild commented Feb 5, 2026

+1

Size: This PR adds an extra 24KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-f5b669/51126/summary.html
COMMIT: 24fc6c4
CMSSW: CMSSW_16_1_X_2026-02-04-2300/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cmssw/50039/51126/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

There are some workflows for which there are errors in the baseline:
34634.114 step 2
34634.21 step 2
34634.703 step 2
34634.9921 step 3
34634.999 step 3
34634.99 step 3
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

  • You potentially added 24 lines to the logs
  • ROOTFileChecks: Some differences in event products or their sizes found
  • Reco comparison results: 6883 differences found in the comparisons
  • DQMHistoTests: Total files compared: 53
  • DQMHistoTests: Total histograms compared: 4172042
  • DQMHistoTests: Total failures: 25329
  • DQMHistoTests: Total nulls: 28
  • DQMHistoTests: Total successes: 4146665
  • DQMHistoTests: Total skipped: 20
  • DQMHistoTests: Total Missing objects: 0
  • DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
  • Checked 242 log files, 212 edm output root files, 53 DQM output files
  • TriggerResults: found differences in 3 / 51 workflows

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants