Sealing won't start when using v1.16.0, stuck at Packing stage

[potato1992]

All works well when using v1.14, when upgrade into v1.16.0, the sealing won't start. All newly added sectors just stuck at Packing stage, the miner log:

2022-07-01T08:37:57.140+0800	DEBUG	advmgr	sector-storage/sched.go:480	SCHED try assign sqi:27 sector 20899 to window 15 (awi:1)
2022-07-01T08:37:57.140+0800	DEBUG	advmgr	sector-storage/sched_resources.go:98	sched: not scheduling on worker 15f18321-8637-430e-afe6-3d021b965a45 for schedAssign; not enough threads, need 0, 0 in use, target 0

Why target =0 ? The miner worker:

Worker 15f18321-8637-430e-afe6-3d021b965a45, host ubuntu0_docker1
	CPU:  [                                                                ] 0/8 core(s) in use
	RAM:  [|||                                                             ] 4% 5.501 GiB/125.7 GiB
	VMEM: [|||                                                             ] 4% 5.501 GiB/125.7 GiB

It clearly shows that 8 cores are available.

By the way I have changed the code in sched_resources.go, (a.cpuUse+1 there)

	if a.cpuUse+1+needRes.Threads(res.CPUs, len(res.GPUs)) > res.CPUs {
		log.Debugf("sched: not scheduling on worker %s for %s; not enough threads, need %d, %d in use, target %d", wid, caller, needRes.Threads(res.CPUs, len(res.GPUs)), a.cpuUse, res.CPUs)
		return false
	}

[potato1992]

update, after some time of debugging, now I have solved it.
I found res.CPUs won't be populated until a task is assigned in v1.16.0 (res.CPUs==0), but this is not the case in v1.14.0. This must be a bug introduced in newer version after v1.14.0. Then I changed the code in shed_resources.go:

	if res.CPUs==0{
		/* do not +1 in this case */
		if a.cpuUse+needRes.Threads(res.CPUs, len(res.GPUs)) > res.CPUs {
			log.Debugf("sched: not scheduling on worker %s for %s; not enough threads, need %d, %d in use, target %d", wid, caller, needRes.Threads(res.CPUs, len(res.GPUs)), a.cpuUse, res.CPUs)
			return false
		}
	} else{
		if a.cpuUse+1+needRes.Threads(res.CPUs, len(res.GPUs)) > res.CPUs {
			log.Debugf("sched: not scheduling on worker %s for %s; not enough threads, need %d, %d in use, target %d", wid, caller, needRes.Threads(res.CPUs, len(res.GPUs)), a.cpuUse, res.CPUs)
			return false
		}
	}

I use +1 in my user case as the sealing will be slower if it uses up all the cores, thus always reserving one would be a bit faster.
The log:

2022-07-02T14:24:14.702+0800	DEBUG	advmgr	sector-storage/sched.go:480	SCHED try assign sqi:6 sector 20899 to window 16 (awi:0)
2022-07-02T14:24:14.702+0800	DEBUG	advmgr	sector-storage/sched_resources.go:100	sched: not scheduling on worker 15f18321-8637-430e-afe6-3d021b965a45 for schedAssign; not enough threads, need 0, 1 in use, target 0
2022-07-02T14:24:14.702+0800	DEBUG	advmgr	sector-storage/sched_resources.go:105	sched: not scheduling on worker 15f18321-8637-430e-afe6-3d021b965a45 for compactWindows; not enough threads, need 1, 7 in use, target 8
2022-07-02T14:24:14.702+0800	DEBUG	advmgr	sector-storage/sched_resources.go:105	sched: not scheduling on worker 15f18321-8637-430e-afe6-3d021b965a45 for startPreparing; not enough threads, need 1, 7 in use, target 8

res.CPUs changed from 0 to the correct value 8 when the task is assigned once.