Fixing pcidevice device plugin stop deadlock

[WebberHuang1118]

Problem:
pci controller stops the device plugin with potential deadlock

Solution:
Refactor the device plugin stop flow to terminate it gracefully, the flow would like this:

• Device plugin close chan dp.done and wait the notification from chan dp.deregistered
• healthCheck()\performCheck() is kicked by chan dp.done and gets out of the loop
• ListAndWatch() is kicked by chan dp.done and gets out of the loop
• ListAndWatch() sends an empty list response and closes chan dp.deregistered
• stopDevicePlugin() is notified by chan dp.deregistered and stops the server gracefully

Related Issue:
harvester/harvester#5164

Test Plan:

• Enable and disable one pcidevice passthrough
• There should be not error message similar as device plugin failed to deregister: rpc error: code = Unavaila │ ble desc = transport is closing","pos":"device_manager.go:250","reason":"rpc error: code = Unavailable desc = transport is closing" in the pci controller daemonset

[Yu-Jack]

I think we just need to add following codes before L165.

if !dp.starter.started {
    return nil
}

close(dp.starter.stopChan)
dp.starter.started = false

pcidevices/pkg/deviceplugins/device_manager.go

Lines 164 to 166 in feb7e8c

	func (dp *PCIDevicePlugin) Stop() error {
	return dp.stopDevicePlugin()
	}

The reason is stop and done are kind of different thoughts here.

• stop is more like someone controls this object from outside, it will tell this object should be stopped.
• done is more like object inner status, it tells whole object about current progress.

Just like I mentioned in that PR #67 (comment), our structure lacks of some objects, but we still used the similar flow of KubeVirt's device plugin. I think we could keep it original flow for now until we have time to redesign it based on our structure.

Then, we won't need to delete all stop channel in the other PR. How do you think about it?

[Yu-Jack]

BTW, this is commit what I changed (Yu-Jack@bac0ffd)

[mergify]

This pull request is now in conflict. Could you fix it @WebberHuang1118? 🙏