test config file update under different scenarios

[tao12345666333]

can you add a followup pr to test config file update under different scenarios?

Originally posted by @rootfs in #84 (comment)

[Xunzhuo]

/area test-and-release
/priority P1
/milestone v0.1

[tao12345666333]

/assign