Deployment of functions failing while using PR 465

[aditya2803]

Describe the bug
I followed the steps to set up vHive using #465 as the base. All pods are running as expected. However, the deployment of functions is failing.

EDIT: I tried using the main branch of the repo, but ended up getting the same issue.

To Reproduce
Follow steps in the quick-start guide to set up vHive, using #465 as the base branch. Then, try to deploy the functions using source /etc/profile && pushd ./examples/deployer && go build && popd && ./examples/deployer/deployer

Expected behavior
Functions should get deployed normally.

Logs
vhive.stdout logs

time="2022-07-09T07:00:04.058528225Z" level=info msg="HEARTBEAT: number of active VMs: 0"
time="2022-07-09T07:00:04.076796785Z" level=info msg="FuncPool heartbeat: ==== Stats by cold functions ====\nfID, #started, #served\n==================================="
time="2022-07-09T07:01:04.058799087Z" level=info msg="HEARTBEAT: number of active VMs: 0"
time="2022-07-09T07:01:04.077084372Z" level=info msg="FuncPool heartbeat: ==== Stats by cold functions ====\nfID, #started, #served\n==================================="
time="2022-07-09T07:01:34.544142901Z" level=error msg="coordinator failed to start VM" error="Failed to get/pull image: failed to stat snapshot sha256:5216338b40a7b96416b8b9858974bbe4acc3096ee60acbc4dfb1ee02aecceb10: snapshotter not loaded: devmapper: invalid argument" image="ghcr.io/ease-lab/helloworld:var_workload" vmID=95
time="2022-07-09T07:01:34.544195993Z" level=error msg="failed to start VM" error="Failed to get/pull image: failed to stat snapshot sha256:5216338b40a7b96416b8b9858974bbe4acc3096ee60acbc4dfb1ee02aecceb10: snapshotter not loaded: devmapper: invalid argument"
time="2022-07-09T07:01:34.549605400Z" level=error msg="VM config for pod 8d68f00f4e584c382e5bb69c03bcb6298241b2efe9a520126c2f18f3d7f4e935 does not exist"
time="2022-07-09T07:01:34.549622754Z" level=error error="VM config for pod does not exist"
time="2022-07-09T07:01:38.484179501Z" level=error msg="coordinator failed to start VM" error="Failed to get/pull image: failed to stat snapshot sha256:5216338b40a7b96416b8b9858974bbe4acc3096ee60acbc4dfb1ee02aecceb10: snapshotter not loaded: devmapper: invalid argument" image="ghcr.io/ease-lab/helloworld:var_workload" vmID=96

firecracker.stdout

time="2022-07-09T07:10:39.230513171Z" level=debug msg="(*service).Write started" expected="sha256:12dc6715ed1a8306f246ceaf7742c09e38a52a79d17421e4a50d7e0e09fdbc25" ref="manifest-sha256:12dc6715ed1a8306f246ceaf7742c09e38a52a79d17421e4a50d7e0e09fdbc25" total=1998
time="2022-07-09T07:10:39.519726077Z" level=debug msg="(*service).Write started" expected="sha256:a87533385b75fd1d476d8a1112a0c5db953e3d5b44d4f9db814b1e2e6abb8734" ref="config-sha256:a87533385b75fd1d476d8a1112a0c5db953e3d5b44d4f9db814b1e2e6abb8734" total=8312
time="2022-07-09T07:10:39.851796881Z" level=debug msg="stat snapshot" key="sha256:5216338b40a7b96416b8b9858974bbe4acc3096ee60acbc4dfb1ee02aecceb10"
time="2022-07-09T07:10:39.865959643Z" level=debug msg="remove content" key="sha256:12dc6715ed1a8306f246ceaf7742c09e38a52a79d17421e4a50d7e0e09fdbc25"
time="2022-07-09T07:10:39.866041891Z" level=debug msg="remove content" key="sha256:a87533385b75fd1d476d8a1112a0c5db953e3d5b44d4f9db814b1e2e6abb8734"
time="2022-07-09T07:10:39.866301996Z" level=debug msg="schedule content cleanup"
time="2022-07-09T07:10:39.866599162Z" level=debug msg="removed content" digest="sha256:12dc6715ed1a8306f246ceaf7742c09e38a52a79d17421e4a50d7e0e09fdbc25"
time="2022-07-09T07:10:39.866708048Z" level=debug msg="removed content" digest="sha256:a87533385b75fd1d476d8a1112a0c5db953e3d5b44d4f9db814b1e2e6abb8734"
time="2022-07-09T07:10:39.866769233Z" level=debug msg="content garbage collected" d="418.05µs"
time="2022-07-09T07:10:39.866805235Z" level=debug msg="garbage collected" d="474.151µs"
time="2022-07-09T07:10:55.974576385Z" level=debug msg="(*service).Write started" expected="sha256:12dc6715ed1a8306f246ceaf7742c09e38a52a79d17421e4a50d7e0e09fdbc25" ref="manifest-sha256:12dc6715ed1a8306f246ceaf7742c09e38a52a79d17421e4a50d7e0e09fdbc25" total=1998
time="2022-07-09T07:10:56.272910625Z" level=debug msg="(*service).Write started" expected="sha256:a87533385b75fd1d476d8a1112a0c5db953e3d5b44d4f9db814b1e2e6abb8734" ref="config-sha256:a87533385b75fd1d476d8a1112a0c5db953e3d5b44d4f9db814b1e2e6abb8734" total=8312
time="2022-07-09T07:10:56.613195119Z" level=debug msg="stat snapshot" key="sha256:5216338b40a7b96416b8b9858974bbe4acc3096ee60acbc4dfb1ee02aecceb10"
time="2022-07-09T07:10:56.623078214Z" level=debug msg="remove content" key="sha256:12dc6715ed1a8306f246ceaf7742c09e38a52a79d17421e4a50d7e0e09fdbc25"
time="2022-07-09T07:10:56.623158747Z" level=debug msg="remove content" key="sha256:a87533385b75fd1d476d8a1112a0c5db953e3d5b44d4f9db814b1e2e6abb8734"
time="2022-07-09T07:10:56.623462626Z" level=debug msg="schedule content cleanup"
time="2022-07-09T07:10:56.623783386Z" level=debug msg="removed content" digest="sha256:12dc6715ed1a8306f246ceaf7742c09e38a52a79d17421e4a50d7e0e09fdbc25"
time="2022-07-09T07:10:56.623911337Z" level=debug msg="removed content" digest="sha256:a87533385b75fd1d476d8a1112a0c5db953e3d5b44d4f9db814b1e2e6abb8734"
time="2022-07-09T07:10:56.624000453Z" level=debug msg="content garbage collected" d="482.167µs"
time="2022-07-09T07:10:56.624069428Z" level=debug msg="garbage collected" d="502.296µs"

Notes
Running on bare-metal, Ubuntu 20.04, Intel Xeon machine.

[aditya2803]

Hi @ustiugov, could you check this once ?

[aditya2803]

Issue got resolved after installation on a fresh disk. Some issue with the old disk perhaps. Closing the issue.