Add critest as step to buildkite pipeline.

Add in step to buildkite pipeline for critest. Use fcnet configuration
rather than the example CNI plugin configurations in
tools/docker/critest

This step will diff the current known status of failing tests with
output in the pipeline such that we can maintain a state of known
failures while not failing the pipeline on every new build. The version
of ginkgo used by critest does not support output to a format file, so
these changes include some scripting to capture output and compare.

Current passing tests are related to image management.

Current tests failing are due to network namespacing issues. Host and VM
are not sharing the same network namespace. critest (crictl) creates CNI
namespace to be used for pod sandbox. FC-CD will create this namespace
as well as a network namespace for the VM itself, but critest cannot
find the CNI namespace as it is contained within the VM's network
namespace.

Signed-off-by: Gavin Inglis <[email protected]>

Author: ginglis13

runtime/Makefile

@@ -153,6 +153,7 @@ critest:
 		--volume $(CURDIR)/../examples/etc/containerd/firecracker-runtime.json:/etc/containerd/firecracker-runtime.json \
 		--volume $(CURDIR)/../tools/demo/fc-br0.interface:/etc/network/interfaces.d/fc-br0 \
 		--volume $(CURDIR)/logs:/var/log/firecracker-containerd-test \
+		--volume $(CURDIR)/../tools/critest:/src/runtime/critest \
 		--volume $(GO_CACHE_VOLUME_NAME):/go \
 		--env FICD_DM_VOLUME_GROUP=$(FICD_DM_VOLUME_GROUP) \
 		--env FICD_DM_POOL=$(FICD_DM_POOL) \
@@ -161,7 +162,8 @@ critest:
       	--env ACK_GINKGO_DEPRECATIONS=1.16.5 \
 		--workdir="/src/runtime" \
 		$(FIRECRACKER_CONTAINERD_TEST_IMAGE):$(DOCKER_IMAGE_TAG) \
-		"make -C ../examples testtap && critest -runtime-endpoint unix:///run/firecracker-containerd/containerd.sock"
+		"sleep 1 && critest -ginkgo.noColor -runtime-endpoint unix:///run/firecracker-containerd/containerd.sock | \
+		./critest/critest_diff.sh"
 
 clean:
 	- rm -f containerd-shim-aws-firecracker

tools/critest/critest_diff.sh

@@ -0,0 +1,36 @@
+#! /bin/bash
+#
+# Copyright Amazon.com, Inc. or its affiliates. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License"). You may
+# not use this file except in compliance with the License. A copy of the
+# License is located at
+#
+# 	http://aws.amazon.com/apache2.0/
+#
+# or in the "license" file accompanying this file. This file is distributed
+# on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either
+# express or implied. See the License for the specific language governing
+# permissions and limitations under the License.
+
+
+# Create temporary critest output file
+critest_output="$(</dev/stdin)"
+critest_output_file="$(mktemp)"
+echo "$critest_output" >> "$critest_output_file"
+
+set -eu
+
+# Remove up until report summary
+sed -i -E '0,/^Summarizing [0-9][0-9]? Failure[s]?:$/d' "$critest_output_file" # Remove empty lines
+sed -i '/^$/d' "$critest_output_file"
+
+# Remove unnecessary error messages
+sed -i '/^\/.*[0-9]$/d' "$critest_output_file"
+sed -i '/^Ran [0-9][0-9] of [0-9][0-9] Specs in .*seconds$/d' "$critest_output_file"
+sed -i '/^--- FAIL: TestCRISuite.*$/d' "$critest_output_file"
+sed -i '/^FAIL.*$/d' "$critest_output_file"
+sed -i '/^Ran.*$/d' "$critest_output_file"
+
+# Diff expected vs. actual
+diff -y <(sort critest/expected_critest_output.out) <(sort "$critest_output_file")

tools/critest/expected_critest_output.out

@@ -0,0 +1,67 @@
+[Fail] [k8s.io] Security Context SeccompProfilePath [It] runtime should support an seccomp profile that blocks setting hostname with SYS_ADMIN 
+[Fail] [k8s.io] Security Context SeccompProfilePath [It] should support seccomp localhost/profile on the container 
+[Fail] [k8s.io] Container runtime should support adding volume and device [BeforeEach] runtime should support starting container with volume [Conformance] 
+[Fail] [k8s.io] Security Context bucket [It] runtime should support that ReadOnlyRootfs is false 
+[Fail] [k8s.io] PodSandbox runtime should support basic operations on PodSandbox [It] runtime should support running PodSandbox [Conformance] 
+[Fail] [k8s.io] Security Context NamespaceOption [It] runtime should support HostIpc is false 
+[Fail] [k8s.io] Security Context NamespaceOption [It] runtime should support ContainerPID 
+[Fail] [k8s.io] Security Context bucket [It] runtime should return error if RunAsGroup is set without RunAsUser 
+[Fail] [k8s.io] Security Context NoNewPrivs [BeforeEach] should allow privilege escalation when false 
+[Fail] [k8s.io] Networking runtime should support networking [It] runtime should support set hostname [Conformance] 
+[Fail] [k8s.io] Security Context bucket [It] runtime should support dropping ALL capabilities 
+[Fail] [k8s.io] Security Context bucket [It] runtime should support dropping capability 
+[Fail] [k8s.io] Security Context SeccompProfilePath docker/default [It] runtime should support setting hostname with docker/default seccomp profile and SYS_ADMIN 
+[Fail] [k8s.io] Security Context bucket [It] runtime should support ReadonlyPaths 
+[Fail] [k8s.io] Security Context bucket [It] runtime should support MaskedPaths 
+[Fail] [k8s.io] PodSandbox runtime should support basic operations on PodSandbox [It] runtime should support removing PodSandbox [Conformance] 
+[Fail] [k8s.io] Container Mount Propagation runtime should support mount propagation [BeforeEach] mount with 'rshared' should support propagation from host to container and vice versa 
+[Fail] [k8s.io] Security Context bucket [It] runtime should support that ReadOnlyRootfs is true 
+[Fail] [k8s.io] Security Context NamespaceOption [It] runtime should support HostPID 
+[Fail] [k8s.io] Security Context bucket [It] runtime should support RunAsUserName 
+[Fail] [k8s.io] Container runtime should support log [BeforeEach] runtime should support starting container with log [Conformance] 
+[Fail] [k8s.io] Container Mount Propagation runtime should support mount propagation [BeforeEach] mount with 'rslave' should support propagation from host to container 
+[Fail] [k8s.io] Security Context bucket [It] runtime should support SupplementalGroups 
+[Fail] [k8s.io] Security Context bucket [It] runtime should support Privileged is true 
+[Fail] [k8s.io] Security Context NamespaceOption [It] runtime should support HostNetwork is true 
+[Fail] [k8s.io] Security Context SeccompProfilePath [It] runtime should not support a custom seccomp profile without using localhost/ as a prefix 
+[Fail] [k8s.io] Security Context SeccompProfilePath [It] runtime should ignore a seccomp profile that blocks setting hostname when privileged 
+[Fail] [k8s.io] Streaming runtime should support streaming interfaces [It] runtime should support exec with tty=false and stdin=false [Conformance] 
+[Fail] [k8s.io] Multiple Containers [Conformance] when running multiple containers in a pod [BeforeEach] should support network 
+[Fail] [k8s.io] Container runtime should support basic operations on container [BeforeEach] runtime should support stopping container [Conformance] 
+[Fail] [k8s.io] Multiple Containers [Conformance] when running multiple containers in a pod [BeforeEach] should support container exec 
+[Fail] [k8s.io] Security Context NoNewPrivs [BeforeEach] should not allow privilege escalation when true 
+[Fail] [k8s.io] Container runtime should support basic operations on container [BeforeEach] runtime should support execSync [Conformance] 
+[Fail] [k8s.io] Container runtime should support basic operations on container [BeforeEach] runtime should support execSync with timeout [Conformance] 
+[Fail] [k8s.io] Streaming runtime should support streaming interfaces [It] runtime should support attach [Conformance] 
+[Fail] [k8s.io] Container runtime should support basic operations on container [BeforeEach] runtime should support starting container [Conformance] 
+[Fail] [k8s.io] Security Context SeccompProfilePath [It] runtime should not block setting host name with unconfined seccomp and SYS_ADMIN 
+[Fail] [k8s.io] PodSandbox runtime should support sysctls [It] should support unsafe sysctls 
+[Fail] [k8s.io] PodSandbox runtime should support basic operations on PodSandbox [It] runtime should support stopping PodSandbox [Conformance] 
+[Fail] [k8s.io] Streaming runtime should support streaming interfaces [It] runtime should support exec with tty=true and stdin=true [Conformance] 
+[Fail] [k8s.io] Security Context SeccompProfilePath docker/default [It] runtime should block sethostname with docker/default seccomp profile and no extra caps 
+[Fail] [k8s.io] Security Context SeccompProfilePath docker/default [It] should support seccomp docker/default on the container 
+[Fail] [k8s.io] Networking runtime should support networking [It] runtime should support DNS config [Conformance] 
+[Fail] [k8s.io] Security Context bucket [It] runtime should support RunAsUser 
+[Fail] [k8s.io] Container runtime should support log [BeforeEach] runtime should support reopening container log [Conformance] 
+[Fail] [k8s.io] Container runtime should support adding volume and device [BeforeEach] runtime should support starting container with volume when host path is a symlink [Conformance] 
+[Fail] [k8s.io] Security Context bucket [It] runtime should support adding capability 
+[Fail] [k8s.io] Multiple Containers [Conformance] when running multiple containers in a pod [BeforeEach] should support container log 
+[Fail] [k8s.io] Streaming runtime should support streaming interfaces [It] runtime should support portforward [Conformance] 
+[Fail] [k8s.io] Container Mount Propagation runtime should support mount propagation [BeforeEach] mount with 'rprivate' should not support propagation 
+[Fail] [k8s.io] Container runtime should support basic operations on container [BeforeEach] runtime should support creating container [Conformance] 
+[Fail] [k8s.io] Container runtime should support basic operations on container [BeforeEach] runtime should support removing running container [Conformance] 
+[Fail] [k8s.io] Security Context NamespaceOption [It] runtime should support PodPID 
+[Fail] [k8s.io] Container runtime should support basic operations on container [BeforeEach] runtime should support listing container stats [Conformance] 
+[Fail] [k8s.io] Networking runtime should support networking [It] runtime should support port mapping with host port and container port [Conformance] 
+[Fail] [k8s.io] Security Context bucket [It] runtime should support RunAsGroup 
+[Fail] [k8s.io] Security Context SeccompProfilePath [It] should support seccomp unconfined on the container 
+[Fail] [k8s.io] PodSandbox runtime should support sysctls [It] should support safe sysctls 
+[Fail] [k8s.io] Security Context bucket [It] runtime should support adding ALL capabilities 
+[Fail] [k8s.io] Security Context SeccompProfilePath [It] should support seccomp default which is unconfined on the container 
+[Fail] [k8s.io] Container runtime should support basic operations on container [BeforeEach] runtime should support removing stopped container [Conformance] 
+[Fail] [k8s.io] Security Context bucket [It] runtime should support Privileged is false 
+[Fail] [k8s.io] Security Context NamespaceOption [It] runtime should support HostIpc is true 
+[Fail] [k8s.io] Security Context NamespaceOption [It] runtime should support HostNetwork is false 
+[Fail] [k8s.io] Streaming runtime should support streaming interfaces [It] runtime should support portforward in host network 
+[Fail] [k8s.io] Networking runtime should support networking [It] runtime should support port mapping with only container port [Conformance] 
+[Fail] [k8s.io] Container runtime should support basic operations on container [BeforeEach] runtime should support removing created container [Conformance]