perf: Enable compression of audit logs, and increase max file size

We enable compression of audit logs. Whenever a log file is rotated, it
is compressed.

Experiments show that gzip yields a compression factor of approximately
12 for audit logs, so we increase the maximum file size by that factor.

With the previous configuration, we had 1 log file of 100MB, and 10
rotated log files of 100MB each, for a total of 1100MB. That's our
target disk space requirement.

We want to keep this disk space usage, but enable compression. We can't
just make the maximum file size larger. For example, if we increase it
to 1000MB, then we'll end up with 1 1000MB log, plus 10 compressed,
rotated logs of approximately 100MB each, for a total of 2000MB, much
higher than our target.

To increase retention, we reduce the size of the uncompressed log. We
keep --audit-log-maxsize unchanged at 100, but increase
--audit-log-maxbackup to 90. We end up with 1 100MB uncompressed log,
plus 90 compressed, rotated logs of approximately 10MB each, for a total
of 1000MB. That's a little below our target!

Author: dlipovetsky

pkg/handlers/generic/mutation/auditpolicy/inject.go

@@ -69,10 +69,15 @@ func (h *auditPolicyPatchHandler) Mutate(
 				apiServer.ExtraArgs = make(map[string]string, 5)
 			}
 
+			// Originally, we had 1 log of 100MB, and 10 rotated logs of 100MB each, for a total of 1100MB.
+			// We wanted to increase retention, but keep the total disk usage about the same.
+			// Now, we have 1 log of 100MB, and 90 compressed, rotated logs of approximately 10MB each,
+			// for a total of approximately 1000MB.
 			apiServer.ExtraArgs["audit-log-path"] = "/var/log/audit/kube-apiserver-audit.log"
-			apiServer.ExtraArgs["audit-log-maxage"] = "30"
-			apiServer.ExtraArgs["audit-log-maxbackup"] = "10"
-			apiServer.ExtraArgs["audit-log-maxsize"] = "100"
+			apiServer.ExtraArgs["audit-log-maxage"] = "30"     // Maximum number of days to retain audit log files.
+			apiServer.ExtraArgs["audit-log-maxbackup"] = "90"  // Maximum number of audit log files to retain.
+			apiServer.ExtraArgs["audit-log-maxsize"] = "100"   // Maximum size of log file before it is rotated.
+			apiServer.ExtraArgs["audit-log-compress"] = "true" // Compress (gzip) audit log file when it is rotated.
 			apiServer.ExtraArgs["audit-policy-file"] = auditPolicyPath
 
 			if apiServer.ExtraVolumes == nil {

pkg/handlers/generic/mutation/auditpolicy/inject_test.go

@@ -49,11 +49,12 @@ var _ = Describe("Generate Audit Policy patches", func() {
 						gomega.HaveKeyWithValue(
 							"extraArgs",
 							map[string]interface{}{
-								"audit-log-maxbackup": "10",
+								"audit-log-maxbackup": "90",
 								"audit-log-maxsize":   "100",
 								"audit-log-path":      "/var/log/audit/kube-apiserver-audit.log",
 								"audit-policy-file":   "/etc/kubernetes/audit-policy.yaml",
 								"audit-log-maxage":    "30",
+								"audit-log-compress":  "true",
 							},
 						),
 						gomega.HaveKeyWithValue(