Unpin td-agent and cap elasticsearch gem

With [0] the version of td-agent being installed was pinned, which isn't
sustainable in the long run, so we drop the pin.

Latest version of the elasticsearch gem no longer works with older
(OSS) versions of Elasticsearch. This is fixed by downgrading the version
of the elasticsearch gems.

Backport includes 711ec08 [1].

[0] Iefcdd3100b7e3c5320bc5f1286a18251bdeab885
[1] I353ef9b6b8da0e726c17ad2e06995d3690731e1d
Related-Bug: 1930867
Closes-Bug: 1954759
Closes-Bug: 1960444
Signed-off-by: Dr. Jens Harbott <[email protected]>
Co-Authored-By: Radosław Piliszek <[email protected]>
Change-Id: I3045786e863b098d7339d1066aef6c857aa5f97f
(cherry picked from commit a8e6924)

Author: osfrickler

docker/base/apt_preferences.debian

@@ -7,11 +7,6 @@ Package: erlang*
 Pin: version 1:23.*
 Pin-Priority: 1000
 
-# FIXME(mgoddard): Pinning to 4.0.* to avoid bug 1930867.
-Package: td-agent*
-Pin: version 4.0.*
-Pin-Priority: 1000
-
 # NOTE(mgoddard): logstash 7.9.x is the last version that supports
 # Elasticsearch OSS.
 Package: logstash-oss

docker/base/apt_preferences.ubuntu

@@ -6,11 +6,6 @@ Package: erlang*
 Pin: version 1:23.*
 Pin-Priority: 1000
 
-# FIXME(mgoddard): Pinning to 4.0.* to avoid bug 1930867.
-Package: td-agent*
-Pin: version 4.0.*
-Pin-Priority: 1000
-
 # NOTE(mgoddard): logstash 7.9.x is the last version that supports
 # Elasticsearch OSS.
 Package: logstash-oss

docker/fluentd/Dockerfile.j2

@@ -27,9 +27,8 @@ LABEL fluentd_version="0.14" fluentd_binary="td-agent"
     ] %}
 
     {% if base_arch in ['aarch64', 'x86_64'] %}
-        # FIXME(mgoddard): Pinning to 4.0.* to avoid bug 1930867.
         {% set fluentd_packages = fluentd_packages + [
-            'td-agent-4.0.*'
+            'td-agent'
         ] %}
     {% else %}
         {% set fluentd_packages = fluentd_packages + [
@@ -62,7 +61,14 @@ LABEL fluentd_version="0.14" fluentd_binary="td-agent"
 {% endif %}
 
 {{ macros.configure_user(name=fluentd_user, groups='mysql') }}
-{{ macros.install_packages(fluentd_packages | customizable("packages")) }}
+
+# NOTE(frickler): Downgrading elasticsearch gems for compatibility with OSS versions of ES
+{{ macros.install_packages(fluentd_packages | customizable("packages")) }}{% if base_arch in ['aarch64', 'x86_64'] %} \
+    && td-agent-gem install elasticsearch:7.13.0 \
+    && td-agent-gem uninstall "elasticsearch:>7.13.0" \
+    && td-agent-gem uninstall "elasticsearch-api:>7.13.0" \
+    && td-agent-gem uninstall "elasticsearch-transport:>7.13.0"
+{% endif %}
 
 # Distro specific files and operations
 # Fluentd on rpm-based non-x86_64 is installed from rpm

releasenotes/notes/cap-fluentd-elasticsearch-18c0ca8e90c1234c.yaml

@@ -0,0 +1,7 @@
+---
+fixes:
+  - |
+    Latest version of the elasticsearch gem no longer works with older
+    (OSS) versions of Elasticsearch. This is fixed by capping the version
+    of the elasticsearch gem installed into the fluentd container.
+    `LP#1954759 <https://launchpad.net/bugs/1954759>`__