Add more options to df_engine docker build, slim down features for some packages (open-telemetry#1540)

This PR addresses a few problems that I ran into getting `df_engine`
running in my kubernetes environment. The end result of all this is that
you can now run a docker build like this: ` docker build --build-arg
FEATURES="azure" --build-arg RUSTFLAGS="-C target-feature=-gfni"
--build-arg DEBUG=1 --build-context otel-arrow=../../ -f Dockerfile -t
df_engine .`

# TLS

Some of the Azure SDK crates + reqwest have a lot of default features
enabled that we are not using and which rely on linking with openssl for
TLS. This was giving me trouble with our musl build.

I've disabled default features for these packages at the workspace level
and opted for using `rustls-tls` with reqwest in the `otap` crate.

# Configuring the docker build

- Added the `FEATURES` build arg to the dockerfile and the cross
platform build script so that you can turn on features like `azure` when
building
- Added the ability to specify `RUSTFLAGS` for the docker build so that
we can turn some features on or off when compiling which brings me to my
next item...

# Adding some optional debug utils to the image

When running on k8s I was getting very mysterious crashes after a couple
of seconds with 0 logs to stdout. Remoting into the container and
running the binary gave me the "Illegal instruction (core dumped)"
message, which was a start.

After checking the obvious vector extensions like SSE/AVX/AVX512, I was
stumped. So, I added gdb to the container and found the exact
instruction which is the longest mnemonic I have ever seen (comes from
arrow's buffer crate):

```
(gdb)  x/i $pc
=> 0x7ffff745fb4f <new+463>:    vgf2p8affineqb $0x0,%ymm3,%ymm5,%ymm5
```

Turns out my laptop has support for a specific feature called `gfni`,
which the nodes with Intel Xeon processors that I was running did not.

I think this will not be the last time someone needs to debug a native
code/cross-compilation problem, so I added an option to the build called
`DEBUG` that will include `gdb` (and maybe other debug utilities in the
future) in the image.

Author: JakeDern

rust/otap-dataflow/Cargo.toml

@@ -85,7 +85,7 @@ miette = { version="7.6.0", features = ["fancy"] }
 mimalloc-rust = "0.2.1"
 nix = { version = "0.30.1", features = ["process", "signal"] }
 num_enum = "0.7"
-object_store = "0.12.3"
+object_store = {version = "0.12.3", default-features = false}
 once_cell = "1.20.2"
 opentelemetry = "0.31.0"
 opentelemetry-proto = { version = "0.31", default-features = false, features = ["gen-tonic-messages", "logs"]} #TODO - use it from submodule instead of crate(?)
@@ -150,10 +150,10 @@ zip = "=4.2.0"
 byte-unit = "5.2.0"
 
 # Azure Monnitor Exporter
-azure_identity = "0.30.0"
+azure_core = {version = "0.30.1", default-features = false }
+azure_identity = {version = "0.30.0", default-features = false }
 flate2 = "1.1.5"
-reqwest = "0.12.24"
-azure_core = "0.30.1"
+reqwest = { version = "0.12.24", default-features = false }
 time = "0.3.44"
 wiremock = "0.6.5"
 

rust/otap-dataflow/Dockerfile

@@ -22,12 +22,27 @@ COPY --from=otel-arrow /proto/opentelemetry-proto /build/proto/opentelemetry-pro
 
 COPY . /build/rust/dataflow/.
 
+# RUSTFLAGS can be used to pass argument to rustc e.g. 
+# --build-arg RUSTFLAGS="-C target-feature=-gfni"
+ARG RUSTFLAGS="-C target-cpu=native"
+ENV RUSTFLAGS=${RUSTFLAGS}
+
+# These are used to enable features and are passed to carg's --features argument
+# e.g. --build-arg FEATURES="azure,quiver-persistence"
+ARG FEATURES
+ENV FEATURES=${FEATURES}
+
 # Build dataflow engine
 RUN ./cross-arch-build.sh
 
 # The runtime image
 FROM docker.io/alpine:3.23@sha256:51183f2cfa6320055da30872f211093f9ff1d3cf06f39a0bdb212314c5dc7375
 LABEL maintainer="The OpenTelemetry Authors"
+
+# Passing --build-arg BUILD=1 adds additional debugging features to the image.
+ARG DEBUG="0"
+RUN if [ "$DEBUG" = "1" ]; then apk add --no-cache gdb; fi
+
 RUN addgroup dataflow \
   && adduser \
   --ingroup dataflow \

rust/otap-dataflow/crates/otap/Cargo.toml

@@ -22,7 +22,7 @@ futures-timer.workspace = true
 http.workspace = true
 humantime-serde.workspace = true
 log.workspace = true
-object_store.workspace = true
+object_store = {workspace = true, features = ["fs"]}
 parquet.workspace = true
 thiserror = { workspace = true }
 serde_with = { workspace = true }
@@ -70,8 +70,8 @@ geneva-uploader = { version = "0.3.0", optional = true }
 # Azure Monitor Exporter dependencies
 azure_identity = { workspace = true, optional = true }
 flate2 = { workspace = true, optional = true }
-reqwest = { workspace = true, optional = true }
-azure_core = { workspace = true, optional = true }
+reqwest = { workspace = true, optional = true, features = ["rustls-tls"] }
+azure_core = { workspace = true, optional = true,  features = ["reqwest"] }
 
 # OpenTelemetry proto with tonic feature enabled
 opentelemetry-proto = { workspace = true, optional = true }

rust/otap-dataflow/cross-arch-build.sh

@@ -16,5 +16,5 @@ else
 fi
 
 rustup target add "${RUST_TARGET}"
-cargo build --release --target="${RUST_TARGET}"
-cp "target/${RUST_TARGET}/release/df_engine" .
+cargo build --release --features "$FEATURES" --target="${RUST_TARGET}"
+cp "target/${RUST_TARGET}/release/df_engine" .