Skip to content

Conversation

@lfrancke
Copy link
Member

@lfrancke lfrancke commented Dec 11, 2024

Description

This PR bundles a few (hopefully) minor changes to our workflows:

  • This uses the Stackable build repo which mirrors maven central and a few other repositories needed by our products
    • We are hoping this is more stable than upstream Maven central where we see frequent connection errors in our builds
  • Adds aCI change to prevent matrix jobs from canceling when one job fails. This should mean there are less jobs to rerun on failure
  • Adds a patch to Druid to not depend on fmpp 0.9.14 anymore which does depend on a version range for Freemarker which would resolve to a SNAPSHOT dependency
  • Adds a patch to Kafka to update the repositories used by Gradle
  • Hopefully consistently adds --batch-mode and --no-transfer-progress to all Maven based builds

Definition of Done Checklist

Make sure all these products build and use the Maven pull-through cache in Nexus:

### Products
- [x] Druid
- [x] HBase
- [x] Hadoop HDFS
- [x] Hive
- [x] Kafka
- [x] NiFi
- [x] Spark
- [x] Trino
- [x] ZooKeeper
- [x] Add an entry to the CHANGELOG.md file
TIP: Running integration tests with a new product image

The image can be built and uploaded to the kind cluster with the following commands:

bake --product <product> --image-version <stackable-image-version>
kind load docker-image <image-tagged-with-the-major-version> --name=<name-of-your-test-cluster>

See the output of bake to retrieve the image tag for <image-tagged-with-the-major-version>.

@lfrancke lfrancke force-pushed the push-wtnvozywmlzk branch 3 times, most recently from 285ffac to 9fb6446 Compare December 12, 2024 05:40
lfrancke and others added 5 commits December 12, 2024 06:44
This is because FMPP Maven Plugin depends on FMPP in version 0.9.14
which itself depends on a Freemarker version that has not been pinned.
Instead it specifies a "range" which resolves to a SNAPSHOT version
which we don't want.
NickLarsenNZ
NickLarsenNZ previously approved these changes Dec 12, 2024
Copy link
Member

@NickLarsenNZ NickLarsenNZ left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lfrancke lfrancke marked this pull request as ready for review December 12, 2024 09:24
@lfrancke lfrancke changed the title Use our build-repo for Maven downloads Use our build-repo for Maven downloads & other updates Dec 12, 2024
Copy link
Member

@NickLarsenNZ NickLarsenNZ left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just one suggestion

@lfrancke lfrancke enabled auto-merge December 12, 2024 09:34
@NickLarsenNZ NickLarsenNZ self-requested a review December 12, 2024 09:36
Copy link
Member

@NickLarsenNZ NickLarsenNZ left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@lfrancke lfrancke added this pull request to the merge queue Dec 12, 2024
Merged via the queue into main with commit efd9b9d Dec 12, 2024
3 checks passed
@lfrancke lfrancke deleted the push-wtnvozywmlzk branch December 12, 2024 09:37
NickLarsenNZ added a commit that referenced this pull request Dec 13, 2024
* Use our build-repo for Maven downloads

* ci(build): Continue matrix builds on failure so there is less jobs to rerun when there are transient failures

* Update Spark Mvn build as well

* Hopefully silence Spark build

* Update FMPP version

This is because FMPP Maven Plugin depends on FMPP in version 0.9.14
which itself depends on a Freemarker version that has not been pinned.
Instead it specifies a "range" which resolves to a SNAPSHOT version
which we don't want.

* Add Stackable repo to Kafka build as well

* Update changelog

* Update CHANGELOG.md

Co-authored-by: Nick <[email protected]>

---------

Co-authored-by: Nick Larsen <[email protected]>
Co-authored-by: Nick <[email protected]>
NickLarsenNZ added a commit that referenced this pull request Dec 13, 2024
* ci: remove redundant usages of the free-disk-space action

Co-authored-by: Techassi <[email protected]>

* ci: Add attempt number to the workflow run name, and URL in the slack message (#941)

Co-authored-by: Techassi <[email protected]>

* ci: register a dummy test_build_images workflow which will be completed in a separate PR (#947)

* Use our build-repo for Maven downloads & other updates (#953)

* Use our build-repo for Maven downloads

* ci(build): Continue matrix builds on failure so there is less jobs to rerun when there are transient failures

* Update Spark Mvn build as well

* Hopefully silence Spark build

* Update FMPP version

This is because FMPP Maven Plugin depends on FMPP in version 0.9.14
which itself depends on a Freemarker version that has not been pinned.
Instead it specifies a "range" which resolves to a SNAPSHOT version
which we don't want.

* Add Stackable repo to Kafka build as well

* Update changelog

* Update CHANGELOG.md

Co-authored-by: Nick <[email protected]>

---------

Co-authored-by: Nick Larsen <[email protected]>
Co-authored-by: Nick <[email protected]>

* ci: Fix static analysis errors (#955)

* ci: Fix high severity errors

* ci: Set persist-credentials to false

* ci: Move step output to env var

* chore: Add changelog entry

* ci(build): stagger the scheduled builds to ease load on external systems (#954)

---------

Co-authored-by: Techassi <[email protected]>
Co-authored-by: Lars Francke <[email protected]>
@nightkr nightkr mentioned this pull request Feb 17, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants