GBE & Easy Foundry v4.1.1

Improvements

• Traefik now runs smoke tests after being deployed.
• Bumped the broker-registrar release to v3.5.1, removing the workaround implemented in cassandra, data-services, postgres and rabbitmq subsystems.
• Logsearch memory consumtion has been reduced.
• Version updates, see below.

Fixes

• Fixed the startup timeout of Kibana in the logsearch subsystem.

• The Postgres subsystem was not working because Prometheus v23.3.0 is not compatible with the Postgres v11.x shipped by the Postgres release v36. This version of Easy Foundry properly pins the Postgres version to v31 for Prometheus.

  A next release version of Easy Foundry will upgrade both the Postgres database and the Prometheus version. The tested upgrade path is the following:

  1. Prometheus v23.3.0 + Postgres v31
  2. Prometheus v25.0.0 + Postgres v32 (and follow instruction from these release notes)
  3. Prometheus v25.0.0 + Postgres v36

Notice

• log-cache is disabled in Cloud Foundry, because of excessive memory consumption in BOSH-Lite.

Components Versions

Component	New Version	Old Version
Træfik	1.6.0	1.5.0

GBE & Easy Foundry v4.0.0

💥 Kown issues notice 💥

• ❗️💥 Do not converge the prometheus subsystem with the Postgres release v36 shipped in this v4.0.0 of Easy Foundry. Instead, pin the Postgres release version to v31 like this in prometheus/conf/spec.yml:

  postgres_version: "31"
  postgres_sha1: bbe4151f4000f349c2ffaf72412aac9cc0a3c741

New Features

• Migrated to Redis 5.0.
• Added a new gbe recover feature that runs bosh cloud-checks (unattended with the -y flag) on each target subsystem to re-create any missing VMs, which is very useful when recreating the Garden VM.
• Add cflinkuxfs3 while bumping cf-deployment from v5.4.0 to v6.10.0 (no breaking changes, though).
• The imported_vars section now supports a from: vars-file feature that allows picking values from a variables file, located in the conf/ subdirectory of a subsystem.
• Add the new --compiled flag, so that gbe converge all --compiled first upload any compiled releases from the local cache (as populated by gbe export) to the BOSH Director.

Improvements

• Improved the Concourse subsystem in many ways, including a fixed CredHub integration, covered by Easy Foundry's exclusive smoke tests for Concourse. This also reflects in the GBE Downstream (“gbeds” for short) for a standalone Concourse: gstackio/concourse-gbeds.
• Reduce the memory used by log-cache.
• Fix issue with gbe converge all when only manifests are converged after converging manifest for one subsystem.
• Fix several issues with gbe converge list.
• Version updates, see below.

Breaking changes

• Switched to Redis 5.0 in this version. There are no guaranties that the in-place upgrade will occur smoothly.
• Moved rabbitmq.<easyfoundry-domain-name> management UI to rabbitmq.sys.<easyfoundry-domain-name>.

Notice

• This release ships Logsearch v209.1.0, in order to prepare migrating to v210.0.0 and above, that will ship ElasticSearch 6. This means that when migrating from prevision versions of Easy Foundry, you must deploy this version.

Components Versions

Component	New Version	Old Version
bosh-deployment	2019-04-12	v1.1.0+ (2018-10-26)
BOSH	268.7.0	268.2.0
CredHub CLI	2.2.1	1.5.3
Xenial Stemcells	250.38 (2019-02-13)	97.32 (2018-11-08)
Cloud Foundry deployment	6.10.0	5.4.0
CF-RabbitMQ	265.0.0	252.0.0
CF-Redis	434.2.1 (Redis 5.0)	434.0.22 (Redis 4.0)
Concourse	4.2.3	4.2.1
Træfik	1.5.0	1.2.0
Kong	1.2.0	1.1.0
Logsearch	209.1.0	209.0.2

GBE & Easy Foundry v3.0.0

New Features

• Migrated to the latest Xenial stemcells v97.x.
• GBE can now converge a unique BOSH environment layout, where the BOSH server is separated from the Garden backend that runs the managed nodes. This is best practice in order to be able to update the BOSH server with no downtime. The technology behind this is the Shadow CPI. To adopt this setup (recommended), use the ddbox-standalone-garden-env (to be converged first) along with ddbox-standalone-bosh-env, instead of the single ddbox-env. The make base-env target converges this for you.
• Easy Foundry now comes with an out-of-the-box integration between minio and scality S3 storages, the shield-v7 backup solution, and the cassandra and redis clusters. As long as you converge them in order, you'll get Cassandra and Redis automatically backuped, with the archives stored in Minio.
• Experimental CockroachDB and Kong CE subsystems have been added.
• GBE downstream projects can now be created. These are GBE projects that track an upstream “distribution” GBE repository, adding their own customizations to it. This is still experimental at the moment, but a working example GBE Downstream (“gbeds” for short) and related instructions can be found at gstackio/concourse-gbeds.
• Subsystems can now define their own cloud configs or runtime configs. This should make subsystems more modular. Many subsystems still need to be refactored in order to adopt this new paradigm.

Improvements

• Deployment order now correctly takes dependencies into consideration, when running gbe converge all or gbe converge deployments. This is also reflected in the experimental make all target, that is approximately the same as gbe converge deployments.
• The scality S3 server subsystem now works properly.
• Version updates, see below.

Fixes

• Fix cf smoke tests when behind traefik.
• Fix the gbe export feature for exporting compiled BOSH Release.
• Fix concourse sanity tests when no fly CLI is downloaded yet
• Reverted back to Docker release v30.1.0, due to stemcell incompatibility in newer v32.0.0.

Breaking changes

• The cf subsys has adopted the pxc-release with no easy upgrade path. If you're upgrading from previous version, you'll have to enable the necessary transition ops files in an intermediary gbe converge step by yourself.

Caveats

• The tcp-router in Cloud Foundry is producing a high number of haproxy zombie processes.

Components Versions

Component	New Version	Old Version
bosh-deployment	1.1.0+ (2018-10-26)	1.0.0 (2018-06-22)
BOSH	268.2.0	266.3.0
BOSH CLI	5.3.1	2.45.0
Stemcells	Xenial 97.32 (2018-11-08)	Trusty 3586.24 (2018-06-18)
Cloud Foundry deployment	5.4.0	1.40.0
Cloud Foundry CLI	6.40.0	6.33.1
Concourse	4.2.1	3.14.1
CF-MySQL	36.15.0	36.14.0
CF-RabbitMQ	252.0.0	247.0.0
CF-Redis	434.0.22	431.0.0
CockroachDB	4	n/a
Kong	1.1.0	n/a
Logsearch	209.0.2	209.0.0
Minio	2018-10-25T01-27-03Z	2018-06-08T03-49-38Z
Prometheus	23.3.0	23.0.0
Scality S3 server (Zenko CloudServer)	Beta7.2.0	n/a
SHIELD v7	7.0.8	7.0.4
SHIELD v8	8.0.16	8.0.10

GBE & Easy Foundry v2.2.0

New Features

• Add support for separate vars files, either secret or not. Exemples are provided here and here.
• Add a new --manifest option to gbe converge in order to only generate the deployment manifest and credentials.
• Add a -y flag to gbe converge, gbe delete and gbe recreate, replacing the former -n for non-interactive runs.
• Allow gbe converge, gbe delete and gbe recreate to take many subsys as argument, which is very convenient to converge only a subset of Easy Foundry.
• Add a new postgres subsys, based on the latest foked Dingo PostgreSQL, based on v0.10.2.
• Add a basic neo4j subsys, based on the work-in-progress Neo4j BOSH Release.

Improvements

• Improved error reporting on missing config vars.
• Reduced persistent disks for Elasticsearch in the logsearch subsys.
• Improved networking support, with idempotent iptables setup on the vbox host, improved setup of routes to ensure the deployed nodes are reachable, and a stub for sshuttle automation.
• Have the Redis nodes belong a dedicated network, so that security groups can be as narrowed as possible.
• Automate SSH key setup when creating a ddbox-type environment.
• Version updates, see below.

Fixes

• Don't always create heavy debug logs from bosh create-env on environment creation. Now a $GBE_DEBUG_LEVEL (integer) controls this.
• Fixed gbe routes on Linux.
• Fixed the minio subsys with correct persistent disk type.
• Fixed the redis subsys, properly adding a missing ops file in Git.
• Fixed gbe import that was generating boggus compiled releases with no packages.
• Fixed the zzz-compiled-release-helper subsys that is to be run before gbe import.
• Add workaround for the 'Too many authentication failures error when running gbe up.

Caveats

• Concourse deployment manifest and credentials must be generated before deploying the cf subsys. For this, gbe converge --manifest concourse is convenient.
• The scality subsys is not working yet in this version.
• The neo4j subsys does not deploy yet a service broker for Cloud Foundry to present a Neo4j service in its marketplace.
• The postgres subsys has a log rotation bug that can lead to full persistent disks.

Components Versions

Component	New Version	Old Version
bosh-deployment	1.0.0	n/a
BOSH	266.4.0	266.3.0
Cassandra	8	6+
Dingo PostgreSQL	0.10.2+	n/a
Træfik	1.2.0	1.1.0

GBE v2.1.0

New Features

• Added a new redis subsystem, providing Redis 3.2.8 databases. The service broker either creates shared instances on the node where the broker is running, or either gives access to a dedicated Redis node. By default, 2 such dedicated nodes are provisioned. This can be increased easily.
• Upgrade to the latest Cloud Foundry deployment version 1.40.0, which will be the last before version 2.0 is out.

Improvements

• The cf subsystem now runs its smoke tests once converged.
• Improved inline help, adding details about gbe converge list and gbe delete list.
• Use the -y flag for non-interactive invocations of gbe update, gbe converge, gbe recreate, or gbe delete, instead of -n.
• Accept many subsystems as arguments for gbe converge, gbe recreate, or gbe delete, which is very convenient for deploying a subset of Easy Foundry components.
• Improve error handling and messages.
• Enable BOSH Backup Restore (BBR) in Cloud Foundry subsystem.
• Give higher precedence to deployment variables over imported variables, in subsystems of type bosh-deployment.
• Version updates, see below.

Fixes

• Fixed network name in cassandra subsystem.
• Favor “subsystem” (or “subsys”) term over “deployment” in the inline help, in order not to confuse GBE subsystems with BOSH deployments. This is an on-going effort though, as the “deployment” wording is still used in some places.

Components Versions

Component	New Version	Old Version
BOSH	266.3.0	265.2.0
Stemcells	3586.24 (2018-06-18)	3541.24 (2018-05-07)
Cloud Foundry deployment	1.40.0	1.30.0
CF-MySQL	36.14.0	36.13.0
CF-RabbitMQ	247.0.0 (RabbitMQ 3.7)	244.0.0
CF-Redis	431.0.0	n/a
Concourse	3.14.1	3.12.0
Prometheus	23.0.0	22.0.2
Minio	2018-06-08T03-49-38Z	2018-04-27T23-33-52Z
SHIELD v8	8.0.10	8.0.8

GBE v2.0.2

New Features

• A new minio deployment has been added. We plan to use it for Cloud Foundry blobstore and SHIELD backups storage. Adding a CF Service Broker to provision buckets for CF apps would be nice too.

Improvements

• Improved error reporting when making a typo on a subsys name, which is a common error! This applies to gbe converge, gbe recreate or gbe delete.
• Better support and documentation for the “distant Virtualbox” use case. It's now proved to work fine, using the sshuttle tool to create a simple tunnel.
• AZ placement for Cassandra nodes is now randomized, which for us better matches with what you would expect from such a deployment.
• Version updates, see below.

Fixes

• The local ~/.ssh/known_hosts file is no more erroneously clobbered.
• On macOS, the sudo password is no more required at every gbe converge invocation.

Components Versions

Component	New Version	Old Version
Cloud Foundry deployment	1.30.0	1.29.0
CF-MySQL	36.13.0	36.12.0
CF Containers Broker	1.0.3	1.0.1
Docker	32.0.0	31.0.1
Stemcells	3541.24 (2018-05-07)	3541.12 (2018-04-07)
Minio	2018-04-27T23-33-52Z	n/a

GBE v2.0.1

Security Fixes

This version addresses the MS-ISAC advisory number 2018-046, as advertised in PHP buildpack v4.3.53 release notes. You'll need to restage any PHP app you had staged with previous versions of the PHP buildpack.

Components Versions

Component	New Version	Old Version
Cloud Foundry deployment	1.29.0	1.28.0
CF-RabbitMQ	244.0.0 (RabbitMQ 3.7)	v243.0.0

GBE v2.0.0

This is a very good version of GBE.

With so many improvements compared to v1.0 !

New Features

• Introduce the concept of modular bricks called “subsys“ (or subsystems), defined by a spec.yml file and specifying features/ which are operations files. Subsystems types are: bosh-environment, cloud-config, runtime-config, bosh-deployment.

• Subsystems can now declare many input resources (which are git repositories, usually pinned to specific revisions), which is very convenient to leverage operations files that come from 3rd party Git repositories.

• Sybsystems declare imported_vars which are grabbed from other subsystems. This is very convenient to avoid copy/paste passwords from one deployment to another when no shared Bosh Link is available. Now stemcells versions are synchronized with this too.

• Imported credentials go to the subsystem's owndepl-creds.yml credentials file. No credentials are ever included in rendered manifests (which are meant to be Git-trackable, so they must be free of any secrets). We stick here to the convention that depl-manifest.yml files can be tracked in Git and depl-creds.yml not.

• Support for pre-deploy and post-deploy hook scripts has been added. Now service broker get registered to Cloud Foundry as an automated part of gbe converge all. This has proved being very convenient to develop new releases, where a pre-deploy hook rebuilds and uploads the release as part of the gbe converge command.

• Added more sysbsystems: logsearch is BOSH 2.0 deployment for an ELK cluster, rabbitmq is a resilient 2-nodes RabbitMQ cluster, traefik is a Træfik reverse-proxy.

• Added gbe cf helper to download the Cloud Foundry CLI easily

• Fixed gbe credhub

• Bump dnscontrol to version 0.2.3

• More consistent UX. Now there is only gbe left. Support for separate helper scripts like create-env or deploy has been dropped. Now Direnv is completely optional.

Components Versions

Component	New Version	Old Version
BOSH	265.2.0	264.4.0
Stemcells	3541.12 (2018-04-07)	3468.11 (2017-11-22)
Cloud Foundry deployment	1.28.0	1.0.0
CF-MySQL	36.12.0	36.9.0
Concourse	3.12.0	3.6.0
CF-RabbitMQ	243.0.0 (RabbitMQ 3.7)	n/a
Logsearch	209.0.0	n/a
Prometheus	22.0.2	20.0.0
SHIELD v7	7.0.4	n/a
SHIELD v8	8.0.8	n/a

GBE v1.0.0

This is an intermediate version of GBE, with a full-featured central gbe script. direnv is made optional and the various helper scripts that require it are still fully supported in this version.

Many, many improvements are shipped in this version.

• Features UAA and Credhub in the BOSH environment.
• Support for importing and exporting compiled releases to accelerate deployment time.
• Doesn't expose the BOSH server to the internet.
• Provides helpers to establish a compatible SOCKS5 proxy
• Provides helpers to converge an external DNS with dnscontrol
• Features more deployments: shield backups, prometheus monitoring
• Features more data-services: cassandra and Docker-based data services (Mysql 5.6, Postgres 9.6 and Redis 3.2)
• Use of BOSH DNS in the Cloud Foundry deployment.
• Updated versions: cf-deployment v1.0.0, Concourse v3.6.0, Bosh v264.4.0, CF-MySQL v36.9.0
• Features gbe sub-commands like gbe bbl, gbe terraform, gbe bosh and gbe dnscontrol to locally install the compatible versions of the external utilities that gbe requires.

GBE v0.5.0

This is an early version of GBE, based on the initial ideas developed for a simple BOSH 2.0 framework.

• Massively based on direnv.
• Allowing interaction with the BOSH environment (create-env and delete-env helper scripts) when in the base directory of the project.
• Enabling interaction with the BOSH director when in the deployments/ sub-directory.
• Allowing interaction of cloud-config, runtime-config, and deployments when in their respective sub-directories.