Skip to content
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/enterprise/embedded-manage-nodes.mdx
Original file line number Diff line number Diff line change
@@ -1,3 +1,5 @@
import HaArchitecture from "../partials/embedded-cluster/_multi-node-ha-arch.mdx"

# Managing Multi-Node Clusters with Embedded Cluster

The topic describes managing nodes in clusters created with Replicated Embedded Cluster, including how to add nodes and enable high-availability for multi-node clusters.
Expand Down Expand Up @@ -87,6 +89,12 @@ Multi-node clusters are not highly available by default. The first node of the c
High availability for Embedded Cluster in an Alpha feature. This feature is subject to change, including breaking changes. To get access to this feature, reach out to Alex Parker at [[email protected]](mailto:[email protected]).
:::

### HA Architecture

<HaArchitecture/>

For more information about the Embedded Cluster built-in extensions, see [Built-In Extensions](/vendor/embedded-overview#built-in-extensions) in _Embedded Cluster Overview_.

### Requirements

Enabling high availability has the following requirements:
Expand Down
9 changes: 9 additions & 0 deletions docs/partials/embedded-cluster/_multi-node-ha-arch.mdx
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
The following diagram shows the architecture of an HA multi-node Embedded Cluster installation:

![Embedded Cluster multi-node architecture](/images/embedded-architecture-multi-node-ha.png)

[View a larger version of this image](/images/embedded-architecture-multi-node-ha.png)

In HA multi-node installations, the Embedded Cluster Operator and the KOTS Admin Console are installed on the primary node only. All other built-in extenions, including the image registry for air gap installations, are installed on each node in the cluster. Any Helm [`extensions`](/reference/embedded-config#extensions) that you include in the Embedded Cluster Config can optionally be installed on one or more nodes.

Additionally, three replicas of rqlite are deployed for HA. Rqlite is used by KOTS to store information such as support bundles, version history, application metadata, and other small amounts of data needed to manage the application.
45 changes: 28 additions & 17 deletions docs/vendor/embedded-overview.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
import EmbeddedCluster from "../partials/embedded-cluster/_definition.mdx"
import Requirements from "../partials/embedded-cluster/_requirements.mdx"
import EmbeddedClusterPortRequirements from "../partials/embedded-cluster/_port-reqs.mdx"
import HaArchitecture from "../partials/embedded-cluster/_multi-node-ha-arch.mdx"

# Embedded Cluster Overview

Expand All @@ -14,7 +15,7 @@ If you are instead looking for information about creating Kubernetes Installers

<EmbeddedCluster/>

{/* [V2 ARCHITECTURE CONTENT] ## Architecture and Built-In Extensions
## Architecture

This section describes the Embedded Cluster architecture, including the built-in extensions deployed by Embedded Cluster.

Expand All @@ -28,41 +29,51 @@ The following diagram shows the architecture of a single-node Embedded Cluster i

As shown in the diagram above, the user downloads the Embedded Cluster installation assets as a `.tgz` in their installation environment. These installation assets include the Embedded Cluster binary, the user's license file, and (for air gap installations) an air gap bundle containing the images needed to install and run the release in an environment with limited or no outbound internet access.

When the user runs the Embedded Cluster install command, the Embedded Cluster binary first installs the k0s cluster as a systemd service. This systemd service is named using the slug of the application (for example, `gitea`).
When the user runs the Embedded Cluster install command, the Embedded Cluster binary first installs the k0s cluster as a systemd service.

After all the Kubernetes components for the cluster are available, the Embedded Cluster binary then installs the Embedded Cluster built-in extensions and any Helm extensions that were included in the [`extensions`](/reference/embedded-config#extensions) field of the Embedded Cluster Config. Each built-in extension is installed in its own namespace. The namespace or namespaces where Helm extensions are installed is defined by the vendor in the Embedded Cluster Config. */}
After all the Kubernetes components for the cluster are available, the Embedded Cluster binary then installs the Embedded Cluster built-in extensions. For more information about these extensions, see [Built-In Extensions](#built-in-extensions) below.

## Embedded Cluster Built-In Extensions
Any Helm extensions that were included in the [`extensions`](/reference/embedded-config#extensions) field of the Embedded Cluster Config are also installed. The namespace or namespaces where Helm extensions are installed is defined by the vendor in the Embedded Cluster Config.

Embedded Cluster deploys built-in extensions to provide capabilities such as application management and storage. These extensions are deployed by default and do not require configuration.
Finally, Embedded Cluster also installs the Local Artifact Mirror (LAM). In air gap installations, the LAM is used to store and update images.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^ not sure if this might imply that the LAM is only installed in air gap installations, but it looked kind of weird to explicitly state it's installed in both online and air gap without then having to say "it doesn't do anything in online installs right now"

I also thought it might not be that big of a deal to be vague about the online situation for the LAM since we do elsewhere say that the LAM port requirements apply to both online and air gap installs


The built-in extensions include:
### Multi-Node Architecture

* **KOTS:** Embedded Cluster installs the KOTS Admin Console in the kotsadm namespace. End customers use the Admin Console to configure and install the application. Rqlite is also installed in the kotsadm namespace alongside KOTS. Rqlite is a distributed relational database that uses SQLite as its storage engine. KOTS uses rqlite to store information such as support bundles, version history, application metadata, and other small amounts of data needed to manage the application. For more information about rqlite, see the [rqlite](https://rqlite.io/) website.
The following diagram shows the architecture of a multi-node Embedded Cluster installation:

* **OpenEBS:** Embedded Cluster uses OpenEBS to provide local PersistentVolume (PV) storage, including the PV storage for rqlite used by KOTS. For more information, see the [OpenEBS](https://openebs.io/docs/) documentation.
![Embedded Cluster multi-node architecture](/images/embedded-architecture-multi-node.png)

* **(Disaster Recovery Only) Velero:** If the installation uses the Embedded Cluster disaster recovery feature, Embedded Cluster installs Velero, which is an open-source tool that provides backup and restore functionality. For more information about Velero, see the [Velero](https://velero.io/docs/latest/) documentation. For more information about the disaster recovery feature, see [Disaster Recovery for Embedded Cluster (Alpha)](/vendor/embedded-disaster-recovery).
[View a larger version of this image](/images/embedded-architecture-multi-node.png)

* **(Air Gap Only) Image registry:** For air gap installations in environments with limited or no outbound internet access, Embedded Cluster installs an image registry where the images required to install and run the application are pushed. For more information about installing in air-gapped environments, see [Air Gap Installation with Embedded Cluster](/enterprise/installing-embedded-air-gap).
As shown in the diagram above, in multi-node installations, the Embedded Cluster Operator, KOTS, and the image registry for air gap installations are all installed on the primary node only.

{/* [V2 ARCHITECTURE CONTENT] Finally, after the built-in extensions and any Helm extensions are installed, The Embedded Cluster binary deploys a second systemd service on the node named `APP_SLUG-manager` (for example, `gitea-manager`). This manager service orchestrates Embedded Cluster and communicates with the KOTS instance running in the cluster through a websocket.
For installations that include disaster recovery with Velero, the Velero Node Agent runs on each node in the cluster. The Node Agent is a Kubernetes DaemonSet that performs backup and restore tasks such as creating snapshots and transferring data during restores.
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

any reason this isn't mentioned in the single-node stuff too?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it was because I wrote this with the purpose of explaining how the node agent is the component that runs on all the nodes in multi-node installs (versus single node installs where that's not relevant).

The basic info about the Velero extension though (ie, agnostic of install type) is over in the Built-In Extensions section: https://deploy-preview-3077--replicated-docs-upgrade.netlify.app/vendor/embedded-overview#built-in-extensions

I could update that section with this velero pod v agent info

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

i'm not particular about it. i assumed it was describing all that's in the diagram, but that's fine if not.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yeah the single node section used to list out all the extensions in the diagram, but I replaced that with "After all the Kubernetes components for the cluster are available, the Embedded Cluster binary then installs the Embedded Cluster built-in extensions. For more information about these extensions, see Built-In Extensions below." since these descriptions of the extensions were not really specific to single-node


### Multi-Node Architecture
Additionally, any Helm [`extensions`](/reference/embedded-config#extensions) that you include in the Embedded Cluster Config can optionally be installed on one or more nodes.

The following diagram shows the architecture of a multi-node Embedded Cluster installation for an application named `Gitea`:
### Multi-Node Architecture with High Availability

:::note
High availability (HA) for multi-node installations with Embedded Cluster is Alpha and is not enabled by default. For more informaiton about enabling HA, see [Enable High Availability for Multi-Node Clusters (Alpha)](/enterprise/embedded-manage-nodes#ha).
:::

![Embedded Cluster multi-node architecture](/images/embedded-architecture-multi-node.png)
<HaArchitecture/>

[View a larger version of this image](/images/embedded-architecture-multi-node.png)
## Built-In Extensions {#built-in-extensions}

As shown in the diagram above, in multi-node installations, an instance of the Embedded Cluster manager systemd service runs on each node and communicates with the KOTS instance running on the primary node through a websocket. This allows Embedded Cluster and KOTS to manage installations where workloads are running on multiple nodes in a cluster.
Embedded Cluster includes several built-in extensions. The built-in extensions provide capabilities such as application management and storage. Each built-in extension is installed in its own namespace.

Additionally, for installations that include disaster recovery with Velero, the Velero Node Agent also runs on each node in the cluster. The Node Agent is a Kubernetes DaemonSet that performs backup and restore tasks such as creating snapshots and transferring data during restores. */}
The built-in extensions installed by Embedded Cluster include:

* **Embedded Cluster Operator**: The Operator is used for reporting purposes as well as some clean up operations.
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

^ would the Operator be considered an extension? Seemed like it since it's installed alongside the others but wasn't sure


* **KOTS:** Embedded Cluster installs the KOTS Admin Console in the kotsadm namespace. End customers use the Admin Console to configure and install the application. Rqlite is also installed in the kotsadm namespace alongside KOTS. Rqlite is a distributed relational database that uses SQLite as its storage engine. KOTS uses rqlite to store information such as support bundles, version history, application metadata, and other small amounts of data needed to manage the application. For more information about rqlite, see the [rqlite](https://rqlite.io/) website.

* **OpenEBS:** Embedded Cluster uses OpenEBS to provide local PersistentVolume (PV) storage, including the PV storage for rqlite used by KOTS. For more information, see the [OpenEBS](https://openebs.io/docs/) documentation.

* **(Disaster Recovery Only) Velero:** If the installation uses the Embedded Cluster disaster recovery feature, Embedded Cluster installs Velero, which is an open-source tool that provides backup and restore functionality. For more information about Velero, see the [Velero](https://velero.io/docs/latest/) documentation. For more information about the disaster recovery feature, see [Disaster Recovery for Embedded Cluster (Alpha)](/vendor/embedded-disaster-recovery).

* **(Air Gap Only) Image registry:** For air gap installations in environments with limited or no outbound internet access, Embedded Cluster installs an image registry where the images required to install and run the application are pushed. For more information about installing in air-gapped environments, see [Air Gap Installation with Embedded Cluster](/enterprise/installing-embedded-air-gap).

## Comparison to kURL

Expand Down
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

updated so that image registry is running on 2 nodes instead of 3

Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified static/images/embedded-architecture-multi-node.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
Binary file modified static/images/embedded-architecture-single-node.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.