Skip to content

Commit 500bf8f

Browse files
emaxerrnoemaxerrno
authored
Merge pull request #449 from rkruze/quickstart-guides
Documentation Updates
2 parents 76cb484 + 6996463 commit 500bf8f

File tree

7 files changed

+741
-0
lines changed

7 files changed

+741
-0
lines changed

‎docs/www/faq.md‎

Lines changed: 25 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,25 @@
1+
---
2+
title: Frequently Asked Questions
3+
order: 0
4+
---
5+
# Frequently Asked Questions
6+
7+
## Is Redpanda Fully Kafka API Compatible?
8+
9+
We support all parts of the Kafka API except for the transactions API. We are
10+
working on adding this shortly and you can find the issue in our public
11+
github here, [Support the Kafka Transactions API](https://github.com/vectorizedio/redpanda/issues/445).
12+
13+
If you run into any issues while working with a Kafka tool, please let us know! [File an issue](https://github.com/vectorizedio/redpanda/issues/new)
14+
15+
## Does Redpanda use Zookeeper?
16+
17+
No, Redpanda is a modern streaming platform that has been built using C++ and
18+
Raft for consensus. Since we use Raft we have no need for an external consensus
19+
system like Zookeeper.
20+
21+
## Can I run Redpanda directly on Windows or MacOS?
22+
23+
Unfortunately, you can only run Redpanda directly on Linux. However, you can
24+
use Docker to run Redpanda on any system supported by Docker. Please see our
25+
[Quick Start Docker Guide](quick-start-docker.md) for more information.

‎docs/www/production-deployment.md‎

Lines changed: 289 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,289 @@
1+
---
2+
title: Production Deployment
3+
order: 0
4+
---
5+
# Production Deployment
6+
7+
This guide will take you through what is needed to setup a production cluster
8+
of Redpanda.
9+
10+
### Requirements
11+
12+
For the best performance the following are strongly recommended:
13+
14+
- XFS for the data directory of Redpanda (/var/lib/redpanda/data)
15+
- A kernel that is at least 3.10.0-514, but a 4.18 or newer kernel is preferred
16+
- Local NVMe, RAID-0 when using multiple disks
17+
- 2GB of memory per core
18+
- The following ports need to be opened
19+
- `33145` - Internal RPC Port
20+
- `9092` - Kafka API Port
21+
- `9644` - Prometheus & HTTP Admin port
22+
23+
## Installation
24+
25+
First we need to provision the hardware needed. This can be done [manually](#manual-installation) or with the provided Terraform deployment files found here: [https://github.com/vectorizedio/deployment-automation].
26+
27+
First, install Terraform using the [Terraform Installation Tutorial](https://learn.hashicorp.com/tutorials/terraform/install-cli).
28+
Then clone the `deployment-automation` repo:
29+
30+
```
31+
git clone git@github.com:vectorizedio/deployment-automation.git
32+
```
33+
34+
Then change into the directory:
35+
36+
```
37+
cd deployment-automation
38+
```
39+
40+
From here you can follow the specific instructions for [AWS](#aws) or
41+
[GCP](#gcp)
42+
43+
### AWS
44+
45+
Within the `deployment-automation` folder change into the `aws` directory
46+
47+
```
48+
cd aws
49+
```
50+
51+
#### Step 1: Set AWS Credentials
52+
53+
First we need to set the AWS secret and key. Terraform provide multiple ways
54+
to set this which is covered here: [https://registry.terraform.io/providers/hashicorp/aws/latest/docs#environment-variables]
55+
56+
#### Step 2: Init Terraform
57+
58+
Before using Terraform to deploy the cluster we need to run the following:
59+
60+
```
61+
terraform init
62+
```
63+
64+
#### Step 3: Create the Cluster
65+
66+
We now can run `terraform apply` to create the cluster. There are a number of
67+
configuration options which can be specified when creating the cluster:
68+
69+
- `aws_region`: The AWS region to deploy the infrastructure on. Default: `us-west-2`.
70+
- `nodes`: The number of nodes to base the cluster on. Default: `1`.
71+
- `enable_monitoring`: Will create a prometheus/grafana instance to be used for monitoring the cluster. Default: `true`.
72+
- `instance_type`: The instance type which redpanda will be deployed on. Default: `i3.8xlarge`.
73+
- `prometheus_instance_type`: The instance type which prometheus and grafana will deployed on. Default: `c5.2xlarge`.
74+
- `public_key_path`: Provide the path to the public key of the keypair used to access the nodes. Default: `~/.ssh/id_rsa.pub`
75+
- `distro`: Linux distribution to install (this settings affects the below variables). Default: `ubuntu-focal`
76+
- `distro_ami`: AWS AMI to use for each available distribution.
77+
These have to be changed with according to the chosen AWS region.
78+
- `distro_ssh_user`: User used to ssh into the created EC2 instances.
79+
80+
The following is an example of creating a three node cluster using i3.large
81+
instances.
82+
83+
```
84+
terraform apply -var="instance_type=i3.large" -var="nodes 3"
85+
```
86+
87+
### GCP
88+
89+
Within the `deployment-automation` folder change into the `gcp` directory
90+
91+
```
92+
cd gcp
93+
```
94+
95+
#### Step 1: Prerequisites
96+
97+
You will need an existing subnet to deploy the VMs into. The subnet's attached
98+
firewall should allow inbound traffic on ports 22, 3000, 8888, 8889, 9090,
99+
9092, 9644 and 33145. This module adds the `rp-node` tag to the deployed VMs,
100+
which can be used as the target tag for the firewall rule.
101+
102+
#### Step 2: Init Terraform
103+
104+
Before using Terraform to deploy the cluster we need to run the following:
105+
106+
```
107+
terraform init
108+
```
109+
110+
#### Step 3: Create the Cluster
111+
112+
We now can run `terraform apply` to create the cluster. There are a number of
113+
configuration options which can be specified when creating the cluster:
114+
115+
- `region` (default: `us-west-1`): The region to deploy the infrastructure on.
116+
- `zone` (default: `a`): The region's zone to deploy the infrastructure on.
117+
- `subnet`: The name of an existing subnet to deploy the infrastructure on.
118+
- `nodes` (default: `1`): The number of nodes to base the cluster on. Keep in mind that one node is used as a monitoring node.
119+
- `disks` (default: `1`): The number of **local** disks to deploy on each machine
120+
- `image` (default: `ubuntu-os-cloud/ubuntu-1804-lts`): The OS image running on the VMs.
121+
- `machine_type` (default: `n2-standard-2`): The machine type.
122+
- `public_key_path`: Provide the path to the public key of the keypair used to access the nodes.
123+
- `ssh_user`: The ssh user. Must match the one in the public ssh key's comments.
124+
125+
The following is an example of creating a three node cluster using the subnet
126+
named `redpanda-cluster-subnet`
127+
128+
```
129+
terraform apply -var nodes=3 -var subnet=redpanda-cluster-subnet -var public_key_path=~/.ssh/id_rsa.pub -var ssh_user=$USER
130+
```
131+
132+
## Install Redpanda
133+
134+
Once the hardware has been provisioned. You can choose to either install
135+
Redpanda using our provided Ansible Playbooks or install Redpanda manually.
136+
137+
- [Installation with Ansible](#Installation-with-Ansible)
138+
- [Manual Installation](#Manual-Installation)
139+
140+
## Installation with Ansible
141+
142+
### Requirements
143+
144+
- Install Ansible - https://docs.ansible.com/ansible/latest/installation_guide/intro_installation.html
145+
146+
### Step 1: Clone the Github Repo
147+
148+
First we need to clone the repo:
149+
150+
```
151+
git clone git@github.com:vectorizedio/deployment-automation.git
152+
```
153+
154+
Then change into the directory:
155+
156+
```
157+
cd deployment-automation
158+
```
159+
160+
You will want install the required roles needed by Ansible with the following
161+
command:
162+
163+
```
164+
ansible-galaxy install -r ansible/requirements.yml
165+
```
166+
167+
### Step 2: Configure the hosts.ini file
168+
169+
In the deployment-automation directory you will find a file called `hosts.ini`.
170+
It will look something like:
171+
172+
```
173+
[redpanda]
174+
ip ansible_user=ssh_user ansible_become=True private_ip=pip id=0
175+
ip ansible_user=ssh_user ansible_become=True private_ip=pip id=1
176+
177+
[monitor]
178+
ip ansible_user=ssh_user ansible_become=True private_ip=pip id=1
179+
```
180+
181+
Under the `[redpanda]` section, you will want to replace the following:
182+
183+
- `ip` - the public ip address of the machine
184+
- `ansible_user` - the username for ansible to use to ssh to the machine
185+
- `private_ip` - the private ip address of the machine, could be the same as the public ip address
186+
- `id` - The node id of the Redpanda instance, this needs to be unique for each host
187+
188+
The `[monitor]` section is if you wish to have Prometheus and Grafana installed
189+
on a give host. If you wish to not have this deployed then remove the
190+
`[monitor]` section.
191+
192+
### Step 3: Run the Ansible Playbook
193+
194+
You can now setup Redpanda on your selected nodes by running the following
195+
command:
196+
197+
```
198+
ansible-playbook --private-key <your_private_key> -i hosts.ini -v ansible/playbooks/provision-node.yml
199+
```
200+
201+
Once this completes you will have a fully running cluster.
202+
203+
## Manual Installation
204+
205+
### Step 1: Install the Binary
206+
207+
On Fedora/RedHat Systems:
208+
209+
```
210+
curl -1sLf 'https://packages.vectorized.io/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.rpm.sh' | sudo -E bash && sudo yum install redpanda -y
211+
```
212+
213+
On Debian Systems:
214+
215+
```
216+
curl -1sLf 'https://packages.vectorized.io/nzc4ZYQK3WRGd9sy/redpanda/cfg/setup/bash.deb.sh' | sudo -E bash && sudo apt install redpanda -y
217+
```
218+
219+
### Step 2: Set Redpanda Production Mode
220+
221+
By default Redpanda is installed in **Development** mode, to we next need to
222+
set Redpanda to run in **Production** mode. This is done with:
223+
224+
```
225+
sudo rpk mode production
226+
```
227+
228+
We then need to tune the hardware, which can be done by running the following
229+
on each node:
230+
231+
```
232+
sudo rpk tune all
233+
```
234+
235+
> **_Optional: Benchmark your SSD_**
236+
>
237+
> On taller machines we recommend benchmarking your SSD. This can be done
238+
> with `rpk iotune`. You only need to run this once. For reference, a decent
239+
> local NVMe SSD should yield around 1GB/s sustained writes.
240+
> `rpk iotune` will capture SSD wear and tear and give accurate measurements
241+
> of what your hardware is actually capable of delivering. It is recommended
242+
> you run this before benchmarking.
243+
>
244+
>If you are on AWS, GCP or Azure, creating a new instance and upgrading to
245+
>an image with a recent Linux Kernel version is often the easiest way to
246+
>work around bad devices.
247+
>
248+
>```
249+
>sudo rpk iotune # takes 10mins
250+
>```
251+
252+
### Step 3: Configure and Start the Root Node
253+
254+
Now that the software is installed we need to configure it. The first step is
255+
to setup the root node. The root node will start as a standalone node, and
256+
every other one will join it, forming a cluster along the way.
257+
258+
For the root node we’ll choose 0 as its ID. --self tells the node which interface address to bind to. Usually you want that to be its private IP.
259+
260+
```
261+
sudo rpk config bootstrap --id 0 --self <ip> && \
262+
sudo systemctl start redpanda-tuner redpanda
263+
```
264+
265+
### Step 4: Configure and Start the Rest of the Nodes
266+
267+
For every other node, we just have to choose a unique integer id for it and let
268+
it know where to reach the root node.
269+
270+
```
271+
sudo rpk config bootstrap --id <unique id> \
272+
--self <private ip> \
273+
--ips <root node ip> && \
274+
sudo systemctl start redpanda-tuner redpanda
275+
```
276+
277+
### Step 5: Verify Installation
278+
279+
You can verify that the cluster is up and running by checking the logs:
280+
281+
```
282+
journalctl -u redpanda
283+
```
284+
285+
You should also be able to create a topic with the following command:
286+
287+
```
288+
rpk api topic create panda
289+
```

0 commit comments

Comments
 (0)