Skip to content

Commit 261711f

Browse files
nrios14nrios14
authored
Create operations-manual-tcie-2.x.md
1 parent dfe8509 commit 261711f

File tree

1 file changed

+379
-0
lines changed

1 file changed

+379
-0
lines changed

‎user/enterprise/operations-manual-tcie-2.x.md

Lines changed: 379 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,379 @@
1+
---
2+
title: TCIE 2.x Operations Manual
3+
layout: en_enterprise
4+
5+
---
6+
7+
## User Management Commands in TCIE 2.x
8+
9+
Travis CI Enterprise (TCIE) version 2.x has the following user management options.
10+
11+
These commands are run via the command line on the platform instance.
12+
13+
### User Information
14+
15+
List basic information about users and their status.
16+
17+
`travis users` - list every user.
18+
19+
`travis users --active` - list every active user.
20+
21+
`travis users --suspended` - list every suspended user.
22+
23+
### Suspend and Unsuspend Users in TCIE 2.x
24+
25+
Suspend or unsuspend users. Builds triggered by suspended users are blocked by `travis-gatekeeper`.
26+
27+
`travis suspend <login>` - suspend a user where `<login>` is the user's GitHub login.
28+
29+
`travis unsuspend <login>` - unsuspend a user where `<login>` is the user's GitHub login.
30+
31+
**Please note**: Using the `suspend` command does not restrict access to the Enterprise platform.
32+
It removes a seat restriction for an archived user. If a *suspended* user logs into the platform, the seat restriction will be imposed again.
33+
34+
### Syncing a User in TCIE 2.x
35+
36+
To sync a user (not technically part of User Management, but a related task):
37+
38+
To sync one user: `travis sync_users --logins=<login>`
39+
40+
To sync all users: `travis sync_users`.
41+
42+
## Travis CI Enterprise 2.x SSL Certificate Management
43+
44+
This section contains information about SSL certificates and their use in Travis CI Enterprise (TCIE).
45+
46+
By default, Travis CI Enterprise verifies SSL for all traffic between internal platform components, RabbitMQ, and, e.g., Github.com/Github Enterprise. SSL verification helps to ensure that traffic is delivered securely.
47+
48+
> It is **strongly recommended** that you configure your installation with a valid certificate from a trusted authority.
49+
50+
### Use a trial license in TCIE 2.x
51+
52+
When prompted to provide a custom SSL certificate during installation, you can select the 'Use Self-Signed Cert' option.
53+
54+
### Disable SSL Verification in TCIE 2.x
55+
56+
To disable SSL verification in Travis CI Enterprise 2.x:
57+
58+
1. Log into your dashboard at `https://<your-travis-ci-enterprise-domain>:8800`.
59+
2. Click on 'Settings' in the topmost menu.
60+
3. Click 'Advanced Settings' in the left menu (or scroll down).
61+
4. Check the box for 'Disable SSL cert verification'.
62+
5. Click 'Save' at the bottom of the page.
63+
64+
> Please note that this will require a restart of Travis CI Enterprise to take effect and may cause brief downtime.
65+
66+
This will remove SSL verification checks with internal traffic between Travis CI components. This will also cause all repository hooks created on GitHub to turn off SSL verification for traffic coming to Travis CI.
67+
68+
> Please note that this setting will only apply to hooks created _after_ the setting was enabled.
69+
70+
### Add or Update Certificates in TCIE 2.x
71+
72+
To upload your certificates:
73+
74+
1. Log into your dashboard at `https://<your-travis-ci-enterprise-domain>:8800`.
75+
2. Click the gear icon on the rightmost side of the top menu and select 'Console Settings' from the dropdown.
76+
3. Click 'TLS Key & Cert' in the left menu (or scroll down).
77+
4. Select the appropriate option depending on the location of your certificate files.
78+
5. Enter the paths or select the files to upload.
79+
6. Click 'Save' at the bottom of the page.
80+
81+
> Please note that this will require a restart of Travis CI Enterprise to take effect and may cause brief downtime.
82+
83+
> A valid x509 certificate and private key files are required. The certificate and key must be in PEM format. The key must be unencrypted.
84+
85+
### Intermediate Certificate Chains in TCIE 2.x
86+
87+
To add your certificate chain:
88+
89+
1. Log into your dashboard at `https://<your-travis-ci-enterprise-domain>:8800`.
90+
2. Click on 'Settings' in the topmost menu.
91+
3. Click 'Advanced Settings' in the left menu (or scroll down).
92+
4. Copy and paste your certificate chain into the text box labeled 'Custom Certificate Authority (CA) Bundle'.
93+
5. Click 'Save' at the bottom of the page.
94+
95+
> Please note that this will require a restart of Travis CI Enterprise to take effect and may cause a brief downtime.
96+
97+
Upon restart, you can verify whether SSL verification is working. If you are still experiencing trouble after setting the chain, please contact us at [[email protected]](mailto:[email protected]) for assistance.
98+
99+
100+
### How to use a Let's Encrypt SSL Certificate in TCIE 2.x
101+
102+
You can use a certificate from [Let's Encrypt](https://letsencrypt.org/) instead of a self-signed certificate or a certificate purchased from a trusted certificate authority. Certificates from Let's Encrypt are free and behave the same as those purchased from a trusted certificate authority.
103+
104+
What you will need:
105+
106+
- An email address (Let's Encrypt will send notifications regarding urgent renewal and security issues).
107+
- A domain name under which your installation is available.
108+
109+
#### Install certbot in TCIE 2.x
110+
111+
We will be using [certbot](https://certbot.eff.org/#ubuntutrusty-other) to obtain an SSL certificate from Let's Encrypt. To install certbot:
112+
113+
1. Open an SSH connection to the platform machine.
114+
2. Add the certbot personal package archive:
115+
```bash
116+
sudo add-apt-repository ppa:certbot/certbot
117+
```
118+
3. Update available packages:
119+
```bash
120+
sudo apt-get update
121+
```
122+
3. Install required certbot dependency:
123+
```bash
124+
sudo apt-get install software-properties-common
125+
```
126+
5. Install the certbot package:
127+
```bash
128+
sudo apt-get install certbot
129+
```
130+
131+
#### Generate a new Let's Encrypt certificate in TCIE 2.x
132+
133+
The certbot tool offers multiple ways to obtain a certificate. We'll pick the 'temporary webserver' option since it requires no additional setup or configuration. The only prerequisite is that the Travis CI Enterprise container must be stopped so that the certbot webserver can bind properly to port 443.
134+
135+
> Your Travis CI Enterprise instance will be unavailable during this process.
136+
137+
Follow these steps to generate a new Let's Encrypt certificate.
138+
139+
1. Open an SSH connection to the platform machine.
140+
2. Stop your Travis CI Enterprise:
141+
```bash
142+
replicatedctl app stop
143+
```
144+
3. Start the interactive certificate process:
145+
```bash
146+
sudo certbot certonly
147+
```
148+
4. Add the prompt, select option `1` to `Spin up a temporary webserver`:
149+
```bash
150+
How would you like to authenticate with the ACME CA?
151+
-------------------------------------------------------------------------------
152+
1: Spin up a temporary webserver (standalone)
153+
2: Place files in webroot directory (webroot)
154+
-------------------------------------------------------------------------------
155+
Select the appropriate number [1-2], then [enter] (press 'c' to cancel): 1
156+
```
157+
5. Fill in your email address:
158+
```bash
159+
Enter email address (used for urgent renewal and security notices) (Enter 'c' to
160+
cancel): [email protected]
161+
```
162+
6. Accept the Terms of Services:
163+
```bash
164+
-------------------------------------------------------------------------------
165+
Please read the Terms of Service at
166+
https://letsencrypt.org/documents/LE-SA-v1.1.1-August-1-2016.pdf. You must agree
167+
to register with the ACME server at
168+
https://acme-v01.api.letsencrypt.org/directory
169+
-------------------------------------------------------------------------------
170+
(A)gree/(C)ancel: A
171+
```
172+
7. Decide if you'd like to share your email address with the EFF (this decision has no effect on your certificate or your Travis CI Enterprise instance):
173+
```bash
174+
-------------------------------------------------------------------------------
175+
Would you be willing to share your email address with the Electronic Frontier
176+
Foundation, a founding partner of the Let's Encrypt project and the non-profit
177+
organization that develops Certbot? We'd like to send you an email about EFF and
178+
our work to encrypt the web, protect its users and defend digital rights.
179+
-------------------------------------------------------------------------------
180+
(Y)es/(N)o: N
181+
```
182+
8. Provide your domain name:
183+
184+
```bash
185+
Please enter your domain name(s) (comma and/or space separated) (Enter 'c'
186+
to cancel): <your-travis-ci-enterprise-domain>
187+
```
188+
9. Upon successful completion, you should see a message similar to the following:
189+
190+
```bash
191+
IMPORTANT NOTES:
192+
Congratulations! Your certificate and chain have been saved at:
193+
/etc/letsencrypt/live/<your-travis-ci-enterprise-domain>/fullchain.pem
194+
Your key file has been saved at:
195+
/etc/letsencrypt/live/<your-travis-ci-enterprise-domain>/privkey.pem
196+
Your cert will expire on 2018-02-07. To obtain a new or tweaked
197+
version of this certificate in the future, simply run certbot
198+
again. To non-interactively renew *all* of your certificates, run
199+
"certbot renew"
200+
```
201+
202+
10. Restart your Travis CI Enterprise instance:
203+
204+
```bash
205+
replicatedctl app start
206+
```
207+
208+
Your certificate is now generated and saved on your Travis CI Enterprise platform machine. However, you must take additional steps to configure your Travis CI Enterprise instance to use the new certificate. For instructions, please see the [Using a Let's Encrypt certificate](#using-a-lets-encrypt-certificate) section.
209+
210+
#### Let's Encrypt certificate in TCIE 2.x
211+
212+
> Your Travis CI Enterprise instance will be unavailable during this process.
213+
214+
To use your generated certificate in your Travis CI Enterprise 2.x instance:
215+
216+
1. Log into your dashboard at `https://<your-travis-ci-enterprise-domain>:8800`.
217+
2. On the rightmost side of the top menu, click the gear icon and select 'Console Settings' from the dropdown.
218+
3. Click 'TLS Key & Cert' in the left menu (or scroll down).
219+
4. Select the 'Server path' option.
220+
5. Enter the paths to your certificate provided in the certbot output above. Example:
221+
```
222+
- SSL Private Key Filename: `/etc/letsencrypt/live/<your-travis-ci-enterprise-domain>/privkey.pem`
223+
- SSL Certificate Filename: `/etc/letsencrypt/live/<your-travis-ci-enterprise-domain>/fullchain.pem`
224+
```
225+
6. Click 'Save' at the bottom of the page.
226+
7. Open an SSH connection to the platform machine.
227+
8. Stop your Travis CI Enterprise instance:
228+
```bash
229+
replicatedctl app stop
230+
```
231+
9. Restart your Travis CI Enterprise instance:
232+
```bash
233+
replicatedctl app start
234+
```
235+
236+
### Renew the Let's Encrypt Certificate for TCIE 2.x
237+
238+
> Your Travis CI Enterprise instance will be unavailable during this process.
239+
240+
To manually renew your certificate for another 90 days:
241+
242+
1. Open an SSH connection to the platform machine.
243+
2. Stop your Travis CI Enterprise instance:
244+
```bash
245+
replicatedctl app stop
246+
```
247+
3. Perform the certificate renewal:
248+
```bash
249+
sudo certbot renew
250+
```
251+
4. Restart your Travis CI Enterprise instance:
252+
```bash
253+
replicatedctl app start
254+
```
255+
256+
### Use Cron Jobs to Automate the renewal of the Let's Encrypt SSL Certificate in TCIE2.x
257+
258+
You can avoid manual renewals by using a cron job to automate the certbot renewal:
259+
260+
1. Create `/home/ubuntu/renew-certs.sh` containing the following script:
261+
262+
```bash
263+
#!/bin/bash
264+
265+
set -u
266+
267+
function usage() {
268+
echo
269+
echo "Usage: $(basename $0)"
270+
echo
271+
echo "Simple script to renew certs, should be used via a cron. Assumes that certs are already set up."
272+
echo
273+
echo "See https://docs.travis-ci.com/user/enterprise/platform-tips/#use-a-lets-encrypt-ssl-certificate for initial setup info."
274+
echo
275+
276+
exit 1
277+
}
278+
279+
if [[ $# -ne 0 ]]; then
280+
usage
281+
fi
282+
283+
replicatedctl app stop
284+
sudo certbot renew
285+
replicatedctl app start
286+
```
287+
{: data-file="/home/ubuntu/renew-certs.sh"}
288+
289+
2. Make the script executable:
290+
291+
```sh
292+
$ chmod +x /home/ubuntu/renew-certs.sh
293+
```
294+
295+
3. Create a cronjob by editing `/etc/crontab` and appending the following:
296+
297+
```sh
298+
# Renews certs at 2 am on the 1st of February, May, August, and November.
299+
# Please change the configuration that applies to the certificate you are creating.
300+
0 2 1 2,5,8,11 * /home/ubuntu/renew-certs.sh
301+
```
302+
{: data-file="/etc/crontab"}
303+
304+
> This process will introduce a small downtime while the certificates are renewed. We recommend that you communicate with your users before each renewal so they know that their builds will temporarily be stopped until the certificate is renewed.
305+
306+
## Travis CI Enterprise 2.x Troubleshooting
307+
308+
This section provides guidelines and suggestions for troubleshooting your Travis CI Enterprise installation.
309+
310+
Throughout this document, we'll be using the following terms to refer to the two components of your Travis CI Enterprise 2.x installation:
311+
312+
- `Platform machine`: The instance that runs the main Travis components, including the web frontend.
313+
- `Worker machine`: The worker instance(s) that process and run the builds.
314+
315+
> This guide is geared towards non-High Availability (HA) setups. Please contact us at [[email protected]](mailto:[email protected]) if you require support for your HA setup.
316+
317+
### Issue when running Enterprise v2.2 or higher
318+
319+
By default, the Enterprise Platform v2.2 or higher will attempt to route builds to the `builds.trusty` queue. This could lead to build issues if you are not running a Trusty worker to process those builds or targeting a different distribution (e.g., `xenial`).
320+
321+
To address this, either:
322+
323+
- Ensure that you have installed a Trusty worker on a new virtual machine instance: [Trusty installation guide](/user/enterprise/trusty/)
324+
- Override the default queuing behavior to specify a new queue. To override the default queue, you must access the Admin Dashboard at `https://<your-travis-ci-enterprise-domain>:8800/settings#override_default_dist_enable` and toggle the 'Override Default Build Environment' button. This will allow you to specify the new default based on your needs and the workers that you have available.
325+
326+
### Enterprise Container Fails to Start due to 'context deadline exceeded' Error
327+
328+
After a fresh installation or configuration change, the Enterprise container doesn't start, and the following error is visible in the Admin Dashboard found at `https://<your-travis-ci-enterprise-domain>:8800/dashboard`:
329+
330+
```
331+
Ready state command canceled: context deadline exceeded
332+
```
333+
334+
### User Accounts Stuck in Syncing State
335+
336+
One or more user accounts are stuck in the `is_syncing = true` state. When you query the database, the number of currently syncing users does not decrease over time. Example:
337+
338+
```sql
339+
travis_production=> select count(*) from users where is_syncing=true;
340+
count
341+
-------
342+
1027
343+
(1 row)
344+
```
345+
#### Workaround
346+
347+
**TCIE 2.x**: Log into the platform machine via SSH. Run `$ travis console`
348+
349+
Next, run:
350+
351+
```bash
352+
>> User.where(is_syncing: true).count
353+
>> ActiveRecord::Base.connection.execute('set statement_timeout to 60000')
354+
>> User.update_all(is_syncing: false)
355+
```
356+
357+
It can happen that organizations are also stuck in the syncing state. Since an organization does not have an `is_syncing` flag, all users that belong to it must be successfully synced.
358+
359+
### Logs contain GitHub API 422 errors
360+
361+
On every commit made when a build runs, a commit status is created for a given SHA. Due to GitHub’s limitations at 1,000 statuses per SHA and context within a repository, if more than 1,000 statuses are created, this leads to a validation error.
362+
This issue should no longer be present in GitHub Apps integrations but will be present in Webhooks integrations.
363+
364+
#### Workaround
365+
366+
The workaround for this issue is to manually re-sync the user account with GitHub. This will generate a fresh token for the user account that has not reached any GitHub API limits.
367+
368+
An administrator can also initiate a sync on behalf of someone else. Follow these steps to sync the account from the CLI with administrator privileges.
369+
370+
**TCIE 2.x**: via the `travis` CLI tool on the platform machine:
371+
372+
> If `—logins=<GITHUB-LOGIN>` is not provided, then this command will trigger a sync on every user. This could result in long runtimes and may impact production operations if you have many total users on your Travis CI Enterprise instance.
373+
374+
1. Open an SSH connection to the platform machine.
375+
2. Initiate a sync by running `travis sync_users —logins=<GITHUB-LOGIN>`
376+
377+
## Contact Enterprise Support
378+
379+
{{ site.data.snippets.contact_enterprise_support }}

0 commit comments

Comments
 (0)