Skip to content

Commit 8a6531f

Browse files
Kaan Yaltipchilacmacknz
authored
Enhancement/6394 allow deb rpm to upgrade with endpoint tamper protection (#6907)
* Update pkg/testing/tools/tools.go Co-authored-by: Paolo Chilà <[email protected]> * enhancement(6394): updated preinstall script, updated service to use uninstall token * enhancmenet(6394): updated the preinstall script * enchancement(6394): started adding integraiton tests * enhancement(6394): updated fixture install, updated endpoint security tests * enhancement(6394): cleaned up fixture_install, added function that exposes fixture's uninstall tokens, updated tests * enhancement(6394): refactored test code so that I can use it with rpm * enhancement(6394): added tests to assert that tamper protection works * enhancement(6394): updated the endpoint testing tools, fixture install functions and the deb rpm upgrade tests * enhancement(6394): added test logs, updated rpm installation to set agent socket path * enhancement(6394): remove commented code * enhancement(6394): remove print statements * enhancement(6394): remove unnecessary comments, refactor unused function * enhancement(6394): revert var name change * enhancement(6394): added changelog * enchancement(6394): update test logs, add non integrative config to deb installation * enhancement(6394): updated the endpoint version comparison and assertion * enhancement(6394): added log in tests * enhancement(6394): resorted to using previous major instead of minor in upgrade test * enhancement(6394): updated endpoint version function in the tests, updated function name in testing tools * enhancement(6394): use previous minor, fix log * enhancement(6394): added comment explaining motive behind simple install functions * enhancement(6394): updated return in tools * Update changelog/fragments/1740166208-allow-deb-rpm-upgrade-with-tamper-protected-endpoint.yaml Co-authored-by: Craig MacKenzie <[email protected]> * enhancement(6394): fixed function call in tests * enhancement(6394): added systemctl start in postinstall, refactored preinstall and added condition to make same version installations work * enhancement(6394): updated the preinstall and postinstall scripts to troubleshoot * enhancement(6394): updated preinstall and postinstall script templates - Updated preinstall to stop endpoint if it is an available service regardless of the version of endpoint that's install - Updated postintall to start endpoint if the old endpoint version and the new version match. * enhancement(6394): removed error exit from postinstall * enhancement(6394): updated postinstall and preinstall templates - Preinstall now does not use a state file. Recovery from failure start ElasticEndpoint if it is not running - Preinstall does not stop endpoint if tamper protection is not enabled - Postinstall does not print an error if service is still running * enhancement(6394): removed debug logs * enhancement(6394): removed unnecessary comment * enhancement(6394): store uninstall token as local var, uninstall through the agent * enhancement(6394): added setclient function * enhancement(6394): added getInstallCommand and replaced SimpleInstall * enhancement(6394): added test case for error recovery. removed unused fixture functions * enhancement(6394): refactored tests, consolidated test scenarios into one function * enhancement(6394): remove unnecessary test functions * enhancement(6394): remove unused fixture function * enhancement(6394): revert unwanted installDeb changes * enhancement(6394): remove unwanted changes in testing tools * enhancement(6394): remove unused function call * enhancement(6394): replacing systemctl instead of adding new one to path * enhancement(6394): update real systemctl path in mock systemctl script * enhancement(6394): fix linting errors * Update changelog/fragments/1740166208-allow-deb-rpm-upgrade-with-tamper-protected-endpoint.yaml Co-authored-by: Paolo Chilà <[email protected]> * Update dev-tools/packaging/templates/linux/postinstall.sh.tmpl Co-authored-by: Paolo Chilà <[email protected]> * Update pkg/testing/tools/tools.go Co-authored-by: Paolo Chilà <[email protected]> * Update dev-tools/packaging/templates/linux/postinstall.sh.tmpl Co-authored-by: Paolo Chilà <[email protected]> * Update dev-tools/packaging/templates/linux/postinstall.sh.tmpl Co-authored-by: Paolo Chilà <[email protected]> * Update pkg/testing/tools/tools.go Co-authored-by: Paolo Chilà <[email protected]> * enhancement(6394): updated print statement * enhancement(6394): remove unnecessary command * enhancement(6394): use addressFromPath and SetClient * enhancement(6394): using service name, fixed indentation * test(debug): add detailed logging to Fixture.SetClient and installDeb for agent client setup debugging * Revert "test(debug): add detailed logging to Fixture.SetClient and installDeb for agent client setup debugging" This reverts commit 390c561. * enhancement(6394): renamed SetClient to SetDebRpmClient. Using hardcoded working dir as fixture working dir does not work for determining socket path * enhancement(6394): consolidated same version upgrade and regular upgrdade test functions * enhancement(6394): simplify preinstall script and enhance upgrade tests for tamper protection - Removed unnecessary endpoint handling logic from preinstall script. - Improved checks for service installation and status before upgrade. - Updated upgrade test functions to handle stopping the endpoint service before upgrades. * enhancement(6394): remove mock systemctl script for tamper protection tests * enhancement(6394): remove unused import * enhancement(6394): fixed order of execution in preinstall * enhancement(6394): added tests to make sure deb/rpm upgrades work when endpoint is not tamper protected --------- Co-authored-by: Paolo Chilà <[email protected]> Co-authored-by: Craig MacKenzie <[email protected]>
1 parent 6fd303e commit 8a6531f

File tree

7 files changed

+520
-43
lines changed

7 files changed

+520
-43
lines changed

‎changelog/fragments/1740166208-allow-deb-rpm-upgrade-with-tamper-protected-endpoint.yaml‎

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
# Kind can be one of:
2+
# - breaking-change: a change to previously-documented behavior
3+
# - deprecation: functionality that is being removed in a later release
4+
# - bug-fix: fixes a problem in a previous version
5+
# - enhancement: extends functionality but does not break or fix existing behavior
6+
# - feature: new functionality
7+
# - known-issue: problems that we are aware of in a given version
8+
# - security: impacts on the security of a product or a user’s deployment.
9+
# - upgrade: important information for someone upgrading from a prior version
10+
# - other: does not fit into any of the other categories
11+
kind: enhancement
12+
13+
# Change summary; a 80ish characters long description of the change.
14+
summary: Allow upgrading deb or rpm agents when using Elastic Defend with tamper protection.
15+
16+
# Long description; in case the summary is not enough to describe the change
17+
# this field accommodate a description without length limits.
18+
# NOTE: This field will be rendered only for breaking-change and known-issue kinds at the moment.
19+
#description:
20+
21+
# Affected component; usually one of "elastic-agent", "fleet-server", "filebeat", "metricbeat", "auditbeat", "all", etc.
22+
component: elastic-agent
23+
# PR URL; optional; the PR number that added the changeset.
24+
# If not present is automatically filled by the tooling finding the PR where this changelog fragment has been added.
25+
# NOTE: the tooling supports backports, so it's able to fill the original PR number instead of the backport PR number.
26+
# Please provide it if you are adding a fragment for a different PR.
27+
pr: https://github.com/elastic/elastic-agent/pull/6907
28+
# Issue URL; optional; the GitHub issue related to this changeset (either closes or is part of).
29+
# If not present is automatically filled by the tooling with the issue linked to the PR number.
30+
issue: https://github.com/elastic/elastic-agent/issues/6394

‎dev-tools/packaging/templates/linux/postinstall.sh.tmpl‎

Lines changed: 29 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,13 +6,42 @@ commit_hash="{{ commit_short }}"
66
version_dir="{{agent_package_version}}{{snapshot_suffix}}"
77
symlink="/usr/share/elastic-agent/bin/elastic-agent"
88
new_agent_dir="/var/lib/elastic-agent/data/elastic-agent-$version_dir-$commit_hash"
9+
new_endpoint_component_bin="$new_agent_dir/components/endpoint-security"
910

1011
# delete $symlink if exists
1112
if test -L "$symlink"; then
1213
echo "found symlink $symlink, unlink"
1314
unlink "$symlink"
1415
fi
1516

17+
SERVICE_NAME="ElasticEndpoint"
18+
19+
echo "Checking if $SERVICE_NAME is installed"
20+
if systemctl list-unit-files --type=service | grep -q "^${SERVICE_NAME}.service"; then
21+
22+
installed_endpoint_version="$(/opt/Elastic/Endpoint/elastic-endpoint version)"
23+
archive_endpoint_version="$("$new_endpoint_component_bin" version)"
24+
25+
echo "${SERVICE_NAME} is installed at version ${installed_endpoint_version}"
26+
if ! systemctl is-active --quiet "$SERVICE_NAME"; then
27+
echo "$SERVICE_NAME is installed but not running"
28+
if [[ "$installed_endpoint_version" == "$archive_endpoint_version" ]]; then
29+
echo "New endpoint and installed endpoint versions are the same: \"${installed_endpoint_version}\""
30+
echo "Starting $SERVICE_NAME"
31+
sudo systemctl start ${SERVICE_NAME}
32+
else
33+
echo "New endpoint version \"${archive_endpoint_version}\" is different than the one that's already
34+
installed \"${installed_endpoint_version}\""
35+
fi
36+
else
37+
# Endpoint can already be running if tamper protection is not enabled
38+
echo "$SERVICE_NAME is already running"
39+
fi
40+
else
41+
echo "$SERVICE_NAME is not installed"
42+
fi
43+
44+
1645
# create symlink to the new agent
1746
echo "create symlink "$symlink" to "$new_agent_dir/elastic-agent""
1847
ln -s "$new_agent_dir/elastic-agent" "$symlink"

‎dev-tools/packaging/templates/linux/preinstall.sh.tmpl‎

Lines changed: 47 additions & 21 deletions
Original file line numberDiff line numberDiff line change
@@ -2,6 +2,33 @@
22

33
set -e
44

5+
SERVICE_NAME="ElasticEndpoint"
6+
7+
# Check if the endpoint service is installed
8+
if systemctl list-unit-files --type=service | grep -q "^${SERVICE_NAME}.service"; then
9+
echo "$SERVICE_NAME is installed"
10+
11+
# Remove the vault directory if it exists
12+
if [ -d "/opt/Elastic/Endpoint/state/vault" ]; then
13+
14+
# Check if the endpoint is running
15+
if systemctl --quiet is-active $SERVICE_NAME; then
16+
echo "$SERVICE_NAME is running, stopping it"
17+
systemctl --quiet stop $SERVICE_NAME
18+
else
19+
echo "$SERVICE_NAME is not running"
20+
fi
21+
22+
echo "$SERVICE_NAME is tamper protected"
23+
echo "Removing $SERVICE_NAME vault"
24+
rm -rf "/opt/Elastic/Endpoint/state/vault"
25+
else
26+
echo "$SERVICE_NAME tamper protection is not enabled"
27+
fi
28+
else
29+
echo "$SERVICE_NAME is not installed"
30+
fi
31+
532
commit_hash="{{ commit_short }}"
633
version_dir="{{agent_package_version}}{{snapshot_suffix}}"
734
symlink="/usr/share/elastic-agent/bin/elastic-agent"
@@ -12,27 +39,27 @@ old_agent_dir=""
1239
# upon upgrade we migrate the current symlink to an upgrade symlink as the previous
1340
# installed version will remove the symlink
1441
if test -L "$symlink"; then
15-
resolved_symlink="$(readlink -- "$symlink")"
16-
if ! [ -z "$resolved_symlink" ]; then
17-
old_agent_dir="$( dirname "$resolved_symlink" )"
18-
echo "previous installation directory $old_agent_dir"
19-
else
20-
echo "unable to read existing symlink"
21-
fi
42+
resolved_symlink="$(readlink -- "$symlink")"
43+
if ! [ -z "$resolved_symlink" ]; then
44+
old_agent_dir="$(dirname "$resolved_symlink")"
45+
echo "previous installation directory $old_agent_dir"
46+
else
47+
echo "unable to read existing symlink"
48+
fi
2249

23-
# copy the state files if there was a previous agent install
24-
if ! [ -z "$old_agent_dir" ] && ! [ "$old_agent_dir" -ef "$new_agent_dir" ]; then
25-
yml_path="$old_agent_dir/state.yml"
26-
enc_path="$old_agent_dir/state.enc"
27-
echo "migrate state from $old_agent_dir to $new_agent_dir"
50+
# copy the state files if there was a previous agent install
51+
if ! [ -z "$old_agent_dir" ] && ! [ "$old_agent_dir" -ef "$new_agent_dir" ]; then
52+
yml_path="$old_agent_dir/state.yml"
53+
enc_path="$old_agent_dir/state.enc"
54+
echo "migrate state from $old_agent_dir to $new_agent_dir"
2855

29-
if test -f "$yml_path"; then
30-
echo "found "$yml_path", copy to "$new_agent_dir"."
31-
mkdir -p "$new_agent_dir"
32-
cp "$yml_path" "$new_agent_dir"
33-
else
34-
echo "didn't find $yml_path"
35-
fi
56+
if test -f "$yml_path"; then
57+
echo "found "$yml_path", copy to "$new_agent_dir"."
58+
mkdir -p "$new_agent_dir"
59+
cp "$yml_path" "$new_agent_dir"
60+
else
61+
echo "didn't find $yml_path"
62+
fi
3663

3764
if test -f "$enc_path"; then
3865
echo "found "$enc_path", copy to "$new_agent_dir"."
@@ -53,7 +80,7 @@ if test -L "$symlink"; then
5380
fi
5481
fi
5582
else
56-
echo "no previous installation found"
83+
echo "no previous installation found"
5784

5885
# create dir in case it does not exist
5986
mkdir -p "$new_agent_dir"
@@ -71,4 +98,3 @@ else
7198
fi
7299
fi
73100
fi
74-

‎pkg/testing/fixture.go‎

Lines changed: 2 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -362,7 +362,6 @@ func (f *Fixture) RunBeat(ctx context.Context) error {
362362
process.WithContext(ctx),
363363
process.WithArgs(args),
364364
process.WithCmdOptions(attachOutErr(stdOut, stdErr)))
365-
366365
if err != nil {
367366
return fmt.Errorf("failed to spawn %s: %w", f.binaryName, err)
368367
}
@@ -417,7 +416,8 @@ func RunProcess(t *testing.T,
417416
lp Logger,
418417
ctx context.Context, runLength time.Duration,
419418
logOutput, allowErrs bool,
420-
processPath string, args ...string) error {
419+
processPath string, args ...string,
420+
) error {
421421
if _, deadlineSet := ctx.Deadline(); !deadlineSet {
422422
t.Fatal("Context passed to RunProcess() has no deadline set.")
423423
}
@@ -435,7 +435,6 @@ func RunProcess(t *testing.T,
435435
process.WithContext(ctx),
436436
process.WithArgs(args),
437437
process.WithCmdOptions(attachOutErr(stdOut, stdErr)))
438-
439438
if err != nil {
440439
return fmt.Errorf("failed to spawn %q: %w", processPath, err)
441440
}
@@ -565,7 +564,6 @@ func (f *Fixture) executeWithClient(ctx context.Context, command string, disable
565564
process.WithContext(ctx),
566565
process.WithArgs(args),
567566
process.WithCmdOptions(attachOutErr(stdOut, stdErr)))
568-
569567
if err != nil {
570568
return fmt.Errorf("failed to spawn %s: %w", f.binaryName, err)
571569
}

‎pkg/testing/fixture_install.go‎

Lines changed: 38 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -26,12 +26,13 @@ import (
2626
"github.com/elastic/elastic-agent-libs/mapstr"
2727
agentsystemprocess "github.com/elastic/elastic-agent-system-metrics/metric/system/process"
2828
"github.com/elastic/elastic-agent/internal/pkg/agent/application/paths"
29+
"github.com/elastic/elastic-agent/pkg/control"
2930
"github.com/elastic/elastic-agent/pkg/control/v2/client"
3031
"github.com/elastic/elastic-agent/pkg/core/process"
3132
)
3233

3334
// ErrNotInstalled is returned in cases where Agent isn't installed
34-
var ErrNotInstalled = errors.New("Elastic Agent is not installed") //nolint:stylecheck // Elastic Agent is a proper noun
35+
var ErrNotInstalled = errors.New("Elastic Agent is not installed") //nolint:staticcheck // Elastic Agent is a proper noun
3536

3637
// CmdOpts creates vectors of command arguments for different agent commands
3738
type CmdOpts interface {
@@ -330,12 +331,12 @@ func (f *Fixture) installNoPkgManager(ctx context.Context, installOpts *InstallO
330331

331332
// environment variable AGENT_KEEP_INSTALLED=true will skip the uninstallation
332333
// useful to debug the issue with the Elastic Agent
333-
if f.t.Failed() && keepInstalledFlag() {
334+
if f.t.Failed() && KeepInstalledFlag() {
334335
f.t.Logf("skipping uninstall; test failed and AGENT_KEEP_INSTALLED=true")
335336
return
336337
}
337338

338-
if keepInstalledFlag() {
339+
if KeepInstalledFlag() {
339340
f.t.Logf("ignoring AGENT_KEEP_INSTALLED=true as test succeeded, " +
340341
"keeping the agent installed will jeopardise other tests")
341342
}
@@ -445,6 +446,19 @@ func getProcesses(t *gotesting.T, regex string) []runningProcess {
445446
return processes
446447
}
447448

449+
func (f *Fixture) SetDebRpmClient() error {
450+
workDir := "/var/lib/elastic-agent"
451+
socketPath, err := control.AddressFromPath(f.operatingSystem, workDir)
452+
if err != nil {
453+
return fmt.Errorf("failed to get control protcol address: %w", err)
454+
}
455+
456+
c := client.New(client.WithAddress(socketPath))
457+
f.setClient(c)
458+
459+
return nil
460+
}
461+
448462
// installDeb installs the prepared Elastic Agent binary from the deb
449463
// package and registers a t.Cleanup function to uninstall the agent if
450464
// it hasn't been uninstalled. It also takes care of collecting a
@@ -482,7 +496,7 @@ func (f *Fixture) installDeb(ctx context.Context, installOpts *InstallOpts, shou
482496
f.t.Logf("error systemctl stop elastic-agent: %s, output: %s", err, string(out))
483497
}
484498

485-
if keepInstalledFlag() {
499+
if KeepInstalledFlag() {
486500
f.t.Logf("skipping uninstall; test failed and AGENT_KEEP_INSTALLED=true")
487501
return
488502
}
@@ -502,6 +516,11 @@ func (f *Fixture) installDeb(ctx context.Context, installOpts *InstallOpts, shou
502516
return out, fmt.Errorf("systemctl start elastic-agent failed: %w", err)
503517
}
504518

519+
err = f.SetDebRpmClient()
520+
if err != nil {
521+
return nil, err
522+
}
523+
505524
if !shouldEnroll {
506525
return nil, nil
507526
}
@@ -520,11 +539,11 @@ func (f *Fixture) installDeb(ctx context.Context, installOpts *InstallOpts, shou
520539
if installOpts.DelayEnroll {
521540
enrollArgs = append(enrollArgs, "--delay-enroll")
522541
}
523-
if installOpts.EnrollOpts.URL != "" {
524-
enrollArgs = append(enrollArgs, "--url", installOpts.EnrollOpts.URL)
542+
if installOpts.URL != "" {
543+
enrollArgs = append(enrollArgs, "--url", installOpts.URL)
525544
}
526-
if installOpts.EnrollOpts.EnrollmentToken != "" {
527-
enrollArgs = append(enrollArgs, "--enrollment-token", installOpts.EnrollOpts.EnrollmentToken)
545+
if installOpts.EnrollmentToken != "" {
546+
enrollArgs = append(enrollArgs, "--enrollment-token", installOpts.EnrollmentToken)
528547
}
529548
out, err = exec.CommandContext(ctx, "sudo", enrollArgs...).CombinedOutput()
530549
if err != nil {
@@ -593,6 +612,11 @@ func (f *Fixture) installRpm(ctx context.Context, installOpts *InstallOpts, shou
593612
return out, fmt.Errorf("systemctl start elastic-agent failed: %w", err)
594613
}
595614

615+
err = f.SetDebRpmClient()
616+
if err != nil {
617+
return nil, err
618+
}
619+
596620
if !shouldEnroll {
597621
return nil, nil
598622
}
@@ -611,11 +635,11 @@ func (f *Fixture) installRpm(ctx context.Context, installOpts *InstallOpts, shou
611635
if installOpts.DelayEnroll {
612636
enrollArgs = append(enrollArgs, "--delay-enroll")
613637
}
614-
if installOpts.EnrollOpts.URL != "" {
615-
enrollArgs = append(enrollArgs, "--url", installOpts.EnrollOpts.URL)
638+
if installOpts.URL != "" {
639+
enrollArgs = append(enrollArgs, "--url", installOpts.URL)
616640
}
617-
if installOpts.EnrollOpts.EnrollmentToken != "" {
618-
enrollArgs = append(enrollArgs, "--enrollment-token", installOpts.EnrollOpts.EnrollmentToken)
641+
if installOpts.EnrollmentToken != "" {
642+
enrollArgs = append(enrollArgs, "--enrollment-token", installOpts.EnrollmentToken)
619643
}
620644
// run sudo elastic-agent enroll
621645
out, err = exec.CommandContext(ctx, "sudo", enrollArgs...).CombinedOutput()
@@ -722,7 +746,7 @@ func (f *Fixture) uninstallNoPkgManager(ctx context.Context, uninstallOpts *Unin
722746
}
723747

724748
if err != nil && topPathStats != nil {
725-
return out, fmt.Errorf("Elastic Agent is still installed at [%s]", topPath) //nolint:stylecheck // Elastic Agent is a proper noun
749+
return out, fmt.Errorf("Elastic Agent is still installed at [%s]", topPath) //nolint:staticcheck // Elastic Agent is a proper noun
726750
}
727751

728752
return out, nil
@@ -825,7 +849,7 @@ func collectDiagFlag() bool {
825849
return v
826850
}
827851

828-
func keepInstalledFlag() bool {
852+
func KeepInstalledFlag() bool {
829853
// failure reports false (ignore error)
830854
v, _ := strconv.ParseBool(os.Getenv("AGENT_KEEP_INSTALLED"))
831855
return v

‎pkg/testing/tools/tools.go‎

Lines changed: 16 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -36,6 +36,22 @@ func IsPolicyRevision(ctx context.Context, t *testing.T, client *kibana.Client,
3636
}
3737
}
3838

39+
func GetUninstallToken(ctx context.Context, kibClient *kibana.Client, policyId string) (string, error) {
40+
resp, err := kibClient.GetPolicyUninstallTokens(ctx, policyId)
41+
if err != nil {
42+
return "", fmt.Errorf("failed to fetch uninstall tokens: %w", err)
43+
}
44+
if len(resp.Items) == 0 {
45+
return "", fmt.Errorf("expected non-zero number of tokens")
46+
}
47+
48+
if len(resp.Items[0].Token) == 0 {
49+
return "", fmt.Errorf("expected non-empty token")
50+
}
51+
52+
return resp.Items[0].Token, nil
53+
}
54+
3955
// InstallAgentWithPolicy creates the given policy, enrolls the given agent
4056
// fixture in Fleet using the default Fleet Server, waits for the agent to be
4157
// online, and returns the created policy.
@@ -137,7 +153,6 @@ func InstallAgentForPolicyWithToken(ctx context.Context, t *testing.T,
137153

138154
installOpts.URL = fleetServerURL
139155
}
140-
141156
output, err := agentFixture.Install(ctx, &installOpts)
142157
if err != nil {
143158
t.Log(string(output))

0 commit comments

Comments
 (0)