Skip to content

Commit b872b1d

Browse files
codingWithJimmycodingWithJimmy
committed
Initial commit
Initial commit
1 parent d996b6c commit b872b1d

14 files changed

+558
-0
lines changed

‎README.md

Lines changed: 134 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,134 @@
1+
# Splunk Forwarder Repair Kit
2+
This kit was compilled based on common issues with Splunk deployments and managing idiosyncrasies that tend to naturally occur.
3+
4+
## Using the app
5+
Given the use-cases listed above, you will likely have multiple copies of the app with different input enabled for each. In any case, the app should restart Splunk when it is installed as all of the inputs are designed to be run when the forwarder starts.
6+
7+
It should be noted that if multiple copies of the app are created, the inputs.conf would need to be adjusted to account for the change in path for Windows Powershell scripts.
8+
9+
Below is the default inputs file. This configuration is responsible for running the scripts each time the forwarder restarts except for the restart script. The restart script is on a cron for every 2 minutes and is designed to only trigger a restart under specific circumstances.
10+
11+
```
12+
### Restart scripts
13+
[script://./bin/restart.sh]
14+
disabled = 1
15+
index = _internal
16+
sourcetype = restart:output
17+
interval = */2 * * * *
18+
source = restart_output
19+
20+
[powershell://restart]
21+
disabled = 1
22+
index = _internal
23+
sourcetype = restart:output
24+
interval = */2 * * * *
25+
source = restart_output
26+
script = . "$SplunkHome\etc\apps\SplunkForwarderRepairKit\bin\restart.ps1"
27+
28+
### GUID regneration scripts
29+
[script://./bin/regenGUID.sh]
30+
disabled = 1
31+
index = _internal
32+
sourcetype = regen_guid:output
33+
interval = -1
34+
source = regen_guid_output
35+
36+
[powershell://regenGUID]
37+
disabled = 1
38+
index = _internal
39+
sourcetype = regen_guid:output
40+
interval = -1
41+
source = regen_guid_output
42+
script = . "$SplunkHome\etc\apps\SplunkForwarderRepairKit\bin\regenGUID.ps1"
43+
44+
### Host/Server correction scripts
45+
[script://./bin/hostCorrect.sh]
46+
disabled = 1
47+
index = _internal
48+
sourcetype = host_rename:output
49+
interval = -1
50+
source = host_rename_output
51+
52+
[powershell://hostCorrect]
53+
disabled = 1
54+
index = _internal
55+
sourcetype = host_rename:output
56+
interval = -1
57+
source = host_rename_output
58+
script = . "$SplunkHome\etc\apps\SplunkForwarderRepairKit\bin\hostCorrect.ps1"
59+
60+
### Local deploymentclient removal scripts
61+
[script://./bin/dsRemove.sh]
62+
disabled = 1
63+
index = _internal
64+
sourcetype = ds_remove:output
65+
interval = -1
66+
source = ds_remove_output
67+
68+
[powershell://dsRemove]
69+
disabled = 1
70+
index = _internal
71+
sourcetype = ds_remove:output
72+
interval = -1
73+
source = ds_remove_output
74+
script = . "$SplunkHome\etc\apps\SplunkForwarderRepairKit\bin\dsRemove.ps1"
75+
76+
### Admin password change scripts
77+
[script://./bin/pwchange.sh]
78+
disabled = 1
79+
index = _internal
80+
sourcetype = pw_change:output
81+
interval = -1
82+
source = pw_change_output
83+
84+
[script://.\bin\pwchange.bat]
85+
disabled = 1
86+
index = _internal
87+
sourcetype = pw_change:output
88+
interval = -1
89+
source = pw_change_output
90+
```
91+
92+
## Use-Cases
93+
1. Local deployment server configurations
94+
2. Inputs and server host name configurations
95+
3. Duplicate forwarder GUIDs
96+
4. Changing the default password (Version <= 7.1.0)
97+
98+
###### Remove local deployment server configurations
99+
Early in a deployment of Splunk, local configurations could be used while getting fmailiar with how Splunk works. These configurations may last for a while and cause issues down the road like if a new deployment server is stood up or an IP address changes.
100+
101+
This app contains scripts for Windows and Linux forwarders that will remove local configurations of "deploymentclient.conf" in favor of a coniguration that has been deployed from the deployment server. This allows for that configuration to only be controlled via the deployment server from that point forward.
102+
103+
Windows - `dsRemove.ps1`
104+
\*Nix - `dsRemove.sh`
105+
106+
###### Correct inputs/server hostname configurations
107+
Many times we've come across an envionment where hundreds of forwarders are reporting with the same name and forwarder GUID. This usually happens when an image template isn't properly maintained after a fowarder has been embedded in it.
108+
109+
This app contains scripts for Windows and Linux forwarders that will determine if correction is necessary in the local "inputs.conf" and "server.conf" and correct them. The scripts are designed to only change what is needed and leave the rest of the files unchanged.
110+
111+
Windows - `hostCorrect.ps1`
112+
\*Nix - `hostCorrect.sh`
113+
114+
###### Regenerate forwarder GUID
115+
Another by-product of the previous use-case is forwarder GUIDs all being the same. While this doesn't affect how a forwarder performs its duties, unique GUIDs ensures if hosts have the same name they are still uniquely idenitifiable for troubleshooting purposes.
116+
117+
This app contains scripts for Windows and Linux forwarders that will move the existing "instance.cfg" to become a backup and restart the forwarder. Upon restarting, a new GUID will be generated.
118+
119+
Windows - `regenGUID.ps1`
120+
\*Nix - `regenGUID.sh`
121+
122+
###### Update default 'changeme' password on Splunk Forwarders (before 7.1.0)
123+
Forwarders deployed before version 7.1.0 didn't require the admin password be changed upon installation. Starting at 7.1.0, the forwarders required either a user-seed file or manual input of the password during first-time run. While the REST API of the fowrwarder is not configured to allow POST requests until the password is changed on versions prior to 7.1.0, changing the password is still recommended.
124+
125+
This app contains scripts for Windows and Linux forwarders that will allow either a static password or random password to be configured. By default, a random password is generated and printed into the log which is sent back to Splunk.
126+
127+
Windows - `pwchange.bat`
128+
\*Nix - `pwchange.sh`
129+
130+
## Restarting the Forwarder
131+
Because most of these use-cases require the forwarders be restarted, an additional script has been introduced that takes the outcome of each of the scripts used and determines if a restart is required. Each script is designed to create an empty file that the restart script uses to determine if a restart is necessary. If the restart script finds one of the files used to trigger a restart, it removes them and should keep the system clean of unnecessary files.
132+
133+
Windows - `restart.ps1`
134+
\*Nix - `restart.sh`

‎bin/dsRemove.bat

Lines changed: 31 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,31 @@
1+
@echo off
2+
3+
REM Define checkpoint varaible
4+
SET CHECKPOINT=%SPLUNK_HOME%\etc\ds_changed
5+
6+
REM Look for the checkpoint file and decide to exit or continue
7+
IF EXIST "%CHECKPOINT%" (
8+
goto NOCHANGE
9+
) ELSE (
10+
goto CHANGE
11+
)
12+
13+
REM Remove "deploymentclient.conf" from $SPLUNK_HOME\etc\system\local
14+
:CHANGE
15+
IF NOT EXIST "%CHECKPOINT%" (
16+
del /q "%SPLUNK_HOME%\etc\system\local\deploymentclient.conf"
17+
) ELSE (
18+
goto FAILED
19+
)
20+
21+
REM Create the checkpoint file and log success
22+
:SUCCESS
23+
echo %date% %time% %HOST%: Deploymentclient.conf removed from local system. > "%CHECKPOINT%"
24+
echo %date% %time% %HOST%: Deploymentclient.conf removed from local system.
25+
exit
26+
27+
REM Log that the deploymentclient.conf was already removed and exit
28+
:NOCHANGE
29+
echo %date% %time% %HOST%: Deploymentclient.conf removed from local system. > "%CHECKPOINT%"
30+
echo %date% %time% %HOST%: Deploymentclient.conf already removed from local system.
31+
exit

‎bin/dsRemove.ps1

Lines changed: 19 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,19 @@
1+
### Configure file paths for the system
2+
$LOCAL = {$(Test-Path "$SPLUNKHOME\etc\system\local\deploymentclient.conf")}
3+
$DEPLOYED = {$(Test-Path "$SPLUNKHOME\etc\apps\*\*\deploymentclient.conf")}
4+
$RESTART_CHECK = "$SPLUNKHOME\etc\restartds.txt"
5+
6+
### Filter to attach timestamps where necessary
7+
filter timestamp {"$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss.fff') ${env:COMPUTERNAME}: $_"}
8+
9+
### Check to see if there is a deploymentclient.conf file under $SPLUNK_HOME\etc\apps and bail out if there isn't
10+
if ($DEPLOYED -eq "False") {
11+
Write-output "No deploymentclient.conf detected in \$SPLUNK_HOME\etc\apps. Bailing out so the fowarder doesn't get orphaned." | timestamp
12+
} elseif ($LOCAL -eq "True") {
13+
### Remove the local "deploymentclient.conf" and flag
14+
Write-output "Removed deploymentclient.conf from local system." | timestamp
15+
Remove-Item -Path "$LOCAL"
16+
Out-File -FilePath "$RESTART_CHECK"
17+
} else {
18+
Write-output "No deploymentclient.conf correction necessary." | timestamp
19+
}

‎bin/dsRemove.sh

Lines changed: 18 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,18 @@
1+
#!/bin/bash
2+
### Look for a deploymentclient.conf file in the apps directory and define the path to the restartds.txt file
3+
DEPLOYED_APP=$(find $SPLUNK_HOME/etc/apps -type f -name deploymentclient.conf | wc -l)
4+
LOCAL=$(find $SPLUNK_HOME/etc/system/local -type f -name deploymentclient.conf | wc -l)
5+
RESTART_CHECK=$SPLUNK_HOME/etc/restartds.txt
6+
7+
### Check variables and take action accordingly
8+
if [ $DEPLOYED_APP = "0" ]; then
9+
echo $(date -R) $HOSTNAME: No deploymentclient.conf detected in \$SPLUNK_HOME/etc/apps. Bailing out so the fowarder doesn\'t get orphaned.
10+
exit
11+
elif [ $LOCAL = "1" ]; then
12+
# Remove the deploymentclient.conf from $SPLUNK_HOME/etc/system/local
13+
rm -f $SPLUNK_HOME/etc/system/local/deploymentclient.conf > /dev/null 2>&1
14+
echo $(date -R) $HOSTNAME: Removed deploymentclient.conf from local system.
15+
touch $RESTART_CHECK
16+
else
17+
echo $(date -R) $HOSTNAME: No deploymentclient.conf correction necessary.
18+
fi

‎bin/hostCorrect.ps1

Lines changed: 42 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,42 @@
1+
### Capture current values for forwarder and configure file path variables for the system
2+
$SPLUNK_LOCAL = "$SPLUNKHOME\etc\system\local"
3+
$currentHost = Select-String $SPLUNK_LOCAL\inputs.conf -pattern "host = ([^$]+)" | Foreach-Object {$_.Matches} | Foreach-Object {$_.Groups[1].Value}
4+
$currentServer = Select-String $SPLUNK_LOCAL\server.conf -pattern "serverName = ([^$]+)" | Foreach-Object {$_.Matches} | Foreach-Object {$_.Groups[1].Value}
5+
$restartInputCheck = "$SPLUNKHOME\etc\restartinput.txt"
6+
$restartServerCheck = "$SPLUNKHOME\etc\restartserver.txt"
7+
8+
### Filter to attach timestamps where necessary
9+
filter timestamp {"$(Get-Date -Format 'yyyy-MM-dd HH:mm:ss.fff') ${env:COMPUTERNAME}: $_"}
10+
11+
### Compare values to actual host value and flag accordingly
12+
if (-not ($currentHost -eq $env:COMPUTERNAME)){
13+
$correctHost="1"
14+
} else {
15+
$correctHost="0"
16+
}
17+
18+
if (-not ($currentServer -eq $env:COMPUTERNAME)){
19+
$correctServer="1"
20+
} else {
21+
$correctServer="0"
22+
}
23+
24+
### Check flags and take appropriate actions for host name
25+
if ($correctHost -eq "1") {
26+
Write-output "Currently configured host name: $currentHost. Reconfiguring inputs.conf..." | timestamp
27+
Copy-Item -Path "$SPLUNK_LOCAL\inputs.conf" -Destination "$SPLUNK_LOCAL\inputs_$(Get-Date -Format 'MMddyyyy').bak"
28+
(Get-Content -path $SPLUNK_LOCAL\inputs.conf -Raw) -replace $currentHost,$env:COMPUTERNAME | Set-Content $SPLUNK_LOCAL\inputs.conf
29+
Out-File -FilePath "$restartInputCheck"
30+
} else {
31+
Write-output "Currently configured host name: $currentHost. No correction necessary..." | timestamp
32+
}
33+
34+
### Check flags and take appropriate actions for server name
35+
if ($correctServer -eq "1") {
36+
Write-output "Currently configured server name: $currentServer. Reconfiguring server.conf..." | timestamp
37+
Copy-Item -Path "$SPLUNK_LOCAL\server.conf" -Destination "$SPLUNK_LOCAL\server_$(Get-Date -Format 'MMddyyyy').bak"
38+
(Get-Content -path $SPLUNK_LOCAL\server.conf -Raw) -replace $currentServer,$env:COMPUTERNAME | Set-Content $SPLUNK_LOCAL\server.conf
39+
Out-File -FilePath "$restartServerCheck"
40+
} else {
41+
Write-output "Currently configured server name: $currentServer. No correction necessary..." | timestamp
42+
}

‎bin/hostCorrect.sh

Lines changed: 24 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,24 @@
1+
#!/bin/bash
2+
### Capture the existing host and serverName values and define path to restart checkpoint file
3+
INPUTS_FILE=$SPLUNK_HOME/etc/system/local/inputs.conf
4+
SERVER_FILE=$SPLUNK_HOME/etc/system/local/server.conf
5+
CURRENT_HOST=$(cat $INPUTS_FILE | grep "host =" | awk '{printf $3}')
6+
CURRENT_SERVER=$(cat $SERVER_FILE | grep "serverName =" | awk '{printf $3}')
7+
RESTART_INPUT_CHECK="$SPLUNK_HOME/etc/restartinput.txt"
8+
RESTART_SERVER_CHECK="$SPLUNK_HOME/etc/restartserver.txt"
9+
10+
### Compare those values and correct if necessary
11+
if [ $CURRENT_HOST = $HOSTNAME ]; then
12+
echo "$(date +"%Y-%m-%d %H:%M:%S.%3N") ${HOSTNAME}: Currently configured host name: $CURRENT_HOST. No correction necessary..."
13+
else
14+
echo "$(date +"%Y-%m-%d %H:%M:%S.%3N") ${HOSTNAME}: Currently configured host name: $CURRENT_HOST. Reconfiguring inputs.conf..."
15+
cp $INPUTS_FILE $INPUTS_FILE.$(date +"%m%d%Y")
16+
touch $RESTART_INPUT_CHECK
17+
fi
18+
if [ $CURRENT_SERVER = $HOSTNAME ]; then
19+
echo "$(date +"%Y-%m-%d %H:%M:%S.%3N") ${HOSTNAME}: Currently configured server name: $CURRENT_SERVER. No correction necessary..."
20+
else
21+
echo "$(date +"%Y-%m-%d %H:%M:%S.%3N") ${HOSTNAME}: Currently configured server name: $CURRENT_SERVER. Reconfiguring inputs.conf..."
22+
cp $SERVER_FILE $SERVER_FILE.$(date +"%m%d%Y")
23+
touch $RESTART_SERVER_CHECK
24+
fi

‎bin/pwchange.bat

Lines changed: 78 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,78 @@
1+
@echo off
2+
REM Define the original and new passwords here. To use automatic password generation, change NEWPASS to 'auto'
3+
SET OLDPASS=changeme
4+
SET NEWPASS=auto
5+
6+
REM Settings for automatic password generation. Not used if NEWPASS is not set to 'auto'
7+
Setlocal EnableDelayedExpansion
8+
SET _RNDLength=16
9+
SET _Alphanumeric=ABCDEFGHIJKLMNOPQRSTUVWXYZabcdefghijklmnopqrstuvwxyz0123456789
10+
SET _Str=%_Alphanumeric%987654321
11+
12+
REM Other variables relating to the checkpoint file and the path to test login and hostname for logging
13+
SET CHECKPOINT=%SPLUNK_HOME%\etc\pwd_changed
14+
SET LOGIN_COMMAND="%SPLUNK_HOME%\bin\splunk.exe" login -auth admin:%OLDPASS%
15+
FOR /F "usebackq" %%i IN (`hostname`) DO SET HOST=%%i
16+
17+
REM Look for the checkpoint file and decide to error or continue
18+
IF EXIST "%CHECKPOINT%" (
19+
goto NOCHANGE
20+
) ELSE IF "%NEWPASS%"=="auto" (
21+
goto AUTOCHANGE
22+
) ELSE (
23+
goto CHANGE
24+
)
25+
26+
REM Attempt to login to local Splunk account. If successful, generate a new password and change it.
27+
:AUTOCHANGE
28+
FOR /F "tokens=2 usebackq" %%C in (`%LOGIN_COMMAND%`) DO SET LOGIN=%%C
29+
:_LenLoop
30+
IF NOT "%LOGIN%"=="Failed" (
31+
IF NOT "%_Str:~18%"=="" SET _Str=%_Str:~9%& SET /A _Len+=9& GOTO :_LenLoop
32+
SET _tmp=%_Str:~9,1%
33+
SET /A _Len=_Len+_tmp
34+
Set _count=0
35+
SET NEWPASS=
36+
:_loop
37+
Set /a _count+=1
38+
SET _RND=%Random%
39+
Set /A _RND=_RND%%%_Len%
40+
SET NEWPASS=!NEWPASS!!_Alphanumeric:~%_RND%,1!
41+
If !_count! lss %_RNDLength% goto _loop
42+
"%SPLUNK_HOME%\bin\splunk.exe" edit user admin -password "%NEWPASS%" >NUL
43+
goto AUTOSUCCESS
44+
) ELSE (
45+
goto FAILED
46+
)
47+
48+
REM Attempt to login to local Splunk account. If successful, generate a new password and change it.
49+
:CHANGE
50+
FOR /F "tokens=2 usebackq" %%C in (`%LOGIN_COMMAND%`) DO SET LOGIN=%%C
51+
IF NOT "%LOGIN%"=="Failed" (
52+
"%SPLUNK_HOME%\bin\splunk.exe" edit user admin -password "%NEWPASS%" >NUL
53+
goto SUCCESS
54+
) ELSE (
55+
goto FAILED
56+
)
57+
58+
REM Create the checkpoint file and log success. This will print the password in the log message passed back to Splunk.
59+
:AUTOSUCCESS
60+
echo %date% %time% %HOST%: Splunk account password successfully changed. > "%CHECKPOINT%"
61+
echo %date% %time% %HOST%: Splunk account password successfully changed. Automatic password: %NEWPASS%
62+
exit
63+
64+
REM Create the checkpoint file and log success.
65+
:SUCCESS
66+
echo %date% %time% %HOST%: Splunk account password successfully changed. > "%CHECKPOINT%"
67+
echo %date% %time% %HOST%: Splunk account password successfully changed.
68+
exit
69+
70+
REM Login failure
71+
:FAILED
72+
echo %date% %time% %HOST%: Splunk account login failed. Old password is not correct for this host.
73+
exit
74+
75+
REM Log that the checkpoint file exists
76+
:NOCHANGE
77+
echo %date% %time% %HOST%: Splunk account password was already changed.
78+
exit

‎bin/pwchange.sh

Lines changed: 30 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,30 @@
1+
#!/bin/bash
2+
# Define the original and new passwords here. To have a password automatically generated, set NEWPASS to 'auto'
3+
OLDPASS=changeme
4+
NEWPASS=auto
5+
6+
# Look for the checkpoint file and error out if it exists
7+
if [ -f $SPLUNK_HOME/etc/pwd_changed ]
8+
then
9+
echo $(date -R) $HOSTNAME: Splunk account password was already changed.
10+
exit
11+
fi
12+
13+
if [ "$NEWPASS" = "auto" ]
14+
then
15+
NEWPASS=$(head -c 500 /dev/urandom | sha256sum | base64 | head -c 16 ; echo)
16+
NEWPASSAUTO=$(echo Automatic password: $NEWPASS)
17+
fi
18+
19+
# Change the password
20+
$SPLUNK_HOME/bin/splunk edit user admin -password $NEWPASS -auth admin:$OLDPASS > /dev/null 2>&1
21+
22+
# Check splunkd.log for any error messages relating to login during the script and determine whether the change was successful or not
23+
CHANGED=$(tail -n 10 $SPLUNK_HOME/var/log/splunk/splunkd.log | grep pwchange | grep Login)
24+
if [ -z "$CHANGED" ]
25+
then
26+
echo $(date -R) $HOSTNAME: Splunk account password successfully changed. $NEWPASSAUTO
27+
echo $(date -R) $HOSTNAME: Splunk account password successfully changed. > $SPLUNK_HOME/etc/pwd_changed
28+
else
29+
echo $(date -R) $HOSTNAME: Splunk account login failed. Old password is not correct for this host.
30+
fi

0 commit comments

Comments
 (0)