Note: Please ignore red font used on tickets 1-48. Some information has been redacted for security purposes.
TASK: The Infrastructure Team is requesting a new CentOS 9 Virtual Machine (VM) to launch a new product for the software development team. Please use the naming convention in the requirements section below. Ensure VM details are added to the asset tiger inventory tool.
STEPS TAKEN:
-
Accessed vSphere and selected the correct sandbox correlated to the Host IP address:
-
Clicked on "Actions" and selected "New Virtual Machine"
-
Selected "Create a new virtual machine"
-
Created VM according to the "Requirements;" however, CentOS 9 was not listed as an option, so I chose CentOS 8 for the guest OS
-
Then later I was able to select the ISO image as listed in the "Requirements"
-
Once it was created, I powered it on and clicked "Launch Web Console"
-
I booted the OS normally and went through the steps to perform a minimal installation:
-
I was able to successfully login to the VM
-
I ran
ip ato obtain the IP address and MAC address (link)
-
I ran
dmidecode -t system | grep -i serialto obtain the serial number
TASK: To efficiently manage and allocate organization's resources, we need to create resource clusters for our deployed and upcoming VMs.
STEPS TAKEN:
-
Followed the instruction listed on the Wiki on "How to create a resource pool in Vsphere"
-
Opened vSphere Client and highlighted Data Center location
-
Right-click on the Data Center and choose 'New Resource Pool'
-
Named my resource pool as JLEWIS-CLUSTER and clicked 'OK'
-
Next, I selected the VM I created for Ticket 1 and right-clicked on it to select 'Migrate'
-
Selected the first option: 'Change compute resource only', and clicked 'Next'
-
Selected 'Resource Pools' tab, chose my resource pool, and clicked 'Next'
-
Selected YT-Intran-VLAN and clicked 'Next' until I was able to click 'Finish'
TASK: Now that the Resource Pool has been successfully created, please proceed with moving your server/s into it.
STEPS TAKEN:
-
I selected the VM I created for Ticket 1 and right-clicked on it to select 'Migrate'
-
Selected the first option: 'Change compute resource only', and clicked 'Next'
-
Selected 'Resource Pools' tab, chose my resource pool, and clicked 'Next'
-
Selected YT-Intran-VLAN and clicked 'Next' until I was able to click 'Finish'
TASK: Please refer to the IP Address Management (IPAM sheet) to locate the necessary network information. Use this information to establish a static connection for your dev-app server.
STEPS TAKEN:
-
Used the following command to add a static connection:
nmcli c a con-name dev-app-static autoconnect yes ifname ens192 type ethernet ip4 <ipaddress> gw4 <gatewayaddress> ipv4.dns <dnsaddress> -
Added the user procore to wheel group and assigned password.
TASK: Proceed with the installation and configuration of the IPA client on your new virtual machine (VM).
STEPS TAKEN:
-
Followed the steps listed for configuring an IPA-Client
-
Ran the following to install package:
dnf install ipa-client -y -
Ran the following to install IPA Client:
ipa-client-install --mkhomedirprocore.devipa.procore.dev
-
Entered username and password to enroll IPA Client and received the following confirmation:
-
Used the
id jlewiscommand to confirm if IPA was working correctly:
-
Used the
ipa user-show jlewiscommand to confirm the following:
-
Note: While configuring IPA if you get an error message, try:
ipa-client-install --mkhomedir --force-joinor usekinit jlewisto use Kerberos (security software used w/IPA to ensure connection is genuine) in order to confirm your access and identity.
TASK: The developer's team recently hired Marie Soriano. Please create a username msoriano for the new user and add them to the group "webmasters." Additionally, set up a temporary password for the user and ensure this information is documented in the ticket.
STEPS TAKEN:
-
Logged into freeIPA, added a new user for Marie Soriano, set password and clicked "Add and Edit"
-
Clicked on the "User Groups" tab in order to add msoriano to the group "Webmasters" by selecting it and moving it from "Available" to "Prospective," then clicked "Add"
-
Confirmed user was created by running the
id msorianocommand:
TASK: The user you recently added has been assigned to another project supporting the team in revamping the production webpage. Please add this user to the "support" group.
STEPS TAKEN:
-
Clicked on "Active Users" and searched for msoriano
-
Clicked on username, clicked on "User Groups" tab, then clicked "+Add" to move the support group from "Available" to "Prospective" and clicked "Add"
-
Confirmed user was added to the group by running the
id msorianocommand:
TASK: Please add the following information to the local DNS file on the dev-app server.
STEPS TAKEN:
-
Added vim by installing it with:
dnf install vim -y -
Used vim to add the following IP addresses to the
/etc/hostsdirectory:10.1.XX.XX vcenter.sandbox.prod10.1.XX.XX ipa.procore.dev10.1.XX.XX dev-nagios.procore.prod110.1.XX.XX stage-foreman.procore.prod10.1.XX.XX stage-bacula.procore.prod110.1.XX.XX dev-ansible.procore.prod1 dev-ansible10.1.XX.XX stage-bastion.procore.prod1 stage-bastion10.1.XX.XX nfs-dev.procore.prod110.1.XX.XX stage-graylog.procore.prod
-
Used the
pingcommand to confirm connection to the DNS files
TASK: Please create the following mount points for the upcoming nfs share on the dev-app server.
STEPS TAKEN:
-
Used the following command:
mkdir -pv /nfs/incoming/{home,vhosts,scripts} -
Then used
cdto enter directory/nfs/incomingto confirm mount points were created:
TASK: Please ensure the following NFS shares are mounted permanently on the dev-app server.
STEPS TAKEN:
-
Updated the
/etc/fstabas shown below:
-
Then ran
mount -aandmount | grep nfsto confirm that the directories were mounted correctly:
TASK: Since you will be using SSH in accessing multiple servers, please create a shared directory using your username and 700 permission.
STEPS TAKEN:
-
Used
cdto enter/nfs/incoming/homedirectory and then usedmkdir jlewisto create my shared directory:
-
Then used
chown -R jlewis:jlewis jlewis/to assign my freeIPA username to be the owner of the directory:
-
Lastly, used
chmod -R 700 jlewis/to assign the proper permissions on the shared directory
TASK: The web development team requires you to deploy a development web server using the NEW-YT-DEV-WEBSERVER-TEMPLATE. Please update the necessary information on the newly deployed server.
STEPS TAKEN:
-
Selected my Cluster folder and clicked on "Actions" to select "New Virtual Machine"
-
For creation type, I selected "Deploy from template"
-
Under "Select a Template," I clicked on the "Data Center" tab in order to select the NEW-YT-DEV-WEBSERVER-TEMPLATE and clicked "Next"
-
Labeled new 'Virtual machine name' to
dev-performance-jl.procore.prod1and clicked "Next"
-
When asked to select a compute resource, I selected my cluster folder and clicked "Next"
-
For Storage, DS-01 was selected from the "Batch Configure" tab and then I clicked "Next"
-
For Clone options, I left everything as default and clicked "Next," then clicked "Finish"
-
Since I did not know the password for the account, I interrupted the GRUB process in order to access rescue mode via the shell
-
Added password and created autorelabel file to bypass SELinux
-
Logged into VM and enabled SSH, then proceeded to update the hostname as requested:
-
Added static connection using the following
nmcli c a con-name dev-perf-static ifname ens192 type ethernet ip4 <ipaddress> gw4 <gatewayaddress> ipv4.dns <dnsaddress> autoconnect yes, then setprod-webtoautoconnect no
-
Next, I installed freeIPA following the steps from ticket 5 and ran
id jlewisto confirm that it was properly configured
-
Added the DNS records as outlined in ticket 8, created the mount points as outlined in ticket 9, and permanently mounted the directories as outlined in ticket 10.
-
Ran the following commands to add user procore with the requested information:
useradd -G wheel procoreecho procoreplus | passwd --stdin procore
TASK: The network and security teams are requesting that your dev-performance web server listens on a non-standard port (8001). Please configure your server to meet this requirement and provide the link for testing.
STEPS TAKEN:
-
Installed
policycoreutilsin order to utilizesemanage
-
Accessed the man page for
semanage-portto view the example on how to allow a port viahttpdservices and then ran the following command:semanage port -a -t http_port_t -p tcp 8001 -
To confirm it was added correctly, I ran the following:
semanage port -l | grep 8001
-
Installed httpd with the following command:
dnf install httpd -
Then started the service by running
systemctl enable --now httpd
-
Edited the
/etc/httpd/conf/httpd.conffile to listen for port 8001
-
Installed netstat (
dnf install net-tools) to confirm the server was listening:netstat -tuln | grep 8001
-
Added the following services and port to the firewall:
firewall-cmd --permanent --add-service=httpfirewall-cmd --permanent --add-service=httpsfirewall-cmd --add-port=8001/tcp
-
Created the link:
http://{insertipaddress}:8001/
TASK: The Development team requires you to install a third-party application that can be found on below repository.
STEPS TAKEN:
-
Added repository by using
vim epel.repoto add the following information in the/etc/yum.repos.ddirectory
-
Ran
dnf clean allanddnf repoinfo, then randnf provides tmuxanddnf install tmux
TASK: The security and network team is requesting that all users generate SSH keys to access the Ansible server and our GitLab repository.
STEPS TAKEN:
-
Logged into jlewis account on
dev-app-jl.procore.prod1server and ranssh-keygen
-
Then ran
ssh-copy-id jlewis@dev-ansible.procore.prod1, entered my password and my key was added
-
Then tested connection with
ssh jlewis@dev-ansible.procore.prod1
-
Ran the ssh-keygen command while in the
dev-ansible.procore.prod1directory
-
Then ran
ssh-copy-id jlewis@10.1.XX.XXsince my server had not been added to the /etc/hosts file yet, entered my password and my key was added
-
Accessed
dev-app-jl.procore.prod1server in order to obtain ssh public key by runningcat id_ed25519.pub
-
Finally, accessed GitLab and went to "User Settings" to click on "SSH Keys" and clicked "Add New Key" to paste key into database. Repeated the above steps for
dev-performanceserver as well and uploaded both keys to GitLab:
TASK: To perform automated actions on our infrastructure, please add your development servers to the Ansible inventory file.
STEPS TAKEN:
-
SSH'd into
dev-ansibleserver and edited the/etc/hostsfile by adding the following to the end of the file:10.1.XX.XX dev-app-jl.procore.prod1 dev-app-jl10.1.XX.XX dev-performance-jl.procore.prod1 dev-perf-jl
-
Accessed the inventory file (
/etc/ansible/hosts) and added my ansible group[dev-jl]
-
Ran the following to test ping compatibility with my ansible group
ansible -m ping dev-jl
TASK: In accordance with company policy, all development servers must be patched regularly. Please follow the instructions provided under Infrastructure Resources to patch your development servers using Ansible.
STEPS TAKEN:
-
Used
dnf update python3to confirm it was installed and up to date on both servers -
Used
cd /opt/ansible/patchingto view thedev-patch.ymlfile and copy the contents intodev-patch-jl.yml
-
Ran
ansible-playbook dev-patch-jl-app.yml -Kand received results that did not match the wiki for a successful patch
-
After reading the fatal error messages, removed the
kernel-*exclusion and reran, which resolved 1 error
-
Removed the filter for
rpm-*and thevar=outline and reran β achieved successful results on both servers
TASK: The programmers on the webmasters team want to have a shared directory for collaboration on scripting knowledge for their division on all development servers.
STEPS TAKEN:
-
Read the documentation for ad-hoc commands and wrote the following command:
ansible dev-jl -m ansible.builtin.file -a "dest=/opt/scripts/jlewis/ state=directory owner=jlewis group=webmasters mode=775" -K
-
Command did not complete correctly due to permissions. Ran
ansible --helpand read the options page for "Privilege Escalation Options:"
-
Added
-bflag to run as root:ansible dev-jl -m ansible.builtin.file -a "dest=/opt/scripts/jlewis/ state=directory owner=jlewis group=webmasters mode=775" -K -b
-
Verified both servers had the shared directory created:
ll -d /opt/scripts/jlewis/
-
Equivalent playbook (
create_shared_directory.yml):- hosts: dev-jl tasks: - name: Create directory ansible.builtin.file: dest: /opt/scripts/jlewis/ state: directory owner: jlewis group: webmasters mode: '775'
-
Run with:
ansible-playbook create_shared_directory.yml -K -b
TASK: For inventory purposes, please populate the system inventory database of your dev servers.
STEPS TAKEN:
-
Created file
git-repository.ymlin/opt/ansiblewith tasks to install git, clone the repository, and runhost_facts.sh
-
Ran
ansible-playbook git-repository.yml -K -b -vvβ git installed but cloning stalled due to credential prompt
-
Ran
git clonemanually and confirmed GitLab required authentication
-
Updated playbook to use a Personal Access Token (PAT) for authentication
-
Reran β error showed file was not being found
-
Corrected the script filename to
host_fact.sh_DoNotDelete(confirmed with Infrastructure team) and added double quotes aroundgit_tokenandrepovalues
-
Reran playbook β all tasks completed successfully
-
Verified
.txtfiles were generated in/tmpon both servers
TASK: The development department will need to have a new version of the host_facts.sh created and uploaded to Gitlab.
STEPS TAKEN:
-
Logged in as
rooton dev-app server and accessed/root/scripts:cp host_fact.sh_DoNotDelete jlewis_host_facts.sh -
Appended a date command and ran
cat jlewis_host_facts.sh
-
Ran
git init,git add jlewis_host_facts.sh, andgit status
-
Ran
git commit -m "Adding jlewis_host_facts.sh file"β error populated
-
Ran
git restore --staged jlewis_host_facts.shand checked status
-
Reran as
git commit -m "Add jlewis_host_facts.sh"and received successful results
-
Pushed with
git push -u origin master
-
Confirmed file was added to GitLab repository
TASK: One of our clients intends to build a staging web server to launch a new product. Please set up a new virtual machine (VM) using the naming convention provided below.
STEPS TAKEN:
-
Created new VM named
stage-web-jl.procore.prod1
-
Input all information requested for requirements
-
Powered on VM and interrupted boot process, pressed
eto edit the boot parameters for thelinuxefiline
-
Added the required line of code for GRUB to run the kickstart configuration file and pressed
Ctrl-xto execute
-
Server successfully booted; corrected the default IP address to allow package installation
-
Completed the steps from tickets 8β10 to configure mount points and DNS records
TASK: All new servers must be enrolled in our IPA server. Please enroll the staging server you just created.
STEPS TAKEN:
-
Installed IPA onto staging server by following Ticket 5 steps
-
Added SSH key from stage-web server:
ssh-keygen -t rsa -C "jlewis@stage-web-jl"and copied to dev-ansible withssh-copy-id jlewis@dev-ansibleand vice versa
-
Added stage-web-jl.procore.prod1 to inventory group and performed Ansible Ping test β all successful
-
Added public key for stage-web server to GitLab using
cat id_rsa.pub
23. Disable and prevent SSH root user access to all your servers as part of the security audit requirement
TASK: Disallowing root logins over SSH requires system administrators to authenticate using their own individual accounts and then escalate to root via sudo or su.
STEPS TAKEN:
-
Confirmed SSH access via
jlewisIPA account, then accessed each server as root and edited/etc/ssh/sshd_configto setPermitRootLogintono
-
Ran
systemctl restart sshdandsystemctl daemon-reload
-
Confirmed SSH as root was blocked;
sustill worked for local escalation
TASK: The security department requests that all servers created on our infrastructure only accept SSH communication from the Bastion host.
STEPS TAKEN:
-
Opened two terminal sessions per server, then allowed the Bastion Host IP:
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="{ipaddressofbastionhost}" port port="22" protocol="tcp" accept'
-
Removed default SSH access rules:
sudo firewall-cmd --permanent --remove-port=22/tcpsudo firewall-cmd --permanent --remove-service=ssh
-
Reloaded the firewall to apply the changes
-
Confirmed direct SSH was blocked
-
SSH'd into the Bastion server with my IPA username
-
Successfully SSH'd into my terminal from the Bastion server
-
Updated
/etc/hostson Bastion and added RSA SSH keys from Bastion to each server
TASK: Please configure your dev-app, dev-performance and stage-web virtual machines to send logs to the Graylog server.
STEPS TAKEN:
-
SSH'd into
stage-graylogand confirmed firewall was configured to accept syslog input over UDP port 5140
-
Created
/etc/rsyslog.d/90-graylog.confon each server with:*.* @10.1.XX.XX:5140;RSYSLOG_SyslogProtocol23Format
-
Restarted rsyslog and confirmed log transmission in the Graylog web interface
TASK: Please install the apache webserver to the stage-web-jl.procore.prod1-IP server. Make sure to start and enable apache.
STEPS TAKEN:
-
Checked if httpd was installed:
sudo dnf info httpdandsystemctl status httpd
-
Installed with
sudo dnf install httpd -y, started and enabled withsystemctl enable --now httpd, and confirmed status
TASK: After installing Apache, please open ports 80 and 443 on the stage web server you created. Additionally, allow the Apache service through the firewall.
STEPS TAKEN:
-
On
stage-webVM, ran the following commands:sudo firewall-cmd --permanent --add-port=80/tcpsudo firewall-cmd --permanent --add-port=443/tcpsudo firewall-cmd --permanent --add-service=httpsudo firewall-cmd --permanent --add-service=https
-
Verified with
sudo firewall-cmd --list-alland reloaded
TASK: Please configure ariclaw web server contents on stage-web server which can be found at the URL below.
STEPS TAKEN:
-
Installed
gitand cloned:git clone git@gitlab.com:procoreplusmd/ariclaw.git
-
Configured git user, moved repo to
/var/www/html, and rangit config --list
-
Accessed the IP address link β site was not displaying content
-
Accessed
/etc/httpd/conf.d/welcome.confand commented out all lines
-
Restarted the Apache services:
systemctl restart httpd
-
Site began displaying content at the IP address
-
Verified SSH key was working β non-fatal GitLab warning noted (expected behavior)
-
Changed permissions of the repository folder
-
Created
/etc/httpd/conf.d/ariclaw.confwith the required content
-
Restarted httpd service
-
Cleared browser cache and confirmed site loaded correctly
TASK: Ariclaw has recently changed their phone number. Please update the website content.
STEPS TAKEN:
-
Identified phone number in two places on the current site
-
Updated the
contact.htmlfile in/var/www/htmland/nfs/incoming/vhosts/ariclaw/htdocswith the new phone number
-
Ran
sudo systemctl restart httpdand confirmed the updated number appeared on the site
TASK: The development team requested the installation of MariaDB version 10.3 on your dev-app-jl.procore.prod1-IP server.
STEPS TAKEN:
-
Created and configured the MariaDB repository:
sudo vi /etc/yum.repos.d/MariaDB.repo
-
Installed MariaDB Server and Client, started and enabled the service, and confirmed status
-
Ran the secure installation script:
sudo mariadb-secure-installation
-
Verified login with:
mysql -u root -p
TASK: The infrastructure team will perform a company wide maintenance next week. Please create a snapshot of all your Virtual Machines.
STEPS TAKEN:
-
Accessed vSphere Client and clicked on each VM to access the "Snapshots" tab and clicked "Take Snapshotβ¦"
-
Named each snapshot respective to the server and Ticket 31
-
Waited until all VM snapshots were created successfully
TASK: The Security Team is requiring that all servers be registered with the Foreman server and patched accordingly moving forward.
STEPS TAKEN:
-
SSH'd into
stage-foremanVM β unable to access due to firewall restrictions
-
Updated
/etc/ssh/sshd_configto allow root login and ranssh-copy-id -i ~foreman-proxy/.ssh/id_rsa_foreman_proxy.pub 10.1.XX.XXfor each VM
-
Downloaded the subscription-manager package
-
Installed the Katello CA cert and registered with Foreman
-
Verified registration in Foreman Web UI under Hosts β Content Hosts
TASK: The Security Team needs to create a local user on all the VMs of the infrastructure using Foreman, the user's name is Reuben Camilo, the username would be rcamilo.
STEPS TAKEN:
-
Selected all 3 VMs in Foreman and clicked "Schedule Remote Job"
-
Added command
useradd -C "Reuben Camilo" rcamiloβ all jobs failed
-
Identified
PermitRootLoginwas set tonoβ updated one server to test
-
Identified the flag error: used
-Cinstead of-c
-
Reran with corrected command β still required
PermitRootLogin yes
-
Set all servers to
PermitRootLogin yesand reran β all 3 VMs showed successful
TASK: We would like to establish a central location on the server dev-app-jl.procore.prod1-IP to store logs.
STEPS TAKEN:
-
Powered off VM, added a new 1 GB hard disk in vSphere, and powered back on
-
Confirmed disk at
/dev/sdbusinglsblk
-
Used
fdisk /dev/sdbto create partition
-
Created
ext4filesystem via VG and LV
-
Created mount point
/lfjs/logs, userlfjs, and assigned ownership:sudo chown -R lfjs:webmasters /lfjs/logs
-
Set SGID permissions:
sudo chmod 2755 /lfjs/logs/
-
Edited
/etc/fstabto mount persistently
-
Confirmed with
lsblk
TASK: We have received numerous emails from developers regarding issues with SSH access to your dev-app server. Please investigate the issue and attach the last 20 lines of the logs to this ticket.
STEPS TAKEN:
-
Used
id apprenticeto confirm user existed, then attemptedssh apprentice@localhost
-
Attempted to access
/lfjs/logsasapprenticeβ permissions had reset to root
-
Re-applied correct permissions and confirmed as
apprenticeviasu
-
Accessed the directory β no contents found
-
Checked
/var/log,tallylog, andlastlogβ nothing notable found
-
Ran
find / -type f -user apprentice | grep sshβ noted similar keys inknown_hostsandknown_hosts.old
-
Checked
/var/log/securefor the last 20 lines β found that userapprenticewas not usingport 22for SSH
TASK: One of our clients has reported that the del.extreme-ix repository is not functioning properly on dev-app-[initials].procore.prod1-IP.
STEPS TAKEN:
-
Accessed the
epel.repofile β notedenabled=1was included during original setup (not part of original instructions). Possible issue if the URL requireshttps.
TASK: User created in ticket 6 has reported that he/she is unable to write to the directory /lfjs/logs when logged into dev-app-[initials].procore.prod.
STEPS TAKEN:
-
msorianois part of the webmasters group but still unable to write. Created an ACL:sudo setfacl -m u:msoriano:rw /lfjs/logs
TASK: The networking team has reported a malicious IP that is trying to get into our network.
STEPS TAKEN:
-
Ran the following on each VM to block the malicious IP:
sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" source address="XX.XX.XX.XX" port port="22" protocol="tcp" drop' -
Ran
sudo firewall-cmd --reloadto apply the rule -
Ran
sudo firewall-cmd --list-allto confirm it was applied
TASK: The development team is helping us troubleshoot an issue with apache on stage-web-jl.procore.prod1-IP.
STEPS TAKEN:
-
Ran both
dnf list httpdanddnf info httpdto obtain the version information in two different formats
TASK: The administration is requesting a detailed list of all users registered on the FreeIPA server. Please generate this list using IPA commands and redirect the output to a file.
STEPS TAKEN:
-
Used
ipa user-find msorianoto confirm user was in database and then ranipa user-del msorianoto delete user
-
Ran
ipa user-find >> ipa_users.txtto export all users into a file
TASK: The engineering team would like to have a cron job configured on the dev-app-jl.procore.prod1-IP server.
STEPS TAKEN:
-
Copied the script file from
/nfs/incoming/scripts/logs.shto~/and changed ownership to my freeIPA user
-
Confirmed
crondwas running and rancrontab -eu jlewisto input the cron schedule
-
Checked
/var/log/cronand confirmed the script was running every 6 hours
TASK: Please create a cronjob on stage-web-jl.procore.prod1-IP server to restart apache every 2 days at 11 AM.
STEPS TAKEN:
-
Set up initial cronjob on
stage-web-jlserver
-
Confirmed cronjob was running by setting up a temporary duplicate that ran every 2 minutes
-
Checked
/var/log/cronβ job ran but produced an error (required authentication)
-
Ran
sudo crontab -eu rootto add the cronjob as root and tested
-
Finalized cronjob:
0 11 */2 * * systemctl restart httpdas root
TASK: To utilize the NAGIOS service, you need to create an HTTP user on the server.
STEPS TAKEN:
-
SSH'd into
dev-nagiosserver
-
Added user with:
htpasswd /usr/local/nagios/etc/htpasswd.users jlewis
-
Edited
/usr/local/nagios/etc/cgi.cfgand appendedjlewisto all directives -
Restarted Nagios and verified user access via the Nagios web UI
TASK: Please add your Virtual machines to Nagios monitoring system.
STEPS TAKEN:
-
Installed NRPE and Nagios plugins on each server:
sudo dnf install epel-release nrpe nagios-plugins-{load,http,users,procs,disk,swap,nrpe,uptime} -y
-
Edited
/etc/nagios/nrpe.cfgand updatedallowed_hoststo include the Nagios server IP
-
Started NRPE and created config files for each server by copying
yt-templates.cfg.bak
-
Linked new config files in
/usr/local/nagios/etc/nagios.cfg
-
Reloaded Nagios β received warning message
-
Ran
sudo /usr/local/nagios/bin/nagios -v /usr/local/nagios/etc/nagios.cfgto diagnose
-
Commented out missing referenced file, reran β no errors or warnings
-
Found that someone had changed the user:group of
/usr/local/nagios/. Ransudo chown -R nagios:nagios /usr/local/nagios/to correct and got the service running again
-
Verified hosts were being monitored in the Nagios web UI β both statuses showed OK!
TASK: Please configure the FTP server on dev-app-jl.procore.prod1-IP and the FTP client on stage-web-jl.procore.prod1-IP.
STEPS TAKEN:
-
Used
sudo dnf install ftp -yto install ftp on both servers, then copiedftp-prod.configfrom the NFS share
-
Added ftp service to both servers:
sudo firewall-cmd --permanent --add-service=ftpand reloaded
-
Ran
ftp stage-web-jlβ received an error
-
Installed
vsftpdon both servers, opened port 21, and started the service
-
Used
ftp dev-app-jlfromstage-web-jl, logged in with freeIPA credentials, and ranget ftp-prod.config
-
Confirmed the file was on the home directory with
ll
TASK: Please create an ansible playbook that would perform the below tasks on dev-app-jl.procore.prod1-IP.
STEPS TAKEN:
-
Created
user_add.ymlin/etc/ansible/playbook/jlewisreferencing Ansible documentation for user and dnf modules
-
Ran
ansible-playbook user_add.yml -K -b
-
Confirmed on
dev-app-jlserver that everything was completed correctly
TASK: Please create an Ansible playbook to close ports 80 and 443 on dev-app-jl.procore.prod1-IP.
STEPS TAKEN:
-
Reviewed
ansible.posix.firewallddocumentation and wroteclose_ports.yml
-
Ran
ansible-playbook close_ports.yml -K -b
TASK: The infrastructure team will deploy a company wide kernel update next week.
STEPS TAKEN:
-
Accessed
/lib/moduleson each server and created backups to the requested location:sudo tar -czvf /nfs/incoming/vhosts/backup/dev-app-jl-kernel-backup.tar.gz 5.14.0-621.el9.x86_64sudo tar -czvf /nfs/incoming/vhosts/backup/stage-web-jl-kernel-backup.tar.gz 5.14.0-575.el9.x86_64/
TASK: Please create a symlink in your home directory on your stage-web VM for the file /var/www/html/ariclaw/elements.html and name the symlink elements.
STEPS TAKEN:
TASK: The development team requires information about the MariaDB database you installed on your dev-app server. Please query the logs using the Graylog server to determine the exact date and time of the installation.
STEPS TAKEN:
-
Accessed Graylog site and searched for "MariaDB"
-
Filtered by source
dev-app-jland changed time range to "last 14 days" β confirmed MariaDB was initially installed on October 18
TASK: The infrastructure team is requesting that you scale up dev-app-jl.procore.prod1-IP due to an increase in user requests. Please create a clone of the virtual machine before making any changes.
STEPS TAKEN:
-
Powered down
dev-app-jland cloned it asdev-app-jl-clonein vSphere
-
Updated memory from 1GB to 1536 MB
-
Powered back on and ran
free -mhto confirm RAM was updated
-
Powered on the clone VM and confirmed the additional memory was reflected
TASK: Users have requested additional space to be allocated to the /lfjs/logs filesystem on dev-app-jl.procore.prod1-IP.
STEPS TAKEN:
-
Ran
lsblkβ confirmed no free space available to extend existing filesystem
-
Ran
fdisk /dev/sdbto add another 100MB partition
-
Created the PV to add it to the VG
-
Extended the VG and LV:
sudo vgextend vglogs /dev/sdb2andlvextend -r -L +96 /dev/vglogs/lvlogs
-
Confirmed new storage amount with
lsblk
TASK: Please create a script to gather information about your VM.
STEPS TAKEN:
-
Identified the required commands after reviewing man pages:
date,last -F 10,free -h,uname -srv,nmcli | grep "inet4 10.*"
-
Created
/tmp/serverinfo.infoand populated it with the script content
-
Made executable with
sudo chmod +x serverinfo.infoand ran on each server
-
Network access was giving issues at the time β unable to provide additional screenshots for the second server.
TASK: For documentation purposes, we request that everyone create their own GitLab repository and upload their Ansible playbooks and Bash scripts.
STEPS TAKEN:
-
Used
rsyncto copy playbooks fromdev-ansibletodev-app-jl:rsync -avs --delete jlewis@dev-ansible:/etc/ansible/playbook/jlewis /home/jlewis/gitlab
-
Created a new blank GitLab project at
https://gitlab.com/justintimejlew/ansible_playbooks
-
Initialized git, added remote, committed, and attempted
git push -u origin masterβ failed because the branch was namedmain
-
Ran
git pull --rebase origin mainthengit push -u origin mainsuccessfully
-
Confirmed files were committed to: https://gitlab.com/justintimejlew/ansible_playbooks/
TASK: The ariclaw website is generating a large number of logs. Please modify the log rotation settings to rotate the HTTPD logs daily and ensure that log rotation retains logs for only 14 days on stage-web-jl.procore.prod1-IP.
STEPS TAKEN:
-
Accessed
/etc/logrotate.d/httpdand setdailyandrotate 14
-
Ran
sudo systemctl status logrotateβ confirmed service runs and deactivates after completion
-
Forced the changes:
sudo logrotate -f /etc/logrotate.d/httpd
TASK: Please add both of your servers to CheckMK monitoring tools.
STEPS TAKEN:
-
Installed wget and the Checkmk agent, then ran
sudo dnf -y localinstall check-mk-agent-2.3.0p2-1.noarch.rpm
-
Opened port 6556 for Checkmk agent communication and started the socket
-
Accessed the Checkmk Web Interface, navigated to the DEV directory, and clicked "Add Host"
-
Entered hostname and IP for each VM and clicked "Save & run service discovery"
-
Clicked "Changes" to activate
-
Clicked "Activate on selected sites"
-
Verified hosts were created by filtering with "jl" in the Hosts view
TASK: We have been receiving a lot of emails about your stage-web-jl-procore.prod1-IP. Verify and troubleshoot critical services using CheckMK.
STEPS TAKEN:
-
Searched for
stage-web-jlin CheckMK and clicked on the hostname
-
Checked "Monitoring Status" β nothing of concern found
-
Reviewed "Rules" β listed several rules CheckMK was running
-
Checked "Test notification" β found fallback email address was missing
-
Added email address to the system
-
Activated changes on selected sites
-
Reviewed all monitoring services β NTP showed "CRIT"
-
Ran
sudo systemctl enable --now chronydand rescanned
-
NTP showed OK after rescan. Removed test email from Global Settings.
TASK: Procore-Plus has deployed a new daemon that needs to be running on every server.
STEPS TAKEN:
-
Copied
procored.shto/usr/bin/procoredon each server
-
Ran
sudo systemctl enable --now procoredβ received an error (no unit file found)
-
Created
/etc/systemd/system/procored.serviceunit file
-
Ran
sudo systemctl daemon-reload,sudo systemctl enable --now procored, and confirmed status
TASK: Please configure an NFS server on dev-app-jl.procore.prod1.
STEPS TAKEN:
-
Created
/nfs-jlewisdirectory and downloaded the required rpm
-
Edited
/etc/exportsand added:/nfs-jlewis stage-web-jl(rw,sync,no_root_squash)
-
Ran
exportfs -ravto refresh the NFS export table
-
Confirmed
nfsservice was allowed in the firewall
-
On
stage-web-jl, created mount point, mounted the NFS share, and confirmed withdf -hT
-
Updated
/etc/fstabfor persistent mounting
-
Ran
sudo systemctl daemon-reloadandsudo mount -aβ confirmed access to/nfs-jlewis
TASK: The infrastructure team needs to back up your VMs. Please install the Bacula Client (downgrade) and Libraries on both of your servers.
STEPS TAKEN:
Client Installation:
-
Installed required development tools and libraries on each server
-
Downloaded and extracted Bacula 9.6.6 source code
-
Configured the build for client only:
sudo ./configure --enable-client-only -
Compiled and installed:
sudo make -j$(nproc)andsudo make install
-
Created the
bacula-fd.serviceunit file
-
Enabled and started the Bacula File Daemon β confirmed symlink was created
Client Configuration:
-
Configured
/etc/bacula/bacula-fd.conf
-
Configured
/etc/bacula/bconsole.conf
-
Added clients in the Bacula Web Interface
-
Verified connectivity for each server via "Status Client"
-
Created backup jobs using the "New job wizard"
-
Ran backup jobs for each server
-
Verified backup ran successfully for each server
TASK: The development team has notified all users to manually run a performance script on all servers. Please execute the script located at /nfs/incoming/scripts/perfomance.sh as the root user. After running the script, a file named /setup_file.cfg should be created.
STEPS TAKEN:
-
Created a snapshot on each VM prior to starting this ticket.
-
Accessed
/nfs/incoming/scripts/and ransudo ./perfomance.shon each VM βsetup_file.cfgwas created on each.
TASK: After running the performance script, the development team noticed some issues with the NFS and HTTPD services. Most of the files for these services were deleted. Please restore these files from the backup you created earlier using Bacula.
STEPS TAKEN:
-
Visited Bacula Web Interface β "Restore wizard" was not working properly, so accessed the "Jobs" tab
-
Clicked "Job history" and selected the "Full" backup to click "Details"
-
Clicked "Restore"
-
Navigated the folder tree to the
etcfolder, added both files, and clicked "Next"
-
Left "Where" settings as default
-
Left "Options" settings as default
-
Clicked "Run restore"
-
Restore job produced warning messages, but files were placed in
/tmp/restore/etc
-
Moved files to
/etcusingmv:sudo mv exports /etc/andsudo mv httpd/ /etc/httpd.bak
-
Confirmed httpd and nfs-server services were running
TASK: The Public Relations department wants to deploy a WordPress website. Please follow the instructions at the URL below to complete this request.
STEPS TAKEN:
-
Ran
docker run --name some-wordpress --network some-network -d wordpressβ selecteddocker.io/library/wordpress:latest
-
Received an error
-
Tried
docker run --name some-wordpress -p 8080:80 -d wordpressβ same error
-
Ran
sudo podman system migrateand reran β ran successfully
-
Accessed
10.1.XX.XX:8080β redirected to WordPress setup page, confirming WordPress was configured
TASK: Some end users are reporting connectivity issues with the internal dev-performance system. Please troubleshoot the issue.
STEPS TAKEN:
-
Checked network connection with
nmcli c sandping -c4 google.com
-
Checked
sudo systemctl status NetworkManagerβ service was running
-
Ran
ss -tuln | grep :80andss -tuln | grep :443β HTTPS did not appear to be working
-
Found that
httpd.confwas missing from/etc/httpd/conf. Copied it from the backup restoration and addedListen 443
-
Restarted httpd and confirmed
curl -k http://localhostreturned HTML content
TASK: Please stop and disable the procored system service on all servers. Afterward, verify that CPU utilization is normal using Check_MK.
STEPS TAKEN:
-
On each server, ran
sudo systemctl stop procored,sudo systemctl disable procored, and confirmed status
-
Visited CheckMK and confirmed CPU utilization was only about 5.06%
TASK: The webmaster would like you to add a banner indicating that the website will be down for maintenance.
STEPS TAKEN:
-
Attempted
sudo git clone git@gitlab.com:procoreplusmd/procore-products.gitβ error
-
Ran
git initthen cloned withsudo git clone
-
Ran
git config --global --add safe.directory /var/www/htmland checked git status
-
Commented out all lines in
/etc/httpd/conf.d/welcome.confand restarted httpd
-
Removed the hidden
.gitdirectory and re-cloned usinghttps://instead ofgit@
-
Copied
index.htmlto/var/www/html/β site displayed web contents athttp://10.1.XX.XX
-
Copied the maintenance banner
index.htmlfromwebsiteDownForMaintenancefolder
-
Copied all site contents and applied the maintenance banner
TASK: Please modify the profile picture for user Jessica on the website. Use the following files to update her information.
STEPS TAKEN:
-
Used browser "Inspect" tool to identify Jessica's current photo as
notification-01.jpg
-
Confirmed the updated photo in GitLab was labeled
notification-dog.img
-
Ran:
sudo cp notification-dog.img /var/www/html/img/notification-01.jpg
TASK: The website requires a new feature to be added to the performance chart.
STEPS TAKEN:
-
Took a snapshot of the performance chart before making changes
-
Confirmed file location in
/var/www/html/jsand ran:sudo cp tooplate-scripts-orig.js /var/www/html/js/tooplate-scripts.js
TASK: Please update the Order List table on the Procore Products website to ensure that all cancelled orders are removed.
STEPS TAKEN:
-
Took a screenshot of the current order list
-
Accessed
/var/www/html/procore-products/removeCancelledProductsand ran:sudo cp index.html /var/www/html/
TASK: Now that all issues on the dev-performance server have been resolved, please remove the maintenance banner that was added.
STEPS TAKEN:
-
Took a screenshot of the site before making changes
-
Accessed
/var/www/html/procore-products/websiteUpAndReadyand ran:sudo cp index.html /var/www/html/
TASK: Please install terraform on your dev-app-jl.procore.prod1 VM.
STEPS TAKEN:
-
Visited https://developer.hashicorp.com/terraform/install and selected ARM64 Version: 1.13.4
-
Downloaded Terraform:
sudo wget https://releases.hashicorp.com/terraform/1.13.4/terraform_1.13.4_linux_arm64.zip
-
Unzipped and moved to
/usr/bin
-
Ran
terraform -vβ received an error
-
Installed via the HashiCorp repo instead β
terraform -vworked successfully
-
Cloned
dev-performance-jlto a template in vSphere
-
Confirmed template was created by deploying a test VM from it
TASK: Please use Terraform code to deploy the newly fixed web server into production. The server should be created within your vSphere resource pool.
STEPS TAKEN:
-
Created a
terraformdirectory ondev-app-jl, initialized git, and cloned the Terraform repo
-
Edited
terraform.tfvarswith the required configuration
-
Updated
main.tfwith hostnameprod-webserver-jl.procore.prod1
-
Ran
sudo terraform init
-
Ran
sudo terraform planβ resource pool not found due to case-sensitivity
-
Updated
terraform.tfvarsand reransudo terraform planβ successful
-
Ran
sudo terraform applyβ received an error (likely a permissions restriction on VM deletion)
-
Despite the error, the VM was created and I was able to log into it
TASK: The web development team requests that PHP be installed on stage-web-jl.procore.prod1-IP.
STEPS TAKEN:
-
Ran
sudo dnf info phpto check availability
-
Ran
sudo dnf install phpto install PHP and all dependencies
TASK: The development team has released a new application, and you have been assigned the task of deploying it and configuring the associated database.
STEPS TAKEN:
-
Cloned the CRM project:
sudo git clone https://gitlab.com/procoreplusmd/crm-project
-
Extracted and copied the CRM project to
/var/www/html
-
Installed MariaDB, enabled the service, and ran
sudo mysql_secure_installation
-
Attempted to install Apache β service could not start because
httpd.confwas deleted by the script
-
Restored
httpd.confvia rsync fromdev-perf-jl
-
Restarted httpd, opened firewall ports, and confirmed site access
-
Installed PHP with opcache and created
/var/www/html/info.phpto test
-
Installed MySQL support and common PHP modules
-
Reran install without
php-xmlrpc(not found) and restarted Apache
-
Revisited info.php to verify module installation
-
Installed phpMyAdmin:
sudo dnf -y install phpMyAdmin
-
Edited Apache's phpMyAdmin config to allow remote access
-
Attempted phpMyAdmin login β did not work initially
-
Updated
dbconnection.phpfiles with the correct password -
Accessed the admin panel at
http://10.1.XX.XX/crm/admin
-
Created the
crmdatabase in phpMyAdmin and imported:mysql -u root -p crm < /var/www/html/crm/crm.sql -
Recreated user account and confirmed access to the site
TASK: Please create a wiki document outlining the steps to install and configure a firewall for an HTTP server. Name the page using your username and place it in the Procore-Plus How-To Guides section.
STEPS TAKEN:
-
Created an account on the wiki site
-
Visited the Procore-Plus Wiki and clicked the "Edit" tab
-
Populated the editing page with my username link
-
Added the following content to the "JLewis" wiki page and confirmed changes were shown
TASK: The security team has notified all administrators to conduct an audit and scan for security vulnerabilities and loopholes on your Linux machine using the Lynis tool.
STEPS TAKEN:
-
Ran
sudo dnf provides lynisand confirmed the package, then installed withsudo dnf install lynis
-
Ran
sudo lynis audit system >> /tmp/systemauditto perform the audit and redirect output
TASK: The security team requires that administrators create a Python script to gather information about the VM they are working on (Hostname, IP address, OS System, Release, Version, Machine architecture).
STEPS TAKEN:
-
Confirmed Python 3.9.23 was installed via
python -V. Createdsudo vim python.shon the dev-app server with the required script content.
-
Made executable with
sudo chmod u+x python.shand ran:sudo ./python.sh >> /tmp/pythoninfo
TASK: Accessing the Windows Active Directory server to manage and administer user accounts.
OBJECTIVE: Use the Remote Desktop Client to connect to Active Directory services.
STEPS TAKEN:
-
Accessed Microsoft Remote Desktop on iMac and clicked on "Add PC"
-
Input the Server and User information as directed on the wiki
OBJECTIVE: Add a new Organizational Unit (OU) under the domain using the Active Directory Users and Computers tool.
STEPS TAKEN:
-
Right clicked on the VITI folder under "sandbox.prod β Yellow tail" and selected "New β Organizational Unit," named it "JLewis-VITI," and unchecked "Protective Container"
OBJECTIVE: Enhance system administration skills through identity management using Active Directory.
STEPS TAKEN:
-
Right clicked on the OU, then selected "New" and "User"
-
Entered user information and clicked "Next"
-
Selected "password never expire" and entered the provided password
-
Clicked "Finish"
OBJECTIVE: Enhance system administration skills through identity management using Active Directory.
STEPS TAKEN:
-
Right clicked on the OU, then selected "New" and "Group"
-
Entered group name and confirmed "Global" and "Security" were selected
-
Right clicked on user "Mike Tyson" and selected "Add to a groupβ¦"
-
Entered group name and clicked "OK"
OBJECTIVE: Research and create a document explaining what an Active Directory Domain is and the steps to join a Windows 10 device to it.
STEPS TAKEN:
-
Researched via Microsoft documentation: https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/get-started/virtual-dc/active-directory-domain-services-overview
-
Summary: An Active Directory Domain is a directory on a network that stores information about users for access across the network based on assigned permissions. Users can be added to groups and permissions can be assigned individually or collectively. Users and groups can also be removed to preserve the structure of any organization.
-
Researched how to join a Windows 10 device via Microsoft documentation: https://learn.microsoft.com/en-us/windows-server/identity/ad-ds/manage/join-computer-to-domain β reviewed the Control Panel Method, Settings App Method, and Command Line Method.
TASK: Scan a Linux System with OpenVAS and generate an auto report of all the vulnerabilities found on the system.
STEPS TAKEN:
-
Opened the firewall on development servers:
sudo firewall-cmd --permanent --add-source=10.1.XX.XXandsudo firewall-cmd --reload
-
Accessed the OpenVAS console and logged in
-
Created new credentials under Configuration β Credentials
-
Created a scan target under Configuration β Targets
-
Created a scan task under Scans β Tasks
-
Clicked "Start" to run the scan
-
After the scan finished, accessed Scans β Reports and downloaded the report
-
Scan results:
OpenVAS-Rescan-Validation.pdf
TASK: This ticket documents the remediation and subsequent validation of vulnerabilities reported in the initial OpenVAS scan. As part of compliance requirements, provide a clean re-scan as evidence.
STEPS TAKEN:
-
Reviewed the scan report. For Host Authentications, checked
/etc/ssh/sshd_configand changedPermitRootLogintoyesto allow root login and result in a "Success." -
Could not update the OpenVAS version (22.4.1 to 23.0.1) as it was outside my control. Instead, ran
sudo dnf updateto update all packages and address a portion of 2.1.1 High general/tcp threats.
-
For 2.1.2 Medium 8080/tcp, ran
sudo podman system migrateto stop the container and remove the public configuration.
-
For 2.1.3 Medium 21/tcp, reviewed
/etc/vsftpd/vsftpd.conf, commented out problematic lines, and removed the ftp service from the firewall:sudo firewall-cmd --remove-service=ftp -
For 2.1.4 Low general/icmp, edited
/etc/sysctl.confto addnet.ipv4.icmp_echo_ignore_all = 1. Then ran the following firewall commands to block ICMP timestamp responses:sudo firewall-cmd --permanent --add-rich-rule='rule family="ipv4" icmp-type name="timestamp-reply" drop'sudo firewall-cmd --reload
-
For 2.1.5 Low general/tcp, added
net.ipv4.tcp_timestamps = 0to/etc/sysctl.confand ransudo sysctl -pto apply. -
Rebooted the VM and reran the scan β all vulnerabilities were addressed. Scan results:
OpenVAS-Rescan-Validation-2025-11-17.pdf
