SPPMon [1.2.4] / SPPCheck [1.0.3] - 2023-02-23

SPPMon [1.2.4] / SPPCheck [1.0.3] - 2023-02-23

Added

• Added a requirements.in file for better versioning of dependencies

Changed

• Added some macOS ignores to the gitignore
• Upgraded the dockerfile to a more secure standard
• Changed the requirements within the requirements.txt to be soft-locked, using ~= to allow patch upgrades.

Fixed

• Fixed security issues with the dependencies by updating them all to the latest state

SPPMon 1.2.3 and SPPCheck 1.0.2 Release

SPPMon [1.2.3] - 2022-12-05

Changed

• Upgraded Dockerfile python version from 3.8 to 3.10

Fixed

• Upgraded requirements file

Known Issues

• There might be an issue with the alerts saved within the dashboard, requires investigation.

SPPCheck [1.0.2] - 2022-12-05

Added

• Debug information for accessing the excel json structure

Fixed

• Removed another feature to allow comparability with python3.8 instead of 3.9
• Choosing the an automatic RP in report controller no longer debugs the keyword-content prefix, but the self defined variable rp_prefix
• Excel Json structure path within the excel controller has now the correct sppCheck case, as windows is case-insensitive
• Temporary directory path within the report controller has now the correct sppCheck case, as windows is case-insensitive
• Crash of Grafana-Panel downloader when using https fixed
• Upgraded requirements file

Known Issues

SPPMon 1.2.2 and SPPCheck 1.0.1 Release

SPPMon [1.2.2] - 2022-08-30

Changed

• Updated the requirements file to the latest versions

Fixed

• Included pandas and numpy in the requirements file
• Removed Python 10 features, allowing compatibility with python 8.

SPPCheck [1.0.1] - 2022-08-30

Fixed

• Split up folder between sppcheck and sppCheck which was not shown on a windows development system
• Included pandas and numpy in the requirements file
• Removed Python 10 features, allowing compatibility with python 8.
• Fix of the default config file, removing an unwanted comma, causing both SPPMon and SPPCheck to fail.

SPPMon 1.2.1 and SPPCheck 1.0.0 Release

Added

• Release of SPPCheck v1.0.0

  • Its changes will be logged within this file
  • SPPCheck function and internal structure
  • Added Grafana Dashboard for SPPCheck
  • Added two new tables to the definitions.py file which are only executed if SPPCheck is executed.
  • Adds pandas-stubs, openpyxl and pyxlsb to requirements file

• Adds FullLogs and LoadedSystem information to the Grafana SPPMon Runtime Duration panel.

• Adds clarification that all timestamps are changed into second precision on insert.

• Adds support for batch insert to insert into a different retention policy

• Selection queries now also support an alternative retention policy to query from

• CreateRP-Method in the influxClient to allow creating non-lasting retention policies

• Prints total count of errors during the execution if there are any - instead of "script finished" output

• Added type spelling dictionary to the settings file

• Added linting settings to the settings file

Changed

• Upgraded CodeQL to v3

• Extended SppUtils.mk_logger_file method by an logger_dir argument to allow different log files for SPPMon and SPPCheck

  • Added "sppcheckLogs" to the gitignore file.

• Moved all cSpell settings from the workspace file to the settings file

• Removed/Moved total count of exceptions during execution

• Moved verbose setting of sub-modules into init method to avoid issues and clarify program structure

• Error-Messages now have the prefix "ERROR: "

• Added InfluxClient function "get_list_rp" to query all retention policies, moving it out of existing functionality

• Moved pid files-functions and other functions from SPPMon to helper functions.

  • Adjusted log messages to make their message generic if required.
  • Added arguments to replace self-access.

• Changed default log and pid-file locations from home/sppmonLogs/FILE to spectrum-protect-sppmon/sppmonLogs/FILE

• Added init declarations in the SPPMon __init__ function to allow overview of all self-vars

• Changed SelectionQueries to only take a single table instead of an List of such, as it isn't required and complicates the code

• Predefined retention policies are no longer protected in the definitions module.

• Sending a select query now raises a error when it fails.

• Allows SelectionQueries from table to be another inner-selection query

  • Changed all calls of the constructor
  • Made sure to only allow inner queries when using the selection-keyword.

• Changed the severity of unknown type-annotation due to submodules not being typed from error to warning.

• Minor printing edits when generifing Dashboard

• Removes deprecated functions

Fixed

• Code scanning alert #2 and #3: Printing passwords into logger due to an faulty if-expression.

• CheckPID file now uses the class-verbose setting and no longer ARGS-Verbose, making it independent of actual args.

• Fixed PID file checks under windows

• Fixed PID file entries not being deleted.

• Removed unnecessary and confusing check when creating a SelectionQuery

• Fixed typo in exception_utils.py file, renaming it and all references on it.

• Lots of typos in the inline documentation

• SelectionQuery and associated methods: Introduces Optional annotation to fix linter error

Known Issues

• SPPCheck lacks a lot of documentation, which is to be delivered on a later date.

• The Grafana Dashboards are missing the alerts. This is known and will be fixed within the next patch (see #115 )

• In newer Grafana versions the dashboard import might corrupt the datasource name. A fix is unavailable yet, though the error is only visual.

• Some typos in the table definitions (commited / uncommited) cannot be fixed. This would not be backward-compatible and break the database.

1.1.1 Release

Summary

• Updates requirements, includes important security patches due to updated dependencies
• Adds skip_cmds option to the config files, skipping mpstat is now possible
• Fixing --test function for multiple declared vSnaps / VADPs
• Improvements to the Grafana-Dashboards

Added

• Adds FullLogs and LoadedSystem information to the Grafana SPPMon Runtime Duration panel.
• Adds license information into each SPPMon code file.
• Specifies the encoding and reading permission when opening config files.
• Added ConnectionUtils function rest_response_error. This function helps to extract the response-error message and includes all important pieces of information into a ValueError. This error should be raised afterward.
• Config-file option for ssh-clients skip_cmds. List of strings like ["mpstat", "ps"] to skip commands on certain clients.

Changed

• Avoids the wget-certificate check when installing and downloading the python tgz in the installer.

• Updates the requirements.txt to include sub-dependencies and updates to latest version.

• Marked the default group time when using the template CQ more clearly as [*]

• Grafana Dashboards:

  • 14-Day dashboard:
    • Changed VADP Proxy state table to support the VADPName grouping and status
      • Used Grafana organizing mechanics to leave query intact, hiding old data with status=null and vadpName="".
    • Changed VADP Proxy state per site to only use new data since 14-day is not too long
      • Changed to use field count and group over status instead of two separate queries
  • 90-Day dashboard:
    • Changed %-Enabled VADP dashboard to total count dashboard
    • Added for both VADP Proxy state and total count dashboard a new query, grouping over status and grouping on count. Old queries left intact for backward compatibility.
    • Hides series with null/none, adding average to legend.

• InfluxDB-Table VADPs:

  • Moved fields state and vadpName to tags
    • Renamed state to status to avoid issues due to double-named tags/fields
  • Changed CQ to group only over 'old' Tags
  • Changed aggregation from a split over enabled/disable to grouping over the state itself.
    This removes now-duplicate CQ definitions and all WHERE-grouping clauses.

• REST-API Login and Statuscheck for get_objects use the new function rest_response_error to raise their error.

• Adds the skip_cmd option to the default-config file.

Fixed

• --test-execution: Fixes unusual KeyError when using a config file with more than one vSnap (or other) ssh-client.
• VADPs are no longer dropped due to being marked as duplicates by the InfluxDB.
• Fixes Issue #86 vSnap hanging up on start due to pool call by checking first if the vsnap command is available.

Check out the complete changelog here.

1.0.1 Release

• SPPMon now features an install script!
  • Check out the documentation on how to easily and quickly install SPPMon
  • Don't forget to also check the new stand-alone scripts for creating config files and adding them to crontab!
• SPPMon now features a --test argument to test your config file before execution
• Reworked Grafana-Dashboard import process for easy monitoring and alerting of multiple servers
• Reduces the load on the SPP-Server to a minimum while also minimizing executing timings
• Fixed many bugs and increased stability

Check out the complete changelog here.

Beta 0.5.4 Hotfix

• Bugfixes flat CPU-statistics due ps unexpected behavior.
  • ps no longer tracks CPU-Data, Track of RAM and some other system informations remains.
  • Re-introduced top-ssh command, but only collecting CPU-Statistics (see top-memory-truncation issue #14 & #32)

--> The process-stats panel is accurate again after applying this fix.

Check below the changelog of version v0.5 and related hotfixes

Beta 0.5.3 Hotfix

• Changed top level execption catching from catching only our own ValueErrors to any execptions - preventing a whole abort of SPPMon if something unexpected happens.
  This will reduce the need of urgent hotfixes like this one.

• Hotfixes SPPMon storages request to fail due free or total beeing none, crashing whole SPPMon execution
• Changes empty result severity of REST-Requests from error to info
• Changed typings from critical components to support better linting

Check below the changelog of version v0.5 and related hotfixes

Beta 0.5.2 Hotfix

Hotfix v0.5.2 Beta:

• Hotfixing version endpoint for SPP 10.1.8.1 breaking SPP REST-Endpoint

• Reduces Spam of empty error messages

• Typing and printing changes within the script sucessfull part

• General Dashboard changes:

  • Alert Notification bodies now contain meaningful messages
  • Changes Office365 type from none to percent

• Other bugfixes:

  • NaN error in CPU-Stats
  • ssh pools now contains the correct time stamp
  • allow mpstat to have a full date format (21 vs 2021)

Check below the changelog of version v0.5 and related hotfixes

Beta 0.5.1 Hotfix

Hotfix v0.5.1 Beta:

• Removed the filter messageId from any query to the vmReplicateStats and vmReplicateSummary table:
  • Additional Statistics-Grafana panel in 14-Days and Multiple Server Dashboard
  • daily total data protected-Grafana panel in 14-Days Dashboard

Note: This does not affect the vmBackupSummary-Table (or queries against this one). This table does require a messageId as filter.

Changelog v0.5 Beta:

Major Changes and Features:

• Includes now SPP 10.1.6 additional job information
  • Sucess ratio of VM's backup (Example: 54/70 VM's backuped)
• Tracks root storage space of Server and vSnap clients
• Check if all components are configured correctly and working with --test
• Office365 Backup analysis (Items backuped, data transferred)
• Allows to copy a database with a new name in case of a naming error / backup.
• Adds and reorganized panels within Grafana
• Improvement of logging messages, code structure for late implementations and lot of bugfixes

Bugfixes:

• Replaced top command by ps command to bugfix the truncation of memory sizes of ssh-clients
• Reduces the partial send error-message of the influxDB to a minimum.
• Influx Version is now displayable even when not using an admin user
• The default table now also has a retention policy, fixing the use of the default split if no table is declared
• Added a missing comma in df_ssh table delcaration
• Fixes a bug with negative ssh-FREE ram values
• Now uses realpath for path-creation

SPPMon startup argument changes:

• Reminder: --minimumLogs depricated, to be removed in V1.0. Use --loadedSystem instead
• --processStats depricated, integrated into --ssh
• transfer_data removed. Use copy_datase instead.
  • removed --old_database, integraded into copy_database CFG file.
• --test implemented

InfluxDB tables:

• jobs: Added new fields numTasks, percent and tag indexStatus.
• jobs_statistics: New
• jobLogs: Renaming of arguments, adding jobExecutionTime.
• sppmon_metrics: Added influxdb_version and new arguments
• vmReplicateSummary and vmReplicateStats: removed tag messageId`
• vadps: Moved 3 tags to fields, adjusted CQ to run on distinct ID's
• ProcessStats: Removed 2 fields and 3 tags due change top to ps command.
• office365stats and office365transfBytes new
• df_ssh: Renamed avail to available, bugfixing a tag