Update readme and add a single sql file for new installations

Author: serkansbo

README.md

@@ -1,19 +1,21 @@
 # SBOanalytics
-Web site log analytics on a budget. 
+Web site and server analytics on a budget. 
 
 **Background and motivation**
 
-Running web sites on a budget is not too hard (especially for Apache, PHP, Mysql people like us) but monitoring web site activity is not so easy 
-(and obviously we do not want to spend any money for it).
+Running web sites on a budget is not too hard (especially for Apache, PHP, Mysql people like us) 
+but monitoring web site and server activity is not so easy (and obviously we do not want to spend any money for it).
 
 This tool is suitable for people running their own web sites on their own VMs, droplets etc. You must be able to run operating system commands on your server 
-to run the log processing component and you should know how to deploy PHP apps.
+to run the log processing component and you should know how to deploy PHP apps. 
+You can monitor web site and operating system metrics with minimal effort.
 
 ## Components
 
 ### Log processor 
-You need to run the log processor, https://github.com/SBOsoft/SBOLogProcessor, to collect web server access logs. 
-It will monitor your access logs, generate metrics and optionally save logs to its database (for now only mysql is supported).
+You need to run the log processor, https://github.com/SBOsoft/SBOLogProcessor, 
+to collect web server access logs, web site metrics and operating system metrics. 
+It will monitor your access logs, generate metrics and optionally save logs to a database (for now only mysql is supported).
 
 ### SBOanalytics (this application), the frontend
 Use this application to view metrics and logs collected by the log processor. Data generated by the log processor will be saved in a mysql database.
@@ -45,18 +47,28 @@ To set up the containers, change into the containers folder and run `docker-comp
 
 ### Production deployments
 
-  - Go to https://github.com/SBOsoft/SBOanalytics/releases and download a release package, i.e SBOanalytics-xxxx.xx.xx.xxx.zip. 
-  - Copy the zip file into a new folder and unzip it
-  - Review the .htaccess file in the package. The .htaccess file contains multiple SetEnv directives which are commented out by default.
+  1. Go to https://github.com/SBOsoft/SBOanalytics/releases and download a release package, i.e SBOanalytics-xxxx.xx.xx.xxx.zip. 
+  2. Copy the zip file into a new folder and unzip it
+  3. Review the .htaccess file in the package. The .htaccess file contains multiple SetEnv directives which are commented out by default.
 Environment variables are used for application configuration. You can either set them in the .htaccess file or in your VirtualHost configuration.
 Setting them up in VirtualHost configuration will allow for easier updates. If you set them in the .htaccess file then  during upgrades 
 you will need to migrate your changes to the .htaccess file manually.
-  - Just placing the folder to a path under your php enabled virtual host will work
-  - To upgrade, download a release package, extract into a new folder and replace your existing deployment folder with the new one after 
+  4. Just placing the folder to a path under a php enabled virtual host will work
+  5. To upgrade, download a release package, extract into a new folder and replace your existing deployment folder with the new one after 
 migrating the changes in your existing .htacess file. Do NOT replace the new .htaccess with your existing .htaccess file as there maybe changes
 required for the new version to function properly.
 
+#### Database setup for new installations
+  1. Review db/db-for-new-installations.sql file in the package. (Ignore db/upgrades folder for new installations)
+  2. Create a new database for SBOanalytics (or use an existing database, all tables have sbo_ prefix)
+  3. Run db/db-for-new-installations.sql 
 
+#### Database updates for upgrades 
+  - To apply database updates, review the files in db/upgrades/ folder in the release package. Run .sql files with names after your old version.
+For example if your old version was 2025.07.25.0001 you need to run all files with names after 20250725*.sql in db/upgrades folder.
+You MUST run sql files in alphabetical order.
+
+#### .htaccess and environment variables
 Please review webapp/DocumentRoot/.htaccess before deployments and upgrades. You can define the required environment variables in VirtualHost settings,
 e.g /etc/apache2/sites-enabled/yoursite.conf, and leave .htaccess as is. 
 If you set environment variables in the .htaccess file then you will need to manually apply your changes during upgrades. 
@@ -74,7 +86,8 @@ defined by SBO_AUTH_SINGLE_USER and SBO_AUTH_SINGLE_PWD environment variables.
   - SBO_DB_PASSWORD mysql password
   - SBO_AUTH_SINGLE_USER username for SBO_AUTH_TYPE single
   - SBO_AUTH_SINGLE_PWD password for SBO_AUTH_TYPE single
+  - SBO_ADMIN_SECRET secret required for accessing the settings page
 
-Always review https://github.com/SBOsoft/SBOanalytics/blob/main/webapp/DocumentRoot/.htaccess for latest updates to configuration options.
-The above list may not be exhaustive.
+Always review https://github.com/SBOsoft/SBOanalytics/blob/main/webapp/DocumentRoot/.htaccess for latest updates to configuration options
+and additional information.
 

containers/docker-compose.yml

@@ -31,7 +31,7 @@ services:
     volumes:
       - ./mysql/my.cnf:/etc/mysql/my.cnf   # Persistent storage for MySQL data
       - ./mysql_data:/var/lib/mysql   # Persistent storage for MySQL data
-      - ../db/:/docker-entrypoint-initdb.d  #init scripts
+      - ../db/upgrades/:/docker-entrypoint-initdb.d  #init scripts
     #restart: no                     # change to always or unless-stopped for non-dev envs
     networks:
       - sboanalytics-network

db/db-for-new-installations.sql

@@ -0,0 +1,124 @@
+/*
+--Run the following if you didn't already create a database
+
+CREATE DATABASE sboanalytics 
+    CHARACTER SET utf8mb4
+    COLLATE utf8mb4_unicode_ci;
+
+USE sboanalytics;
+*/
+
+
+CREATE TABLE sbo_domains (
+    domain_id INT AUTO_INCREMENT,
+    domain_name VARCHAR(255) NOT NULL,
+    description VARCHAR(255) null,
+    created DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    PRIMARY KEY (domain_name),
+    UNIQUE KEY sbo_domains_id (domain_id)
+) ENGINE=InnoDB CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
+
+
+CREATE TABLE sbo_users (
+    user_id INT AUTO_INCREMENT,
+    email VARCHAR(255) NOT NULL,
+    password_hash VARCHAR(255) NOT NULL,
+    created DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    PRIMARY KEY (email),
+    UNIQUE KEY sbo_users_id (user_id)
+) ENGINE=InnoDB CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
+
+
+CREATE TABLE sbo_log_files (
+    file_id int not null AUTO_INCREMENT,
+    domain_id int not null,
+    host_name VARCHAR(100) NOT NULL,
+    file_path VARCHAR(500) NOT NULL,
+    created DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    PRIMARY KEY (domain_id, host_name, file_path),
+    UNIQUE KEY sbo_log_files_file_id (file_id)
+) ENGINE=InnoDB CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
+
+
+CREATE TABLE sbo_metrics (
+    domain_id int not null,
+    metric_type int not null,
+    key_value varchar(100) not null,
+    time_window bigint not null,
+    metric_value bigint not null,
+    created DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    PRIMARY KEY (domain_id, metric_type, time_window, key_value),
+    KEY sbo_metrics_time (domain_id, time_window, metric_type, key_value, metric_value)
+) ENGINE=InnoDB CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
+
+
+
+CREATE TABLE sbo_rawlogs (
+    domain_id int not null,
+    host_id int not null,
+    request_ts datetime not null,
+    client_ip varbinary(16) default null,
+    remote_user varchar(100) not null,
+    http_method varchar(20) not null,
+    path3 varchar(100) not null,
+    request_uri varchar(100) not null,
+    http_status int not null,
+    bytes_sent int not null,
+    referer varchar(100) default null,
+    is_malicious tinyint not null default 0,
+    ua_string varchar(100) default null,
+    ua_os varchar(20) default null,
+    ua_family varchar(20) default null,
+    ua_device_type varchar(20) default null,
+    ua_is_human varchar(20) default null,
+    ua_intent varchar(20) default null,
+
+    KEY sbo_rawlogs_method (domain_id, request_ts, http_method),
+    KEY sbo_rawlogs_status (domain_id, request_ts, http_status),
+    KEY sbo_rawlogs_referer (domain_id, request_ts, referer),
+    KEY sbo_rawlogs_ua_os (domain_id, request_ts, ua_os),
+    KEY sbo_rawlogs_ua_family (domain_id, request_ts, ua_family),
+    KEY sbo_rawlogs_ua_device_type (domain_id, request_ts, ua_device_type),
+    KEY sbo_rawlogs_ua_is_human (domain_id, request_ts, ua_is_human),
+    KEY sbo_rawlogs_ua_intent (domain_id, request_ts, ua_intent),
+    KEY sbo_rawlogs_client_ip (domain_id, request_ts, client_ip),
+    KEY sbo_rawlogs_path3 (domain_id, request_ts, path3)
+) ENGINE=InnoDB CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
+
+
+ALTER TABLE sbo_domains ADD COLUMN timeWindowSizeMinutes int not null default 10;
+
+
+
+
+CREATE TABLE sbo_os_metrics(
+    host_id int not null,
+    metrics_ts datetime not null,
+    up_duration_minutes int not null default 0,
+    users int not null default 0,
+    load_average1 double not null default 0,
+    load_average5 double not null default 0,
+    load_average15 double not null default 0,
+
+    swap_use bigint not null default 0,
+    cache_use bigint not null default 0,
+    memory_use bigint not null default 0,
+    memory_free bigint not null default 0,
+
+    PRIMARY KEY sbo_rawlogs_method (host_id, metrics_ts)
+) ENGINE=InnoDB CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
+
+
+CREATE TABLE sbo_hosts(
+    host_id int not null AUTO_INCREMENT,
+    host_name VARCHAR(255) NOT NULL,
+    description VARCHAR(255) null,
+    created DATETIME NOT NULL DEFAULT CURRENT_TIMESTAMP,
+    PRIMARY KEY (host_id),
+    UNIQUE KEY sbo_hosts_name (host_name)
+) ENGINE=InnoDB CHARACTER SET utf8mb4 COLLATE utf8mb4_unicode_ci;
+
+
+ALTER TABLE sbo_os_metrics ADD COLUMN memory_available bigint not null default 0;
+
+/* end of 2025.08.10 up to this point */

db/202506040000-create-db.sql db/upgrades/202506040000-create-db.sqldb/202506040000-create-db.sql renamed to db/upgrades/202506040000-create-db.sql

@@ -48,6 +48,9 @@ RewriteCond %{REQUEST_URI} /version.txt$
 RewriteRule version.txt   - [R=404,L]
 
 
+#return 404 for all .sql files
+RewriteRule \.sql$ - [R=404,NC,L]
+
 #You can set environment variables here or in virtual host config
 
 #Authentication type. REQUIRED. Supported values: none, single (more to be added later)