Merge branch 'main' of https://github.com/CactuseSecurity/firewall-orchestrator

Author: SolidProgramming

.vscode/launch.json

@@ -82,7 +82,7 @@
             "name": "py-normalizeNwData",
             "type": "debugpy",
             "request": "launch",
-            "program": "${workspaceFolder}/scripts/customizing/modelling/convertNwObjDataExample.py",
+            "program": "${workspaceFolder}/scripts/customizing/modelling/convertNwObjDataFromGitPlain.py",
             "console": "integratedTerminal",
             "env": {
                 "PYTHONPATH": "${PYTHONPATH}:${workspaceRoot}"
@@ -101,15 +101,14 @@
                 "PYTHONPATH": "${PYTHONPATH}:${workspaceRoot}"
             },
             "args": [
-                "-m23",
+                "-m6",
                 "-d1",
                 "-f",
                 "-s",
-                //"-l66",
+                //"-l",
                 // "-c"
                 //"-l250"
                 //  41 - lab fortimanager
-                //"-nhttps://fwodemodata.cactus.de/demo07_dummyGw1.json"
                 //"-ihttps://fwodemodata.cactus.de/demo04_cpr8x.json",
                 //"-ihttps://fwodemodata.cactus.de/demo01_fortiMgrLab.json"
                 //"-ihttps://fwodemodata.cactus.de/demo05_fortiMgr2.json"

CODING_GUIDELINES.md

@@ -27,3 +27,72 @@
 ## C# specific
 - avoid null references
 - methods should be preceded with standard comment header (///)
+
+## Conventional Commits
+The Conventional Commits specification is a lightweight convention on top of commit messages. It provides an easy set of rules for creating an explicit commit history; which makes it easier to write automated tools on top of
+
+### Why Use Conventional Commits
+- Automatically generating CHANGELOGs.
+- Automatically determining a semantic version bump (based on the types of commits landed).
+- Communicating the nature of changes to teammates, the public, and other stakeholders.
+- Triggering build and publish processes.
+- Making it easier for people to contribute to your projects, by allowing them to explore a more structured commit history.
+
+The commit message should be structured as follows:
+```
+<type>[optional scope]: <description>
+[optional body]
+[optional footer(s)]
+```
+
+The commit contains the following structural elements, to communicate intent to the consumers:
+
+- fix: a commit of the type fix patches a bug in your codebase (this correlates with PATCH in Semantic Versioning).
+- feat: a commit of the type feat introduces a new feature to the codebase (this correlates with MINOR in Semantic Versioning).
+- BREAKING CHANGE: a commit that has a footer BREAKING CHANGE:, or appends a ! after the type/scope, introduces a breaking API change (correlating with MAJOR in Semantic Versioning). A BREAKING CHANGE can be part of commits of any type.
+- types other than fix: and feat: are allowed, for example @commitlint/config-conventional (based on the Angular convention) recommends build:, chore:, ci:, docs:, style:, refactor:, perf:, test:, and others.
+- footers other than BREAKING CHANGE: <description> may be provided and follow a convention similar to git trailer format.
+
+### Examples
+Commit message with description and breaking change footer
+```
+feat: allow provided config object to extend other configs
+
+BREAKING CHANGE: `extends` key in config file is now used for extending other config files
+```
+
+Commit message with ! to draw attention to breaking change
+```
+feat!: send an email to the customer when a product is shipped
+```
+
+Commit message with scope and ! to draw attention to breaking change
+```
+feat(api)!: send an email to the customer when a product is shipped
+```
+
+Commit message with ! to draw attention to breaking change
+```
+chore!: drop support for Node 6
+
+BREAKING CHANGE: use JavaScript features not available in Node 6.
+```
+
+Commit message with no body
+```
+docs: correct spelling of CHANGELOG
+```
+
+Commit message with multi-paragraph body and multiple footers
+```
+fix: prevent racing of requests
+
+Introduce a request id and a reference to latest request. Dismiss
+incoming responses other than from latest request.
+
+Remove timeouts which were used to mitigate the racing issue but are
+obsolete now.
+
+Reviewed-by: Z
+Refs: #123
+```

CODING_GUIDELINES_FRONTEND.md

@@ -0,0 +1,16 @@
+
+# Code Guidelines
+
+
+## General
+- Close your tag - Leaving some tags open is simply a bad practice. Only self-closing tags are valid. Normal elements can never have self-closing tags.
+- Don't use inline styles(if possible) - When creating your markup, do not use inline styling because it would be very hard to override these styles in case you need to.
+- Try not to use "!important" - Using the !important declaration is often considered bad practice because it has side effects that mess with one of CSS's core mechanisms: specificity. In many cases, using it could indicate poor CSS architecture.
+
+## Components
+- Organize files and components in a folder structure like this. This makes it easy to find the code related to a page, without having to browse the entire file explorer. Try, as much as possible, to respect the SOLID principles. Mainly by creating autonomous and extensible components: inject the smallest possible service or parameter, manage all the possibilities offered by the component. For example, a data modification page should display the data, check their values and save the data at the end of the process.
+
+## UI
+# Responsiveness
+- Use the bootstrap grid and it's column classes to have easy and responsive design. [Bootstrap](https://getbootstrap.com/docs/5.3/layout/columns/)
+- Decide if you want to develop mobile or desktop design first and test respectively.

documentation/certificates.md

@@ -0,0 +1,27 @@
+# Certificates in FWO
+
+The expected paths for keys and certificates are /etc/apache2/ssl/server.key and /etc/apache2/ssl/server.crt respectivly. If you want to change them, use these names and paths. Make sure server.key has these permissions
+
+```
+-rw-r----- 1 root root
+```
+
+After the change restart apache2
+
+```
+ sudo systemctl restart apache2
+```
+
+## Change Root Certificate
+
+Copy root cert to
+
+```
+/usr/local/share/ca-certificates/
+```
+
+and update
+
+```
+sudo update-ca-certificates
+```

documentation/developer-docs/importer/FWO-import-api.md

@@ -140,7 +140,8 @@ here we describe a single rule:
     "rule_head_text": null,                                 // string: for section headers this is the field to use
     "rule_from_zone": null,                                 // string: source zone (if applicable) of the rule
     "rule_to_zone": null,                                   // string: destination zone (if applicable) of the rule
-    "rule_type": "access"                                   // string: type of the nat rule: "access|combined|original|xlate", default "access"
+    "rule_type": "access",                                  // string: type of the nat rule: "access|combined|original|xlate", default "access"
+    "rule_custom_fields": "{\"field1\": \"value1\"}"        // string: json serialized user defined fields
 }
 ```
 - rule_track can be any of log, none, alert, userdefined, mail, account, userdefined 1, userdefined 2, userdefined 3, snmptrap, log count, count, log alert, log alert count, log alert count alarm, log count alarm, count alarm, all, all start, utm, utm start, network log

documentation/installer/install-advanced.md

@@ -1,6 +1,6 @@
 # Advanced installation options
 
-always change into the firewwall-orchestrator directory before starting the installation!
+always change into the firewwall-orchestrator directory before starting the installation.
 
 ## Install parameters
 
@@ -38,7 +38,7 @@ ansible-playbook site.yml -K
 ### Installation behind a proxy (no direct Internet connection)
 
 By default, during installation or upgrade the proxy settings are read from the OS environment of the installer host.
-For example you may have a global system-wide config file /etc/profile.d/proxy.sh with the following content:
+For example you may either use /etc/environment or add a global system-wide config file /etc/profile.d/proxy.sh and add the following content:
 
 ```console
 export http_proxy=http://proxy.int:3128
@@ -49,7 +49,7 @@ export no_proxy=127.0.0.1,localhost
 Also make sure that your proxy is configured in your .gitconfig to be able to do the initial repo cloning.
 See https://gist.github.com/evantoli/f8c23a37eb3558ab8765.
 
-If instead you need to individually set a proxy before installation/upgrade, use the following comamnds in your terminal:
+If instead you need to individually set a proxy before installation/upgrade, use the following commands in your terminal:
 ```console
 export http_proxy=http://proxy.int:3128
 export https_proxy=http://proxy.int:3128
@@ -75,6 +75,8 @@ Note that the following domains must be reachable through the proxy:
     postgresql.org
     microsoft.com     
     nuget.org
+    storage.googleapis.com
+    googlechromelabs.github.io
 
   Only for the initial setup of python venv
 
@@ -88,6 +90,8 @@ NB: for vscode-debugging, you also need access to
     visualstudio.com
 
 
+#### Pyhton proxy config
+
 Remember if your server resides behind a proxy that you will have to set the proxy for pip as follows before installing ansible:
 
          pip config set global.proxy http://proxy:3128
@@ -97,6 +101,16 @@ In case of timeout issues (you might be behind a security proxy that does intens
 
           pip --default-timeout=3600 install ansible
           
+##### issues with existing pip config
+
+In case of errors with existing pip config, do not use the script to create the venv but proceed as follows:
+
+remove any local pip config and install manually:
+    
+    rm -f $HOME/.config/pip/pip.conf
+    python3 -m venv ansible-venv
+    source ansible-venv/bin/activate
+    pip install ansible
 
 ### Parameter "api_no_metadata" to prevent meta data import
 

documentation/installer/install-for-testing.md

@@ -12,7 +12,7 @@ This includes:
 Note: the relevant secrets are displayed at the very end of the installation. They can also be found in the etc/secrets directory.
 
 ```console
-ansible-playbook/ site.yml -e "testkeys=yes" -K
+ansible-playbook site.yml -e "testkeys=yes" -K
 ```
 
 A static jwt key helps with debugging c# code in visual studio (code) - you can use a static backend (ldap & api) with these keys.

documentation/revision-history-develop.md

@@ -217,3 +217,40 @@ bugfix release:
 # 8.2.4 - 19.06.2024 DEVELOP
 - owner-filtering for new report type
 - new setting for email recipients
+
+# 8.3.1 - 08.07.2024 DEVELOP
+- workflow: external state handling
+- fix config value
+- remove uniqueness of owner names
+
+# 8.3.2 - 09.09.2024 DEVELOP
+- Added welcome message and settings
+
+# 8.4.1 - 15.10.24 DEVELOP
+- Add missing FK connection.proposed_app_id #2591
+
+# 8.4.2 - 17.10.2024 DEVELOP
+- external request
+
+# 8.4.3 - 05.11.2024 DEVELOP
+- extra parameters in modelling connection
+
+# 8.5.1 - 18.11.2024 DEVELOP
+- reporting - fixing PDF generation on various platforms
+- modelling - fixing AR editing: strict prevention of all area mixing
+
+# 8.5.2 - 27.11.2024 DEVELOP
+- some check point importer fixes
+  - 4 new colors
+  - added Internet object
+  - added voip one more object
+
+# 8.5.3 - 27.11.2024 DEVELOP
+- owner import - make ldap selectable (internal/external)
+- small fixes regarding missing config data for two schedulers (daily, app data import)
+
+# 8.5.4 - 04.12.2024 DEVELOP
+- external request: introduce wait cycles
+
+# 8.6.1 - 12.12.2024 DEVELOP
+- external request: introduce locks

documentation/revision-history-main.md

@@ -395,3 +395,87 @@ Maintenance release
 # 8.3.1 - 14.08.24 MAIN
 Hotfix:
 - in CheckPoint importer: fix missing group members
+
+# 8.4 - 30.09.24 MAIN
+Stability release
+- various small bug fixes
+  - installer (redundant code deleting test user)
+  - importer (switching from full details to standard, re-adding VSX gateway support, voip domain handling in cp parser)
+  - reporting (app-rule report containing multiple objects)
+  - middleware (config subscriptions)
+  - reporting (temporarily highlight linked to object in rsb)
+  - modelling (sync connections - not always part of overview table after creation)
+  - RBA (role picking when user has multiple roles)
+  - UI various: adding missing pager control
+  - UI various: spinner clean-up
+- features/upgrades
+  - Added login page welcome message and settings
+  - Added last hit information in app-rule report
+  - API - upgrading to 2.43.0
+  - various security upgrades dotnet (restsharp, jwt, ...)
+
+# 8.4.1 - 30.10.24 MAIN
+Network Modelling feature update
+- import of app server IP addresses via CSV upload
+- import of multiple sources for area IP data 
+- new option email notification: fall-back to main owner if group is empty
+Fixes
+- corrections in displaying UI messages
+- converting owner network ip data to standard format "range"
+- importer 
+  - check point - fix import of all VSX instances
+  - fortinet - add hit counts and install on information
+
+# 8.5 - 13.11.24 MAIN
+Network Modelling feature update
+- modelling can be requested as firewall change via external ticketing tool
+- includes all approle handling
+- simple form of rule change request (always request all connections as rules)
+- api hasura upgrade to 2.44.0
+Fixes
+- various small UI fixes
+- importer (CP: handle None objects)
+
+# 8.6 - 11.12.2024 MAIN
+Features
+- Modelling 
+  - Create Application Zones
+  - Add monitoring for external requests for admins 
+  - Add re-initialization for external requests
+  - consolidation modelling external requests
+  - adding optional access requst on behalf of UI user
+  - adding live update of external task/ticket status
+  - app server name handling rework (NONAME --> <prefix>_<IP address>)
+  - owner groups can now also be external LDAP groups
+
+- Reporting
+  - refining connection report (adding Common service, app role, network area details)
+Fixes
+- Importer
+  - adding missing colors in Check Point importer
+  - new VOIP service object and Internet object
+
+- UI
+  - SECURITY: updating System.Text.Encodings.Web v4.5.0 --> v8.0.0
+
+# 8.6.1 17.12.2024 MAIN
+Fixes network modelling
+- lock external requests to avoid multiple external tickets
+- fix missing comments
+- wait cycles for access request after group changes
+- save publish flag at interface creation
+- disregard dummyAppRole for status determination
+- inherit extra configs from interface
+- sanitize extra configs
+- sort tasks for connection Id and show already adapted name of new members
+- small monitoring adaptations
+- some cleanup + removal of compiler warnings
+- fix ldap group creation regression
+- restrict owner_network uniqness constraint to same import source
+- UI interface search pop-up transformed into filterable table
+
+Upgrade Hasura API to v2.45.1
+
+# 8.6.2 03.01.2025 MAIN
+Hotfix for network modelling:
+- fix: when visiting the library for the second time, app servers were missing due to uninitialized area data.

inventory/group_vars/all.yml

@@ -1,5 +1,5 @@
 ### general settings
-product_version: "8.3.1"
+product_version: "8.6.2"
 ansible_user: "{{ lookup('env', 'USER') }}"
 ansible_become_method: sudo
 ansible_python_interpreter: /usr/bin/python3