Merge pull request #2849 from intelowlproject/develop

v6.4.0

Author: drosetti

.github/CHANGELOG.md

@@ -2,6 +2,22 @@
 
 [**Upgrade Guide**](https://intelowlproject.github.io/docs/IntelOwl/installation/#update-to-the-most-recent-version)
 
+## [v6.4.0](https://github.com/intelowlproject/IntelOwl/releases/tag/v6.4.0)
+This release mostly provides important changes in the backend part that will be supported in the UI in the next releases.
+* Analyzable: Representation of an observable or a sample: every job is linked to the scanned analyzable.
+* Data models: A new system to normalize the output of analyzers (already available in the job raw section). It allows analyzers to specify the evaluation, reliability and many others information about the analyzable. 
+* User Event: Users can create reports for analyzables: indicating additional information or a custom evaluation.
+* Engine: Evaluate user reports and analyzers's data to assign a score to the job.
+
+As usual, we add new plugins. This release brings the following new ones:
+* [Nuclei](https://github.com/projectdiscovery/nuclei): A modern, high-performance vulnerability scanner that leverages simple YAML-based templates.
+* [ipquery](https://ipquery.io/): a flexible api for vpn detection, geolocation, and threat intelligence.
+* [mullvad](https://mullvad.net/en): VPN provider.
+* [spamhaus](https://www.spamhaus.org/): Updated existing analyzer to support IPv6.
+* [bbot](https://github.com/blacklanternsecurity/bbot): multipurpose scanner.
+* [debloat](https://github.com/Squiblydoo/debloat): remove excess garbage from bloated executables.
+
+
 ## [v6.3.1](https://github.com/intelowlproject/IntelOwl/releases/tag/v6.3.1)
 This release provides fixes to the recent added ARM support. (ARM build for v6.3.0 was broken due to some dependencies)
 

.github/dependabot.yml

@@ -56,6 +56,18 @@ updates:
       - dependency-name: "*"
         update-types: [ "version-update:semver-patch" ]
 
+  - package-ecosystem: "pip"
+    directory: "/integrations/nuclei_analyzer"
+    schedule:
+      interval: "weekly"
+      day: "tuesday"
+    target-branch: "develop"
+    ignore:
+      # ignore all patch updates since we are using ~=
+      # this does not work for security updates
+      - dependency-name: "*"
+        update-types: [ "version-update:semver-patch" ]
+
   - package-ecosystem: "pip"
     directory: "/integrations/phishing_analyzers"
     schedule:
@@ -137,6 +149,18 @@ updates:
       - dependency-name: "*"
         update-types: ["version-update:semver-patch"]
 
+  - package-ecosystem: "docker"
+    directory: "/integrations/nuclei_analyzer"
+    schedule:
+      interval: "weekly"
+      day: "tuesday"
+    target-branch: "develop"
+    ignore:
+      # ignore all patch updates since we are using ~=
+      # this does not work for security updates
+      - dependency-name: "*"
+        update-types: ["version-update:semver-patch"]
+
   - package-ecosystem: "docker"
     directory: "/integrations/malware_tools_analyzers"
     schedule:

.github/pull_request_template.md

@@ -25,11 +25,13 @@ Please delete options that are not relevant.
     - [ ] If you created a new analyzer and it is free (does not require any API key), please add it in the `FREE_TO_USE_ANALYZERS` playbook by following [this guide](https://intelowlproject.github.io/docs/IntelOwl/contribute/#how-to-modify-a-plugin).
     - [ ] Check if it could make sense to add that analyzer/connector to other [freely available playbooks](https://intelowlproject.github.io/docs/IntelOwl/usage/#list-of-pre-built-playbooks).
     - [ ] I have provided the resulting raw JSON of a finished analysis and a screenshot of the results.
-    - [ ] If the plugin interacts with an external service, I have created an attribute called precisely `url` that contains this information. This is required for Health Checks. 
+    - [ ] If the plugin interacts with an external service, I have created an attribute called precisely `url` that contains this information. This is required for Health Checks (HEAD HTTP requests). 
     - [ ] If the plugin requires mocked testing, `_monkeypatch()` was used in its class to apply the necessary decorators.
     - [ ] I have added that raw JSON sample to the `MockUpResponse` of the `_monkeypatch()` method. This serves us to provide a valid sample for testing.
+    - [ ] I have created the corresponding `DataModel` for the new analyzer following the [documentation](https://intelowlproject.github.io/docs/IntelOwl/contribute/#how-to-create-a-datamodel)
 - [ ] I have inserted the copyright banner at the start of the file: ```# This file is a part of IntelOwl https://github.com/intelowlproject/IntelOwl # See the file 'LICENSE' for copying permission.```
-- [ ] If external libraries/packages with restrictive licenses were used, they were added in the [Legal Notice](https://github.com/certego/IntelOwl/blob/master/.github/legal_notice.md) section.
+- [ ] Please avoid adding new libraries as requirements whenever it is possible. Use new libraries only if strictly needed to solve the issue you are working for. In case of doubt, ask a maintainer permission to use a specific library.
+- [ ] If external libraries/packages with restrictive licenses were added, they were added in the [Legal Notice](https://github.com/certego/IntelOwl/blob/master/.github/legal_notice.md) section.
 - [ ] Linters (`Black`, `Flake`, `Isort`) gave 0 errors. If you have correctly installed [pre-commit](https://intelowlproject.github.io/docs/IntelOwl/contribute/#how-to-start-setup-project-and-development-instance), it does these checks and adjustments on your behalf.
 - [ ] I have added tests for the feature/bug I solved (see `tests` folder). All the tests (new and old ones) gave 0 errors.
 - [ ] If the GUI has been modified:

.github/workflows/pull_request_automation.yml

@@ -57,6 +57,9 @@ jobs:
       - name: isort
         run: |
           isort . --profile black --filter-files --check-only --diff --skip configuration/ldap_config.py
+      
+      - name: Perform ShellCheck Analysis
+        run: bash <(curl -s https://raw.githubusercontent.com/CICDToolbox/shellcheck/master/pipeline.sh)
 
   backend-tests:
     runs-on: ubuntu-latest
@@ -152,4 +155,4 @@ jobs:
       - name: Test with Jest
         run: |
           npm run test -- --silent --coverage
-        working-directory: ./frontend
+        working-directory: ./frontend

.github/workflows/stale.yml

@@ -19,8 +19,8 @@ jobs:
         stale-pr-label: "stale"
         exempt-pr-labels: "keep-open"
         # Issues
-        stale-issue-message: "This issue has been marked as stale because it has had no activity for 10 days. If you are still working on this, please provide some updates."
-        days-before-issue-stale: 10
+        stale-issue-message: "This issue has been marked as stale because it has had no activity for 30 days. If you are still working on this, please provide some updates."
+        days-before-issue-stale: 30
         days-before-issue-close: -1  # do not close Issues
         stale-issue-label: "stale"
         exempt-issue-labels: "keep-open"

.pre-commit-config.yaml

@@ -50,3 +50,8 @@ repos:
           - [email protected]
         args: ["--fix"]
         files: frontend/src/styles/.*(css|scss)$
+  - repo: https://github.com/koalaman/shellcheck-precommit
+    rev: v0.7.2
+    hooks:
+      - id: shellcheck
+        args: ["--severity=warning"] 

api_app/admin.py

@@ -49,10 +49,8 @@ class JobAdminView(CustomAdminView):
         "id",
         "status",
         "user",
-        "observable_name",
-        "observable_classification",
-        "file_name",
-        "file_mimetype",
+        "get_analyzable_name",
+        "get_analyzable_classification",
         "received_request_time",
         "analyzers_executed",
         "connectors_executed",
@@ -64,13 +62,16 @@ class JobAdminView(CustomAdminView):
         "user",
         "status",
     )
-    search_fields = (
-        "md5",
-        "observable_name",
-        "file_name",
-    )
     list_filter = ("status", "user", "tags")
 
+    @admin.display(description="Name")
+    def get_analyzable_name(self, instance):
+        return instance.analyzable.name
+
+    @admin.display(description="Classification")
+    def get_analyzable_classification(self, instance):
+        return instance.analyzable.classification
+
     @staticmethod
     def has_add_permission(request: HttpRequest) -> bool:
         return False
@@ -122,8 +123,6 @@ class PluginConfigAdminView(ModelWithOwnershipAdminView, CustomAdminView):
         "pk",
         "get_config",
         "parameter",
-        "for_organization",
-        "get_owner",
         "get_type",
         "value",
     ) + ModelWithOwnershipAdminView.list_display

api_app/analyzables_manager/__init__.py

@@ -0,0 +1,11 @@
+from django.contrib import admin
+
+from api_app.analyzables_manager.models import Analyzable
+
+
+@admin.register(Analyzable)
+class AnalyzableAdmin(admin.ModelAdmin):
+    list_display = ["pk", "name", "sha1", "sha256", "md5"]
+    search_fields = ["name", "sha1", "sha256", "md5"]
+    ordering = ["name"]
+    list_filter = ["discovery_date"]

api_app/analyzables_manager/admin.py

@@ -0,0 +1,12 @@
+# This file is a part of IntelOwl https://github.com/intelowlproject/IntelOwl
+# See the file 'LICENSE' for copying permission.
+
+from django.apps import AppConfig
+
+
+class AnalyzablesManagerConfig(AppConfig):
+    name = "api_app.analyzables_manager"
+
+    @staticmethod
+    def ready() -> None:
+        from . import signals  # noqa