[feature] Estimated location: Setup and Management #1034

Closes #1034

Signed-off-by: DragnEmperor <[email protected]>

Author: DragnEmperor

docs/index.rst

@@ -49,6 +49,7 @@ the OpenWISP architecture.
     user/openvpn.rst
     user/subnet-division-rules.rst
     user/whois.rst
+    user/estimated-location.rst
     user/rest-api.rst
     user/settings.rst
 

docs/user/estimated-location.rst

@@ -0,0 +1,56 @@
+Estimated Location
+==================
+
+.. important::
+
+    The **Estimated Location** feature is **disabled by default**.
+
+    Before enabling it, the :doc:`WHOIS Lookup feature <whois>` must be
+    enabled. Then set
+    :ref:`OPENWISP_CONTROLLER_ESTIMATED_LOCATION_ENABLED` to ``True``
+
+.. contents:: **Table of contents**:
+    :depth: 1
+    :local:
+
+Overview
+--------
+
+The Estimated Location feature automatically creates or updates a device’s
+location based on latitude and longitude information retrieved from the
+WHOIS Lookup feature.
+
+Trigger Conditions
+------------------
+
+Estimated Location is triggered when:
+
+- A **fresh WHOIS lookup** is performed for a device.
+- Or when a WHOIS record already exists for the device’s IP **and**:
+
+  - The device’s last IP address is **public**.
+  - WHOIS lookup and Estimated Location is **enabled** for the device’s
+    organization.
+
+Behavior
+--------
+
+The system will **attach the already existing matching location** of
+another device with same ip to the current device if:
+
+- Only one device is found with that IP and it has a location.
+- The current device **has no location** or that location is
+  **estimated**.
+
+If there are multiple devices with location for the same IP, the system
+will **not attach any location** to the current device and a notification
+will be sent suggesting the user to manually assign/create a location for
+the device.
+
+If there is **no matching location**, a new estimated location is created
+or the existing one is updated using coordinates from the WHOIS record,
+but only if the existing location is estimated.
+
+If two devices share the same IP address and are assigned to the same
+location, and the last IP of one of the devices is updated, the system
+will create a new estimated location for that device.

docs/user/settings.rst

@@ -795,3 +795,20 @@ Maxmind Account ID required for the :doc:`WHOIS Lookup feature <whois>`.
 ============ =======
 
 Maxmind License Key required for the :doc:`WHOIS Lookup feature <whois>`.
+
+.. _openwisp_controller_whois_estimated_location_enabled:
+
+``OPENWISP_CONTROLLER_WHOIS_ESTIMATED_LOCATION_ENABLED``
+--------------------------------------------------------
+
+============ =========
+**type**:    ``bool``
+**default**: ``False``
+============ =========
+
+Allows enabling the optional :doc:`Estimated Location feature
+<estimated-location>`.
+
+.. image:: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/estimated-location-setting.png
+    :target: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/estimated-location-setting.png
+    :alt: Estimated Location setting

docs/user/whois.rst

@@ -40,25 +40,23 @@ A WHOIS lookup is triggered automatically when:
 However, the lookup will only run if **all** the following conditions are
 met:
 
+- The device is either **newly created** or has a **changed last IP**.
 - The device's last IP address is **public**.
 - There is **no existing WHOIS record** for that IP.
 - WHOIS lookup is **enabled** for the device's organization.
 
-Behavior with Shared IP Addresses
----------------------------------
+Managing WHOIS Records
+----------------------
 
-If multiple devices share the same public IP address and one of them
-switches to a different IP, the following occurs:
-
-- A lookup is triggered for the **new IP**.
-- The WHOIS record for the **old IP** is deleted.
-- The next time a device still using the old IP fetches its checksum, a
-  new lookup is triggered, ensuring up-to-date data.
+If a device updates its last IP address, lookup is triggered for the **new
+IP** and the **WHOIS record for the old IP** is deleted if no active
+devices are associated with that IP address.
 
 .. note::
 
     When a device with an associated WHOIS record is deleted, its WHOIS
-    record is automatically removed.
+    record is automatically removed only if no active devices are
+    associated with that IP address.
 
 .. _controller_setup_whois_lookup:
 
@@ -79,6 +77,26 @@ Setup Instructions
    - Set :ref:`OPENWISP_CONTROLLER_WHOIS_GEOIP_ACCOUNT` to **Account ID**.
    - Set :ref:`OPENWISP_CONTROLLER_WHOIS_GEOIP_KEY` to **License Key**.
 
+6. Restart the application/containers if using ansible-openwisp2 or
+   docker.
+7. Run the ``clear_last_ip`` management command to clear the last IP
+   address of **all active devices across organizations**.
+
+   - If using ansible-openwisp2 (default directory is /opt/openwisp2,
+     unless changed in Ansible playbook configuration):
+
+     .. code-block:: bash
+
+         source /opt/openwisp2/env/bin/activate
+         python /opt/openwisp2/src/manage.py clear_last_ip
+
+   - If using docker:
+
+     .. code-block:: bash
+
+         docker exec -it <openwisp_container_name> sh
+         python manage.py clear_last_ip
+
 Viewing WHOIS Lookup Data
 -------------------------
 

openwisp_controller/config/admin.py

@@ -1384,7 +1384,7 @@ def get_fields(self, request, obj=None):
         if app_settings.REGISTRATION_ENABLED:
             fields += ["registration_enabled", "shared_secret"]
         if app_settings.WHOIS_CONFIGURED:
-            fields += ["whois_enabled"]
+            fields += ["whois_enabled", "estimated_location_enabled"]
         fields += ["context"]
         return fields
 

openwisp_controller/config/base/device.py

@@ -287,7 +287,7 @@ def save(self, *args, **kwargs):
         state_adding = self._state.adding
         super().save(*args, **kwargs)
         if app_settings.WHOIS_CONFIGURED:
-            self._check_last_ip()
+            self._check_last_ip(creating=state_adding)
         if state_adding and self.group and self.group.templates.exists():
             self.create_default_config()
         # The value of "self._state.adding" will always be "False"
@@ -510,9 +510,13 @@ def whois_service(self):
         """
         return WHOISService(self)
 
-    def _check_last_ip(self):
-        """Trigger WHOIS lookup if last_ip is not deferred."""
+    def _check_last_ip(self, creating=False):
+        """
+        Process details and location related to last_ip if last_ip has
+        changed or is being set for the first time.
+        """
         if self._initial_last_ip == models.DEFERRED:
             return
-        self.whois_service.trigger_whois_lookup()
+        if creating or self.last_ip != self._initial_last_ip:
+            self.whois_service.process_ip_data_and_location()
         self._initial_last_ip = self.last_ip

openwisp_controller/config/base/multitenancy.py

@@ -39,6 +39,11 @@ class AbstractOrganizationConfigSettings(UUIDModel):
         fallback=app_settings.WHOIS_ENABLED,
         verbose_name=_("WHOIS Enabled"),
     )
+    estimated_location_enabled = FallbackBooleanChoiceField(
+        help_text=_("Whether the estimated location feature is enabled"),
+        fallback=app_settings.ESTIMATED_LOCATION_ENABLED,
+        verbose_name=_("Estimated Location Enabled"),
+    )
     context = JSONField(
         blank=True,
         default=dict,
@@ -71,6 +76,15 @@ def clean(self):
                     )
                 }
             )
+        if not self.whois_enabled and self.estimated_location_enabled:
+            raise ValidationError(
+                {
+                    "estimated_location_enabled": _(
+                        "Estimated Location feature requires "
+                        "WHOIS Lookup feature to be enabled."
+                    )
+                }
+            )
         return super().clean()
 
     def save(

openwisp_controller/config/base/whois.py

@@ -1,10 +1,12 @@
 from ipaddress import ip_address, ip_network
 
+from django.contrib.gis.db.models import PointField
 from django.core.cache import cache
 from django.core.exceptions import ValidationError
 from django.db import models, transaction
 from django.utils.translation import gettext_lazy as _
 from jsonfield import JSONField
+from swapper import load_model
 
 from openwisp_utils.base import TimeStampedEditableModel
 
@@ -51,6 +53,12 @@ class AbstractWHOISInfo(TimeStampedEditableModel):
         blank=True,
         help_text=_("CIDR"),
     )
+    coordinates = PointField(
+        null=True,
+        blank=True,
+        help_text=_("Coordinates"),
+        srid=4326,
+    )
 
     class Meta:
         abstract = True
@@ -74,6 +82,20 @@ def clean(self):
                 raise ValidationError(
                     {"cidr": _("Invalid CIDR format: %(error)s") % {"error": str(e)}}
                 )
+
+        if self.coordinates:
+            if not (-90 <= self.coordinates.y <= 90):
+                raise ValidationError(
+                    {"coordinates": _("Latitude must be between -90 and 90 degrees.")}
+                )
+            if not (-180 <= self.coordinates.x <= 180):
+                raise ValidationError(
+                    {
+                        "coordinates": _(
+                            "Longitude must be between -180 and 180 degrees."
+                        )
+                    }
+                )
         return super().clean()
 
     @staticmethod
@@ -82,8 +104,18 @@ def device_whois_info_delete_handler(instance, **kwargs):
         Delete WHOIS information for a device when the last IP address is removed or
         when device is deleted.
         """
-        if instance._get_organization__config_settings().whois_enabled:
-            transaction.on_commit(lambda: delete_whois_record.delay(instance.last_ip))
+        Device = load_model("config", "Device")
+
+        last_ip = instance.last_ip
+        existing_devices = Device.objects.filter(_is_deactivated=False).filter(
+            last_ip=last_ip
+        )
+        if (
+            last_ip
+            and instance._get_organization__config_settings().whois_enabled
+            and not existing_devices.exists()
+        ):
+            transaction.on_commit(lambda: delete_whois_record.delay(last_ip))
 
     # this method is kept here instead of in OrganizationConfigSettings because
     # currently the caching is used only for WHOIS feature
@@ -113,3 +145,28 @@ def formatted_address(self):
                 ],
             )
         )
+
+    @property
+    def _location_name(self):
+        """
+        Used to get location name based on the address and IP.
+        """
+        address = self.formatted_address
+        if address:
+            parts = [part.strip() for part in address.split(",")[:2] if part.strip()]
+            location = ", ".join(parts)
+            return _(f"{location} (Estimated Location: {self.ip_address})")
+        return _(f"Estimated Location: {self.ip_address}")
+
+    def _get_defaults_for_estimated_location(self):
+        """
+        Used to get default values for creating or updating
+        an estimated location based on the WHOIS information.
+        """
+        return {
+            "name": self._location_name,
+            "type": "outdoor",
+            "is_mobile": False,
+            "geometry": self.coordinates,
+            "address": self.formatted_address,
+        }

openwisp_controller/config/controller/views.py

@@ -153,15 +153,6 @@ def get(self, request, pk):
         # updates cache if ip addresses changed
         if updated:
             self.update_device_cache(device)
-        # When update fields are present then save() will run the WHOIS
-        # lookup. But if there are no update fields, we still want to
-        # trigger the WHOIS lookup if there is no record for the device's
-        # last_ip.
-        elif (
-            app_settings.WHOIS_CONFIGURED
-            and not device.whois_service.get_device_whois_info()
-        ):
-            device.whois_service.trigger_whois_lookup()
         checksum_requested.send(
             sender=device.__class__, instance=device, request=request
         )