Skip to content

Commit c266e80

Browse files
nemesifierDragnEmperorpandafy
committed
[chores] Improved WHOIS & Estimated Location features
Fixed multiple issues discovered during testing and review across WHOIS, estimated location, and device IP handling, and consolidated QA and maintenance changes. Key changes: - Ensure clear_last_ip command calls model save() to trigger signals and cache invalidation - Fix WHOIS and estimated location bugs, including shared location recreation and modified timestamp updates - Improve handling of unchanged IPs in DeviceChecksumView - Rename and adjust migrations for estimated location settings - Optimize queries and improve error handling (global HttpError handling) - Improve UI (CSS/JS/templates) and documentation - Improved efficiency of `manage_estimated_locations` - Resolved N+1 query issue in `LocationListCreateView` - Fixed HTTP request inside `transaction.atomic()` - Shifted to Celery `send_task` to retain dependency flow - Removed whois info from device list API to avoid N+1 - Simplified auto-naming of estimated locations - Updated location update description messages - Increased WHOIS refresh threshold to 90 days (~92% reduction in API calls) - WHOIS data refresh now executes on transaction commit - Skips `delete_whois_record` if `initial_ip_address` is None - Fixed late-binding closure issue with `_initial_last_ip` - Error notifications now only sent if `device_pk` is not None - CIDR field length increased to accommodate IPv6 - Added `is_estimated` filter and column to admin - Added `is_estimated` to `LocationSerializer` and API filter - Improved test coverage - Fixed parallel test 401 errors - Several other improvements and fixes for problems found during testing Co-authored-by: DragnEmperor <dragnemperor@gmail.com> Co-authored-by: Gagan Deep <pandafy.dev@gmail.com>
1 parent 703f288 commit c266e80

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

44 files changed

+1656
-719
lines changed

‎docs/index.rst‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -47,9 +47,9 @@ the OpenWISP architecture.
4747
user/vxlan-wireguard.rst
4848
user/zerotier.rst
4949
user/openvpn.rst
50-
user/subnet-division-rules.rst
5150
user/whois.rst
5251
user/estimated-location.rst
52+
user/subnet-division-rules.rst
5353
user/rest-api.rst
5454
user/settings.rst
5555

‎docs/user/estimated-location.rst‎

Lines changed: 89 additions & 53 deletions
Original file line numberDiff line numberDiff line change
@@ -1,13 +1,19 @@
11
Estimated Location
22
==================
33

4+
.. image:: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/estimated-locations/is-estimated-flag.png
5+
:target: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/estimated-locations/is-estimated-flag.png
6+
:alt: Estimated location flag
7+
48
.. important::
59

610
The **Estimated Location** feature is **disabled by default**.
711

8-
Before enabling it, the :doc:`WHOIS Lookup feature <whois>` must be
9-
enabled. Then set
10-
:ref:`OPENWISP_CONTROLLER_ESTIMATED_LOCATION_ENABLED` to ``True``
12+
Before enabling it, the :ref:`WHOIS Lookup feature
13+
<controller_setup_whois_lookup>` must be enabled.
14+
15+
Then set :ref:`OPENWISP_CONTROLLER_ESTIMATED_LOCATION_ENABLED` to
16+
``True``.
1117

1218
.. contents:: **Table of contents**:
1319
:depth: 1
@@ -16,72 +22,102 @@ Estimated Location
1622
Overview
1723
--------
1824

19-
This feature automatically creates or updates a device’s location based on
20-
latitude and longitude information retrieved from the WHOIS Lookup
25+
This feature automatically creates or updates a device's location based on
26+
latitude and longitude information retrieved from the :doc:`whois`
2127
feature.
2228

23-
Trigger Conditions
24-
------------------
29+
It is very useful for those users who have devices scattered across
30+
different geographic regions and would like some help to place the devices
31+
on the map, while being gently reminded to improve the precision of the
32+
location with a direct link for doing so.
2533

26-
This feature is triggered when:
34+
It also significantly reduces the effort required to assign a geographic
35+
location manually when many devices are deployed in large buildings like
36+
schools, offices, hospitals, libraries, etc. Improve the precision of the
37+
estimated location just once and all the other devices sharing the same
38+
public IP will automatically inherit the same location.
2739

28-
- A **fresh WHOIS lookup** is performed for a device.
29-
- Or when a WHOIS record already exists for the device’s IP **and**:
40+
The feature is not useful in the following scenarios:
3041

31-
- The device’s last IP address is **public**.
32-
- WHOIS lookup and Estimated Location is **enabled** for the device’s
33-
organization.
42+
- Most devices are deployed in one single location.
43+
- Most devices are mobile (e.g. moving vehicles).
3444

35-
Behavior
36-
--------
45+
Visibility of Estimated Status
46+
------------------------------
3747

38-
The system will **attach the already existing matching location** of
39-
another device with same ip to the current device if:
48+
The estimated status of a location is visible in the admin interface in
49+
several ways:
4050

41-
- Only one device is found with that IP and it has a location.
42-
- The current device **has no location** or that location is
43-
**estimated**.
51+
- The location name will mention the *IP address* from which it was
52+
estimated.
53+
- A *warning message* appears at the top of the location list page as in
54+
the image below.
4455

45-
If there are multiple devices with location for the same IP, the system
46-
will **not attach any location** to the current device and a notification
47-
will be sent suggesting the user to manually assign/create a location for
48-
the device.
56+
.. image:: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/estimated-locations/estimated-warning.png
57+
:target: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/estimated-locations/estimated-warning.png
58+
:alt: Estimated location warning
4959

50-
If there is **no matching location**, a new estimated location is created
51-
or the existing one is updated using coordinates from the WHOIS record,
52-
but only if the existing location is estimated.
60+
- The *Is Estimated?* flag is displayed both in the location list page and
61+
in the location detail page, as in the images below.
5362

54-
If two devices share the same IP address and are assigned to the same
55-
location, and the last IP of one of the devices is updated, the system
56-
will create a new estimated location for that device.
63+
.. image:: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/estimated-locations/admin-list.png
64+
:target: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/estimated-locations/admin-list.png
65+
:alt: Estimated location admin list
5766

58-
Visibility of Estimated Status
59-
------------------------------
67+
.. image:: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/estimated-locations/is-estimated-flag.png
68+
:target: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/estimated-locations/is-estimated-flag.png
69+
:alt: Estimated location flag
70+
71+
- The device list page also allows filtering devices which are associated
72+
with estimated locations as shown in the image below.
6073

61-
The estimated status of a location is visible on the location page if the
62-
feature is enabled for the organization. The location admin page also
63-
includes indicators for the estimated status.
74+
.. image:: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/estimated-locations/filter-devices-by-estimated-location.png
75+
:target: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/estimated-locations/filter-devices-by-estimated-location.png
76+
:alt: Filter devices associated to estimated locations
6477

65-
- The name of the location will have suffix **(Estimated Location :
66-
<ip_address>)**.
67-
- A warning on top of the page.
68-
- **Is Estimated** field.
78+
Any change to the geographic coordinates of an estimated location will set
79+
the ``is_estimated`` field to ``False``.
6980

70-
Changes to the ``coordinates`` and ``geometry`` of the estimated location
71-
will set the ``is_estimated`` field to ``False`` and remove the
72-
"(Estimated Location)" suffix with IP from the location name.
81+
When manually increasing the precision of estimated locations, it is
82+
highly recommended to also change the auto-generated location name.
7383

74-
In REST API, the field will be visible in the :ref:`Device Location
75-
<device_location_estimated>`, :ref:`Location list
84+
In the REST API, the ``is_estimated`` field is visible in the :ref:`Device
85+
Location <device_location_estimated>`, :ref:`Location list
7686
<location_list_estimated>`, :ref:`Location Detail
77-
<location_detail_estimated>` and :ref:`Location list (GeoJson)
78-
<location_geojson_estimated>` if the feature is **enabled**. The field can
79-
also be used for filtering in the location list (including geojson)
80-
endpoints and in the :ref:`Device List <device_list_estimated_filters>`.
87+
<location_detail_estimated>` and :ref:`Location list (GeoJSON)
88+
<location_geojson_estimated>` endpoints if the feature is enabled. The
89+
field can also be used for filtering in the location list endpoints,
90+
including the GeoJSON endpoint, and in the :ref:`Device List
91+
<device_list_estimated_filters>`.
92+
93+
Triggers and Record Management
94+
------------------------------
95+
96+
The feature is triggered automatically when all the following conditions
97+
are met:
98+
99+
- A WHOIS lookup is performed.
100+
- The last IP is a public IP address.
101+
- Both WHOIS lookup and Estimated Location features are enabled for the
102+
device's organization.
103+
104+
If no matching location exists, a new estimated location is created using
105+
coordinates from the WHOIS record. If an estimated location already
106+
exists, it will be updated with the new coordinates.
107+
108+
If another device with the same IP already has a location, the system will
109+
assign the same location for any device having the same IP and not being
110+
assigned to any other location.
111+
112+
If two devices share the same IP and are assigned to the same location,
113+
and one of them updates its last IP, the system will create a new
114+
estimated location for that device.
81115

82-
Managing Older Estimated Locations
83-
----------------------------------
116+
When multiple devices with the same IP already have a location assigned
117+
but the locations differ, the system will send a notification to network
118+
administrators asking to manually resolve the ambiguity.
84119

85-
Whenever location related fields in WHOIS records are updated as per
86-
:ref:`Managing WHOIS Older Records <whois_older_records>`; the location
87-
will also be updated automatically.
120+
When WHOIS records are updated as described in :ref:`the WHOIS Lookup
121+
section <controller_whois_auto_management>`, any related estimated
122+
location will also be updated, if needed and only if the estimated
123+
location has not been manually modified to increase precision.

‎docs/user/intro.rst‎

Lines changed: 5 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -35,6 +35,8 @@ following features:
3535
- **VPN management**: automatically provision VPN tunnel configurations,
3636
including cryptographic keys and IP addresses, e.g.: :doc:`OpenVPN
3737
</user/vpn>`, :doc:`WireGuard <wireguard>`
38+
- :doc:`whois`: display information about the public IP address used by
39+
devices to communicate with OpenWISP
3840
- :doc:`import-export`
3941

4042
It exposes various :doc:`REST API endpoints <rest-api>`.
@@ -96,6 +98,9 @@ The geographic app is based on `django-loci
9698
geographic coordinates of the devices, as well as their indoor coordinates
9799
on floor plan images.
98100

101+
It also provides an :doc:`estimated-location` feature which automatically
102+
creates or updates device locations based on WHOIS data.
103+
99104
It exposes various :doc:`REST API endpoints <rest-api>`.
100105

101106
Subnet Division App

‎docs/user/rest-api.rst‎

Lines changed: 4 additions & 4 deletions
Original file line numberDiff line numberDiff line change
@@ -80,9 +80,9 @@ information.
8080

8181
**Estimated Location Filters**
8282

83-
if :doc:`Estimated Location feature <estimated-location>` is enabled,
83+
If :doc:`Estimated Location feature <estimated-location>` is enabled,
8484
devices can be filtered based on the estimated nature of their location
85-
using the ``geo_is_estimated``.
85+
using the ``geo_is_estimated`` filter.
8686

8787
**Available filters**
8888

@@ -814,7 +814,7 @@ List Locations
814814

815815
|estimated_details|
816816

817-
Locations can also be filtered using the ``is_estimated``.
817+
Locations can also be filtered using the ``is_estimated`` filter.
818818

819819
**Available filters**
820820

@@ -951,7 +951,7 @@ List Locations with Devices Deployed (in GeoJSON Format)
951951

952952
|estimated_details|
953953

954-
Locations can also be filtered using the ``is_estimated``.
954+
Locations can also be filtered using the ``is_estimated`` filter.
955955

956956
**Available filters**
957957

‎docs/user/settings.rst‎

Lines changed: 31 additions & 23 deletions
Original file line numberDiff line numberDiff line change
@@ -774,8 +774,8 @@ documentation regarding automatic retries for known errors.
774774

775775
Allows enabling the optional :doc:`WHOIS Lookup feature <whois>`.
776776

777-
.. image:: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/whois-admin-setting.png
778-
:target: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/whois-admin-setting.png
777+
.. image:: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/whois/admin-setting.png
778+
:target: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/whois/admin-setting.png
779779
:alt: WHOIS admin setting
780780

781781
After enabling this feature, you have to set
@@ -796,10 +796,10 @@ After enabling this feature, you have to set
796796

797797
============ =======
798798
**type**: ``str``
799-
**default**: None
799+
**default**: ``""``
800800
============ =======
801801

802-
Maxmind Account ID required for the :doc:`WHOIS Lookup feature <whois>`.
802+
MaxMind Account ID required for the :doc:`WHOIS Lookup feature <whois>`.
803803

804804
.. _openwisp_controller_whois_geoip_key:
805805

@@ -808,15 +808,28 @@ Maxmind Account ID required for the :doc:`WHOIS Lookup feature <whois>`.
808808

809809
============ =======
810810
**type**: ``str``
811-
**default**: None
811+
**default**: ``""``
812812
============ =======
813813

814-
Maxmind License Key required for the :doc:`WHOIS Lookup feature <whois>`.
814+
MaxMind License Key required for the :doc:`WHOIS Lookup feature <whois>`.
815815

816-
.. _openwisp_controller_whois_estimated_location_enabled:
816+
.. _openwisp_controller_whois_refresh_threshold_days:
817+
818+
``OPENWISP_CONTROLLER_WHOIS_REFRESH_THRESHOLD_DAYS``
819+
----------------------------------------------------
817820

818-
``OPENWISP_CONTROLLER_WHOIS_ESTIMATED_LOCATION_ENABLED``
819-
--------------------------------------------------------
821+
============ =======
822+
**type**: ``int``
823+
**default**: ``90``
824+
============ =======
825+
826+
Specifies the number of days after which the WHOIS information for a
827+
device is considered stale and eligible for refresh.
828+
829+
.. _openwisp_controller_estimated_location_enabled:
830+
831+
``OPENWISP_CONTROLLER_ESTIMATED_LOCATION_ENABLED``
832+
--------------------------------------------------
820833

821834
============ =========
822835
**type**: ``bool``
@@ -826,19 +839,14 @@ Maxmind License Key required for the :doc:`WHOIS Lookup feature <whois>`.
826839
Allows enabling the optional :doc:`Estimated Location feature
827840
<estimated-location>`.
828841

829-
.. image:: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/estimated-location-setting.png
830-
:target: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/estimated-location-setting.png
831-
:alt: Estimated Location setting
842+
.. warning::
832843

833-
.. _openwisp_controller_whois_refresh_threshold_days:
844+
:ref:`OPENWISP_CONTROLLER_WHOIS_ENABLED
845+
<openwisp_controller_whois_enabled>` must be set to ``True`` before
846+
enabling this feature. Enabling estimated locations while
847+
``WHOIS_ENABLED=False`` will raise ``ImproperlyConfigured`` at startup
848+
time.
834849

835-
``OPENWISP_CONTROLLER_WHOIS_REFRESH_THRESHOLD_DAYS``
836-
----------------------------------------------------
837-
838-
============ =======
839-
**type**: ``int``
840-
**default**: ``14``
841-
============ =======
842-
843-
Specifies the number of days after which the WHOIS information for a
844-
device is considered stale and eligible for refresh.
850+
.. image:: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/estimated-locations/admin-setting.png
851+
:target: https://raw.githubusercontent.com/openwisp/openwisp-controller/docs/docs/1.3/estimated-locations/admin-setting.png
852+
:alt: Estimated Location setting

0 commit comments

Comments
 (0)