Duplicate NETWORK.CREATE usage events resulting in duplicate usage records

[rajujith]

problem

For isolated networks and VPC tiers, a usage event NETWORK.CREATE is created in the cloud.usage_event table whenever the network transitions from 'Allocated' to 'Implemented' state. A NETWORK.DELETE event is created when the network is deleted.

I suspect there is a bug in the code or a bad design causing this issue. These events result in duplicate usage records for the network for the same period.

Either the NETWORK.CREATE should be created only when it is created, or if we need to capture usage for Allocated and Implemented networks separately, there could be another event.

The network below is not deleted.

mysql> select * from cloud.usage_event where  resource_id=207 and type in ('NETWORK.CREATE','NETWORK.DELETE');
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| id  | type           | account_id | created             | zone_id | resource_id | resource_name | offering_id | template_id | size | resource_type | processed | virtual_size |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| 146 | NETWORK.CREATE |          2 | 2025-04-10 06:02:14 |       1 |         207 | myNet1        |          10 |        NULL | NULL | NULL          |         0 |         NULL |
| 151 | NETWORK.CREATE |          2 | 2025-04-10 06:08:45 |       1 |         207 | myNet1        |          10 |        NULL | NULL | NULL          |         0 |         NULL |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+

The network below got deleted:

mysql> select * from usage_event where  resource_id=204 and type in ('NETWORK.CREATE','NETWORK.DELETE');
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
| id  | type           | account_id | created             | zone_id | resource_id | resource_name | offering_id | template_id | size | resource_type | processed | virtual_size |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+
|  12 | NETWORK.CREATE |          2 | 2025-03-21 07:01:05 |       1 |         204 | Berlin        |          10 |        NULL | NULL | NULL          |         0 |         NULL |
|  29 | NETWORK.CREATE |          2 | 2025-03-21 07:34:56 |       1 |         204 | Berlin        |          10 |        NULL | NULL | NULL          |         0 |         NULL |
|  36 | NETWORK.CREATE |          2 | 2025-03-21 08:28:06 |       1 |         204 | Berlin        |          10 |        NULL | NULL | NULL          |         0 |         NULL |
|  55 | NETWORK.CREATE |          2 | 2025-03-24 06:17:48 |       1 |         204 | Berlin        |          10 |        NULL | NULL | NULL          |         0 |         NULL |
|  72 | NETWORK.CREATE |          2 | 2025-03-24 08:33:39 |       1 |         204 | Berlin        |          10 |        NULL | NULL | NULL          |         0 |         NULL |
| 142 | NETWORK.DELETE |          2 | 2025-04-10 05:08:30 |       1 |         204 | Berlin        |          10 |        NULL | NULL | NULL          |         0 |         NULL |
+-----+----------------+------------+---------------------+---------+-------------+---------------+-------------+-------------+------+---------------+-----------+--------------+

Usage records for the same:

mysql> select account_id,usage_id,usage_type,start_date,end_date,count(*) as count from cloud_usage.cloud_usage where usage_id=204 and DATE(start_date)='2025-04-10' group by account_id,usage_id,usage_type,start_date,end_date having count >1;
+------------+----------+------------+---------------------+---------------------+-------+
| account_id | usage_id | usage_type | start_date          | end_date            | count |
+------------+----------+------------+---------------------+---------------------+-------+
|          2 |      204 |         30 | 2025-04-10 00:00:00 | 2025-04-10 00:29:59 |     5 |
|          2 |      204 |         30 | 2025-04-10 00:30:00 | 2025-04-10 01:29:59 |     5 |
|          2 |      204 |         30 | 2025-04-10 01:30:00 | 2025-04-10 02:29:59 |     5 |
|          2 |      204 |         30 | 2025-04-10 02:30:00 | 2025-04-10 03:29:59 |     5 |
|          2 |      204 |         30 | 2025-04-10 03:30:00 | 2025-04-10 04:29:59 |     5 |
|          2 |      204 |         30 | 2025-04-10 04:30:00 | 2025-04-10 05:29:59 |     5 |
+------------+----------+------------+---------------------+---------------------+-------+

mysql> select * from cloud_usage.usage_networks where network_id=204;
+----+---------------------+---------+------------+------------+-----------+-------------+---------------------+---------------------+
| id | network_offering_id | zone_id | network_id | account_id | domain_id | state       | removed             | created             |
+----+---------------------+---------+------------+------------+-----------+-------------+---------------------+---------------------+
|  1 |                  10 |       1 |        204 |          2 |         1 | Destroy     | 2025-04-10 05:08:30 | 2025-03-21 07:01:05 |
|  2 |                  10 |       1 |        204 |          2 |         1 | Allocated   | NULL                | 2025-03-21 07:34:56 |
|  3 |                  10 |       1 |        204 |          2 |         1 | Implemented | NULL                | 2025-03-21 08:28:06 |
|  4 |                  10 |       1 |        204 |          2 |         1 | Allocated   | NULL                | 2025-03-24 06:17:48 |
|  5 |                  10 |       1 |        204 |          2 |         1 | Allocated   | NULL                | 2025-03-24 08:33:39 |
+----+---------------------+---------+------------+------------+-----------+-------------+---------------------+---------------------+

Workaround

update configuration set value='-1' where name='network.gc.interval';

Note that it has a side effect that it will not release guest VLANs even when there are no running instances on the network, and the VRs will continue to run.

Existing duplicate usage events and usage records need to be fixed separately.

versions

4.19.1.2, 4.19.2

The steps to reproduce the bug

1. Create an isolated network and deploy an instance
2. Stop the (or all) instance and let the network GC shutdown the network.
3. Start the instance.
4. Observe the usage_event table entries cloud.
5. After the usage processing, observe the tables usage_event, usage_networks and cloud_usage in cloud_usage database. List usage records using API or CMK to verify the duplicate records.
   ...

What to do about it?

Review the issue, This looks like a bug that needs to be fixed soon.

[winterhazel]

#9888 fixed this and some other issues on 4.20.1. I'll backport the changes to 4.19.3, and work on a normalization script.

[Pearl1594]

@winterhazel thanks for sharing, any timeline on when you can raise the backport PR. Thanks.

[winterhazel]

@Pearl1594 today 🙂

Regarding the normalization script, though, I hope to look into it within this week.

[Pearl1594]

Thanks @winterhazel! ..

[DaanHoogland]

@winterhazel , I noticed a small issue in your solution that I want to discuss. Not sure if this is the place but let me start here at least.

What happens in your solution is that for all CREATE events a remove is forced if a DESTROY event is found for the same network. I think the issue is actually that CREATE events are created on implementation as well, which should be a separate type of event. I am not sure how harmfull this is but we are now hiding the bug instead of solving it.

If you look at for instance VMs, these can have a sequence of events like,
CREATE, START, STOP, START, STOP, DESTROY.
Networks now would have
CREATE, CREATE, CREATE, DESTROY for the same type of actions.
One would expect
CREATE, IMPLEMENT, STOP, IMPLEMENT, STOP, DESTROY

All that said, I am not against backporting your fix. I think in the network case, usage might not require the specifics of implement, though organisations with strict resource billing would.

future work?

[winterhazel]

Hey @DaanHoogland, thanks for bringing this up.

We're dealing with two different issues here.

The first issue is regarding existing environments. These environments already have multiple CREATE events for the same network, and are seeing (or will see after the next Usage job if we do not address this) duplicated usage record generation. This behavior impacts the usage rating through Quota (because it uses the generated usage records) and through external tools based on the generated usage records.

I don't usually see organizations performing resource usage rating by directly consuming the events from usage_event, only the two options I mentioned. That's why I was concerned in addressing this issue first (through #10712 and a normalization script). Though, now that you pointed it out, I do recognize that this is probably an existing scenario.

Also, I think you may have misunderstood something in my solution by your description, so just making sure that we are aligned on this: my changes mark existing network usage helper entries for a network as removed whenever a CREATE event is processed, independently of there being a DESTROY or not, so that two consecutive CREATE events for the same network do not result in duplicated usage records being generated.

The second issue is what you pointed out: network state changes are not tracked properly. We only have three usage events published for networks, and one of them is published for two different situations:

• NETWORK.CREATE: when the network is created, or implemented.
• NETWORK.DELETE: when the network is deleted.
• NETWORK.UPDATE: when updateNetwork is called for the network.

I agree with you that this should be improved. It would be good to have more event types so that we can properly identify when a network transitioned to Implemented, or from Implemented to Allocated. This would be useful for organizations that perform rating based on the usage events (and Quota as well). However, only improving this would not address existing environments that already have duplicate events and duplicate record generation.