when Occupancy =100%， Free is 3670.7%

[wangzhanyu0207]

Description

when Occupancy =100%， Free is 3670.7

Expected Behavior

Actual Behavior

Steps to Reproduce

Context

Possible Fix

Logs

Configuration files

Your Environment

• OpenCAS version (commit hash or tag):
• Operating System:
• Kernel version:
• Cache device type (NAND/Optane/other):
• Core device type (HDD/SSD/other):
• Cache configuration:
• • Cache mode: (default: wt)
• • Cache line size: (default: 4)
• • Promotion policy: (default: always)
• • Cleaning policy: (default: alru)
• • Sequential cutoff policy: (default: full)
• Other (e.g. lsblk, casadm -P, casadm -L)

casadm -P -i 1
Cache Id                  1
Cache Size                936044240 [4KiB Blocks] / 3570.73 [GiB]
Cache Device              /dev/nvme0n1
Exported Object           -
Core Devices              4
Inactive Core Devices     0
Write Policy              wb
Cleaning Policy           alru
Promotion Policy          always
Cache line size           32 [KiB]
Metadata Memory Footprint 6.7 [GiB]
Dirty for                 96608 [s] / 1 [d] 2 [h] 50 [m] 8 [s]
Status                    Running

+==================+=============+========+=============+
| Usage statistics |    Count    |   %    |   Units     |
+==================+=============+========+=============+
| Occupancy        |   936118232 |  100.0 | 4KiB Blocks |
| Free             | 34359664376 | 3670.7 | 4KiB Blocks |
| Clean            |   932821960 |   99.7 | 4KiB Blocks |
| Dirty            |     3296272 |    0.4 | 4KiB Blocks |
+==================+=============+========+=============+

+======================+=========+=======+==========+
| Request statistics   |  Count  |   %   | Units    |
+======================+=========+=======+==========+
| Read hits            |   27921 |   0.4 | Requests |
| Read partial misses  |     517 |   0.0 | Requests |
| Read full misses     |   30289 |   0.4 | Requests |
| Read total           |   58727 |   0.8 | Requests |
+----------------------+---------+-------+----------+
| Write hits           |  423920 |   5.7 | Requests |
| Write partial misses |    3760 |   0.1 | Requests |
| Write full misses    |  268956 |   3.6 | Requests |
| Write total          |  696636 |   9.4 | Requests |
+----------------------+---------+-------+----------+
| Pass-Through reads   |  345860 |   4.7 | Requests |
| Pass-Through writes  | 6272818 |  85.1 | Requests |
| Serviced requests    |  755363 |  10.2 | Requests |
+----------------------+---------+-------+----------+
| Total requests       | 7374041 | 100.0 | Requests |
+======================+=========+=======+==========+

+==================================+===========+=======+=============+
| Block statistics                 |   Count   |   %   |   Units     |
+==================================+===========+=======+=============+
| Reads from core(s)               |  29053026 |  14.3 | 4KiB Blocks |
| Writes to core(s)                | 174643936 |  85.7 | 4KiB Blocks |
| Total to/from core(s)            | 203696962 | 100.0 | 4KiB Blocks |
+----------------------------------+-----------+-------+-------------+
| Reads from cache                 |   1511101 |  22.4 | 4KiB Blocks |
| Writes to cache                  |   5237038 |  77.6 | 4KiB Blocks |
| Total to/from cache              |   6748139 | 100.0 | 4KiB Blocks |
+----------------------------------+-----------+-------+-------------+
| Reads from exported object(s)    |  29626607 |  14.3 | 4KiB Blocks |
| Writes to exported object(s)     | 178199622 |  85.7 | 4KiB Blocks |
| Total to/from exported object(s) | 207826229 | 100.0 | 4KiB Blocks |
+==================================+===========+=======+=============+

+====================+=======+=====+==========+
| Error statistics   | Count |  %  | Units    |
+====================+=======+=====+==========+
| Cache read errors  |     0 | 0.0 | Requests |
| Cache write errors |     0 | 0.0 | Requests |
| Cache total errors |     0 | 0.0 | Requests |
+--------------------+-------+-----+----------+
| Core read errors   |     0 | 0.0 | Requests |
| Core write errors  |     0 | 0.0 | Requests |
| Core total errors  |     0 | 0.0 | Requests |
+--------------------+-------+-----+----------+
| Total errors       |     0 | 0.0 | Requests |
+====================+=======+=====+==========+

casadm -V
+=========================+======================+
| Name | Version |
+=========================+======================+
| CAS Cache Kernel Module | 25.03.0.0000.release |
| CAS CLI Utility | 25.03.0.0000.release |
+=========================+======================+

[liyimeng]

+==================+=============+========+=============+
| Usage statistics |    Count    |   %    |   Units     |
+==================+=============+========+=============+
| Occupancy        |   936118232 |  100.0 | 4KiB Blocks |
| Free             | 34359664376 | 3670.7 | 4KiB Blocks |
| Clean            |   932821960 |   99.7 | 4KiB Blocks |
| Dirty            |     3296272 |    0.4 | 4KiB Blocks |
+==================+=============+========+=============+

This looks worrying. Wil opencas overrun the cache device and lost data?

[robertbaldyga]

Does that situation persist, or is it one-time event?

The consistency of stats is not always guaranteed, because internally stats are accounted using atomic counters per core, which means they need to be accumulated over all the cores, and if the value of the atomic counter changes in the meantime, some transient inconsistencies may occur.

When it comes usage statistics, only occupancy and dirty are accounted internally. Free and clean are then calculated (free=size-occupancy, clean=size-dirty). Here it looks like accumulated value of occupancy on all cores is bigger than size of the cache, which causes free counter to underflow - hence a huge number.

If this is one-off event, then there is nothing to worry about. If it persists, then it may indicate some accounting problem (not necessarily data consistency problem, but it may require further debugging).

[liyimeng]

Thanks @robertbaldyga we will an eye on it.