Skip to content

Commit fdc3036

Browse files
murrantmurrant
authored
Refactored storage module and added yaml support (librenms#17024)
* Storage Module WIP * HrStorage discovery, need to add ignore rules * isValid WIP * HRStorage working, shared ratio value filling with mempools * Test override for aix (need to migrate data) Start on yaml implementation, but I think SnmpQuery needs extra juice * Cleanups and fixes * WIP * mempools wip * Abort for bad numbers during fill for mempools * Small hr cleanups * WIP * Fix up after rebase * Getting things in shape * Automatically fill num_oid for polling * Fix translated OID missing index * Improve should_poll heuristic Handle string index Implement pre fetch (should rename from pre-cache) * Fix data structure * Actually call callback * Match previous should poll heuristic * assume negative used means memory calculation is incorrect * awplus cache and buffers should not have been discovered * Oid fields shoul * Fix skip_values input * Fix skip values and snmp_flags * Don't cache numeric IDs because of braindead vendors * Fix various issues with numeric oids * Invalid mempools for draytek and exa * data updates * Set up some tests and handle oid non-numeric strings better (null instead of casting to 0) * Reject mempools with null fields * Skip indices with no value * Bad test data (only total, no actual memory usage info) * Fix typo * ZHONE data exceeds max index according to MIB... * Add UCD storage discovery * Cache Oid::toNumeric for runtime only, allows us to improve cache hits * Fixup storage, if no yaml results, try HR and UCD * storage_type no .0 for hrStorage * Skip negative values and some small cleanups * Skip storage size 0 volumes, add debug messages default units = 1 * add /sys/fs/cgroup and /run/user/* to mount ignores * datadomain add skip_values, fix total->size * Add script to convert storage test data * datadomain merged yaml and hr * eltex * ericsson-ipos wrong units in original data * forcepoint * hpe-ilo partial * Apply fixes from StyleCI * Fix sort * fix up hpe * intelliflash + polling fixes * Cisco flash * through K * Fix bad storage type responses. * oceanstor * pfsense fix fs types * ucd refinement * truenas - was a little broken, yaml+hr * vrp * enterasys * equallogic no data * cleanup * Fix up test data after fixes to host resources handling * remove data conversion tool * Apply fixes from StyleCI * Fix schema validation, more cleanup * update db_schema.yaml * lint fixes * Ignore scalar .0 removal for all numeric oids * Updates from review * Add missing types * Add missing types - test data * split migrations for sqlite --------- Co-authored-by: Tony Murray <[email protected]>
1 parent e5ce7bf commit fdc3036

File tree

297 files changed

+9650
-21896
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

297 files changed

+9650
-21896
lines changed

‎LibreNMS/Device/YamlDiscovery.php‎

Lines changed: 6 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -180,7 +180,7 @@ public static function computeNumericalOID(OS $os, array $data): string
180180
* @param array $pre_cache snmp data fetched from device
181181
* @return mixed|string|string[]|null
182182
*/
183-
public static function replaceValues($name, $index, $count, $def, $pre_cache)
183+
public static function replaceValues($name, $index, $count, $def, $pre_cache = [])
184184
{
185185
$value = static::getValueFromData($name, $index, $def, $pre_cache);
186186

@@ -253,6 +253,11 @@ public static function getValueFromData($name, $index, $discovery_data, $pre_cac
253253
return $value;
254254
}
255255

256+
// if index is 0 and this is a scalar value, remove the scalar .0
257+
if ($index == 0 && str_ends_with($name, '.0') && ! Oid::of($name)->isNumeric()) {
258+
$name = substr($name, 0, -2);
259+
}
260+
256261
if (isset($pre_cache[$index][$name])) {
257262
$value = $pre_cache[$index][$name];
258263
Log::debug("Using $value from \$pre_cache[$index][$name] for $name");

‎LibreNMS/Discovery/Yaml/IndexField.php‎

Lines changed: 50 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
<?php
2+
/*
3+
* IndexField.php
4+
*
5+
* -Description-
6+
*
7+
* This program is free software: you can redistribute it and/or modify
8+
* it under the terms of the GNU General Public License as published by
9+
* the Free Software Foundation, either version 3 of the License, or
10+
* (at your option) any later version.
11+
*
12+
* This program is distributed in the hope that it will be useful,
13+
* but WITHOUT ANY WARRANTY; without even the implied warranty of
14+
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the
15+
* GNU General Public License for more details.
16+
*
17+
* You should have received a copy of the GNU General Public License
18+
* along with this program. If not, see <http://www.gnu.org/licenses/>.
19+
*
20+
* @package LibreNMS
21+
* @link http://librenms.org
22+
* @copyright 2025 Tony Murray
23+
* @author Tony Murray <[email protected]>
24+
*/
25+
26+
namespace LibreNMS\Discovery\Yaml;
27+
28+
use Illuminate\Support\Str;
29+
use LibreNMS\Util\Oid;
30+
31+
class IndexField extends YamlDiscoveryField
32+
{
33+
public function calculateValue(array $yaml, array $data, string $index, int $count): void
34+
{
35+
if (array_key_exists($this->key, $yaml)) {
36+
parent::calculateValue($yaml, $data, $index, $count);
37+
38+
return;
39+
}
40+
41+
if (Str::startsWith($index, '.') && Oid::of($index)->isNumeric()) {
42+
// if this is a full numeric oid instead of an index, assume it is a scalar
43+
$this->value = 0;
44+
45+
return;
46+
}
47+
48+
$this->value = $index;
49+
}
50+
}

includes/polling/storage/eltex-mes24xx.inc.php renamed to LibreNMS/Discovery/Yaml/LiteralField.php

Lines changed: 15 additions & 11 deletions
Original file line numberDiff line numberDiff line change
@@ -1,6 +1,8 @@
11
<?php
22
/*
3-
* LibreNMS storage poller module for Eltex-MES24xx
3+
* LiteralField.php
4+
*
5+
* -Description-
46
*
57
* This program is free software: you can redistribute it and/or modify
68
* it under the terms of the GNU General Public License as published by
@@ -13,18 +15,20 @@
1315
* GNU General Public License for more details.
1416
*
1517
* You should have received a copy of the GNU General Public License
16-
* along with this program. If not, see <https://www.gnu.org/licenses/>.
18+
* along with this program. If not, see <http://www.gnu.org/licenses/>.
1719
*
1820
* @package LibreNMS
19-
* @link https://www.librenms.org
20-
*
21-
* @author Peca Nesovanovic <[email protected]>
22-
* @copyright 2024 Peca Nesovanovic
21+
* @link http://librenms.org
22+
* @copyright 2025 Tony Murray
23+
* @author Tony Murray <[email protected]>
2324
*/
2425

25-
if ($device['os'] == 'eltex-mes24xx') {
26-
$storage['units'] = 1;
27-
$storage['size'] = 100;
28-
$storage['used'] = SnmpQuery::hideMib()->get('ARICENT-ISS-MIB::issSwitchCurrentFlashUsage.0')->value();
29-
$storage['free'] = $storage['size'] - $storage['used'];
26+
namespace LibreNMS\Discovery\Yaml;
27+
28+
class LiteralField extends YamlDiscoveryField
29+
{
30+
public function calculateValue(array $yaml, array $data, string $index, int $count): void
31+
{
32+
$this->value = $yaml[$this->key] ?? $this->default;
33+
}
3034
}

‎LibreNMS/Discovery/Yaml/OidField.php‎

Lines changed: 47 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,47 @@
1+
<?php
2+
/*
3+
* OidField.php
4+
*
5+
* -Description-
6+
*
7+
* This program is free software: you can redistribute it and/or modify
8+
* it under the terms of the GNU General Public License as published by
9+
* the Free Software Foundation, either version 3 of the License, or
10+
* (at your option) any later version.
11+
*
12+
* This program is distributed in the hope that it will be useful,
13+
* but WITHOUT ANY WARRANTY; without even the implied warranty of
14+
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the
15+
* GNU General Public License for more details.
16+
*
17+
* You should have received a copy of the GNU General Public License
18+
* along with this program. If not, see <http://www.gnu.org/licenses/>.
19+
*
20+
* @package LibreNMS
21+
* @link http://librenms.org
22+
* @copyright 2025 Tony Murray
23+
* @author Tony Murray <[email protected]>
24+
*/
25+
26+
namespace LibreNMS\Discovery\Yaml;
27+
28+
use LibreNMS\Discovery\YamlDiscoveryDefinition;
29+
30+
class OidField extends YamlDiscoveryField
31+
{
32+
public bool $isOid = true;
33+
34+
public function __construct(string $key, ?string $model_column = null, ?string $default = null, ?\Closure $callback = null, \Closure|bool|null $should_poll = null)
35+
{
36+
parent::__construct($key, $model_column, $default, $callback);
37+
38+
// should poll heuristic
39+
if (is_bool($should_poll)) {
40+
$this->should_poll = fn (YamlDiscoveryDefinition $def) => $should_poll;
41+
} elseif ($should_poll === null) {
42+
$this->should_poll = fn (YamlDiscoveryDefinition $def) => is_numeric($this->value);
43+
} else {
44+
$this->should_poll = $should_poll;
45+
}
46+
}
47+
}

‎LibreNMS/Discovery/Yaml/YamlDiscoveryField.php‎

Lines changed: 90 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,90 @@
1+
<?php
2+
/*
3+
* YamlDiscoveryField.php
4+
*
5+
* -Description-
6+
*
7+
* This program is free software: you can redistribute it and/or modify
8+
* it under the terms of the GNU General Public License as published by
9+
* the Free Software Foundation, either version 3 of the License, or
10+
* (at your option) any later version.
11+
*
12+
* This program is distributed in the hope that it will be useful,
13+
* but WITHOUT ANY WARRANTY; without even the implied warranty of
14+
* MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.See the
15+
* GNU General Public License for more details.
16+
*
17+
* You should have received a copy of the GNU General Public License
18+
* along with this program. If not, see <http://www.gnu.org/licenses/>.
19+
*
20+
* @package LibreNMS
21+
* @link http://librenms.org
22+
* @copyright 2025 Tony Murray
23+
* @author Tony Murray <[email protected]>
24+
*/
25+
26+
namespace LibreNMS\Discovery\Yaml;
27+
28+
use LibreNMS\Device\YamlDiscovery;
29+
use LibreNMS\Discovery\YamlDiscoveryDefinition;
30+
use LibreNMS\Util\Number;
31+
use LibreNMS\Util\StringHelpers;
32+
33+
class YamlDiscoveryField
34+
{
35+
public bool $isOid = false;
36+
public mixed $value = null;
37+
public ?\Closure $should_poll;
38+
39+
public function __construct(
40+
public readonly string $key,
41+
public readonly ?string $model_column = null,
42+
public readonly int|string|null $default = null,
43+
public readonly ?\Closure $callback = null,
44+
) {
45+
$this->should_poll = fn (YamlDiscoveryDefinition $def) => false;
46+
}
47+
48+
public function calculateValue(array $yaml, array $data, string $index, int $count): void
49+
{
50+
if (array_key_exists($this->key, $yaml)) {
51+
$key = $this->key;
52+
} else {
53+
$key = $this->default;
54+
$yaml = [$key => $this->default];
55+
}
56+
57+
if (empty($yaml[$key]) || is_numeric($yaml[$key])) {
58+
// if default is an empty or simple value, just set it
59+
$this->setValue($yaml[$key]);
60+
61+
return;
62+
}
63+
64+
$value = YamlDiscovery::replaceValues($key, $index, $count, $yaml, $data);
65+
66+
// oid is specifically looking for a number. So if it is not a number, return null
67+
if ($this->isOid && ! StringHelpers::hasNumber($value)) {
68+
$this->setValue(null);
69+
70+
return;
71+
}
72+
73+
$this->setValue($value);
74+
}
75+
76+
private function setValue(mixed $value): void
77+
{
78+
if (is_callable($this->callback)) {
79+
$this->value = call_user_func($this->callback, $value);
80+
81+
return;
82+
}
83+
84+
if ($this->isOid && $value !== null) {
85+
$value = Number::cast($value);
86+
}
87+
88+
$this->value = $value;
89+
}
90+
}

0 commit comments

Comments
 (0)