Skip to content

Commit 47cacb2

Browse files
alexandraBaraalexandraBara
committed
Merge branch 'development' into alex_syslog
2 parents 93a97a0 + bf2923a commit 47cacb2

File tree

14 files changed

+450
-22
lines changed

14 files changed

+450
-22
lines changed

‎.github/workflows/code_quality_checks.yml‎

Lines changed: 6 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -13,8 +13,12 @@ jobs:
1313

1414
steps:
1515
- uses: actions/checkout@v3
16-
- uses: pre-commit/[email protected]
17-
16+
- name: setup environment
17+
run: |
18+
./dev-setup.sh
19+
- name: run pre-commit hooks
20+
run: |
21+
pre-commit run --all-files --show-diff-on-failure --color=always
1822
- name: Print message on failure
1923
if: failure()
2024
run: |

‎.mypy.ini‎

Lines changed: 122 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,122 @@
1+
[mypy]
2+
# Global mypy configuration
3+
4+
[mypy-nodescraper.base.regexanalyzer]
5+
ignore_errors = True
6+
7+
[mypy-nodescraper.cli.cli]
8+
ignore_errors = True
9+
10+
[mypy-nodescraper.cli.dynamicparserbuilder]
11+
ignore_errors = True
12+
13+
[mypy-nodescraper.cli.helper]
14+
ignore_errors = True
15+
16+
[mypy-nodescraper.cli.inputargtypes]
17+
ignore_errors = True
18+
19+
[mypy-nodescraper.configbuilder]
20+
ignore_errors = True
21+
22+
[mypy-nodescraper.configregistry]
23+
ignore_errors = True
24+
25+
[mypy-nodescraper.enums.eventpriority]
26+
ignore_errors = True
27+
28+
[mypy-nodescraper.enums.systeminteraction]
29+
ignore_errors = True
30+
31+
[mypy-nodescraper.interfaces.connectionmanager]
32+
ignore_errors = True
33+
34+
[mypy-nodescraper.interfaces.datacollectortask]
35+
ignore_errors = True
36+
37+
[mypy-nodescraper.interfaces.dataplugin]
38+
ignore_errors = True
39+
40+
[mypy-nodescraper.interfaces.task]
41+
ignore_errors = True
42+
43+
[mypy-nodescraper.models.analyzerargs]
44+
ignore_errors = True
45+
46+
[mypy-nodescraper.models.datamodel]
47+
ignore_errors = True
48+
49+
[mypy-nodescraper.pluginexecutor]
50+
ignore_errors = True
51+
52+
[mypy-nodescraper.plugins.inband.bios.analyzer_args]
53+
ignore_errors = True
54+
55+
[mypy-nodescraper.plugins.inband.cmdline.cmdline_analyzer]
56+
ignore_errors = True
57+
58+
[mypy-nodescraper.plugins.inband.dimm.dimm_collector]
59+
ignore_errors = True
60+
61+
[mypy-nodescraper.plugins.inband.dkms.analyzer_args]
62+
ignore_errors = True
63+
64+
[mypy-nodescraper.plugins.inband.dmesg.dmesg_analyzer]
65+
ignore_errors = True
66+
67+
[mypy-nodescraper.plugins.inband.dmesg.dmesgdata]
68+
ignore_errors = True
69+
70+
[mypy-nodescraper.plugins.inband.memory.memory_collector]
71+
ignore_errors = True
72+
73+
[mypy-nodescraper.plugins.inband.os.os_collector]
74+
ignore_errors = True
75+
76+
[mypy-nodescraper.plugins.inband.package.analyzer_args]
77+
ignore_errors = True
78+
79+
[mypy-nodescraper.plugins.inband.process.analyzer_args]
80+
ignore_errors = True
81+
82+
[mypy-nodescraper.plugins.inband.process.process_collector]
83+
ignore_errors = True
84+
85+
[mypy-nodescraper.plugins.inband.rocm.rocm_plugin]
86+
ignore_errors = True
87+
88+
[mypy-nodescraper.taskresulthooks.filesystemloghook]
89+
ignore_errors = True
90+
91+
[mypy-nodescraper.typeutils]
92+
ignore_errors = True
93+
94+
[mypy-nodescraper.utils]
95+
ignore_errors = True
96+
97+
[mypy-test.unit.conftest]
98+
ignore_errors = True
99+
100+
[mypy-test.unit.framework.common.shared_utils]
101+
ignore_errors = True
102+
103+
[mypy-test.unit.framework.test_cli_helper]
104+
ignore_errors = True
105+
106+
[mypy-test.unit.framework.test_dataplugin]
107+
ignore_errors = True
108+
109+
[mypy-test.unit.framework.test_plugin_executor]
110+
ignore_errors = True
111+
112+
[mypy-nodescraper.models.collectorargs]
113+
ignore_errors = True
114+
115+
[mypy-nodescraper.plugins.inband.kernel_module.*]
116+
ignore_errors = True
117+
118+
[mypy-nodescraper.plugins.inband.nvme.nvme_collector]
119+
ignore_errors = True
120+
121+
[mypy-test.unit.framework.test_cli]
122+
ignore_errors = True

‎.pre-commit-config.yaml‎

Lines changed: 6 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -15,3 +15,9 @@ repos:
1515
rev: 25.1.0
1616
hooks:
1717
- id: black
18+
- repo: https://github.com/pre-commit/mirrors-mypy
19+
rev: v1.15.0
20+
hooks:
21+
- id: mypy
22+
args: [--install-types, --non-interactive, --explicit-package-bases, --allow-redefinition]
23+
language: system

‎README.md‎

Lines changed: 38 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -17,10 +17,9 @@ The Node Scraper CLI can be used to run Node Scraper plugins on a target system.
1717
options are available:
1818

1919
```sh
20-
usage: node-scraper [-h] [--sys-name STRING] [--sys-location {LOCAL,REMOTE}] [--sys-interaction-level {PASSIVE,INTERACTIVE,DISRUPTIVE}]
21-
[--sys-sku STRING] [--sys-platform STRING] [--plugin-configs [STRING ...]] [--system-config STRING]
22-
[--connection-config STRING] [--log-path STRING] [--log-level {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}]
23-
[--gen-reference-config]
20+
usage: node-scraper [-h] [--sys-name STRING] [--sys-location {LOCAL,REMOTE}] [--sys-interaction-level {PASSIVE,INTERACTIVE,DISRUPTIVE}] [--sys-sku STRING]
21+
[--sys-platform STRING] [--plugin-configs [STRING ...]] [--system-config STRING] [--connection-config STRING] [--log-path STRING]
22+
[--log-level {CRITICAL,FATAL,ERROR,WARN,WARNING,INFO,DEBUG,NOTSET}] [--gen-reference-config] [--skip-sudo]
2423
{summary,run-plugins,describe,gen-plugin-config} ...
2524

2625
node scraper CLI
@@ -39,8 +38,7 @@ options:
3938
--sys-location {LOCAL,REMOTE}
4039
Location of target system (default: LOCAL)
4140
--sys-interaction-level {PASSIVE,INTERACTIVE,DISRUPTIVE}
42-
Specify system interaction level, used to determine the type of actions that plugins can perform (default:
43-
INTERACTIVE)
41+
Specify system interaction level, used to determine the type of actions that plugins can perform (default: INTERACTIVE)
4442
--sys-sku STRING Manually specify SKU of system (default: None)
4543
--sys-platform STRING
4644
Specify system platform (default: None)
@@ -55,9 +53,43 @@ options:
5553
Change python log level (default: INFO)
5654
--gen-reference-config
5755
Generate reference config from system. Writes to ./reference_config.json. (default: False)
56+
--skip-sudo Skip plugins that require sudo permissions (default: False)
5857

5958
```
6059
60+
### Execution Methods
61+
62+
Node Scraper can operate in two modes: LOCAL and REMOTE, determined by the `--sys-location` argument.
63+
64+
- **LOCAL** (default): Node Scraper is installed and run directly on the target system. All data collection and plugin execution occur locally.
65+
- **REMOTE**: Node Scraper runs on your local machine but targets a remote system over SSH. In this mode, Node Scraper does not need to be installed on the remote system; all commands are executed remotely via SSH.
66+
67+
To use remote execution, specify `--sys-location REMOTE` and provide a connection configuration file with `--connection-config`.
68+
69+
#### Example: Remote Execution
70+
71+
```sh
72+
node-scraper --sys-name <remote_host> --sys-location REMOTE --connection-config ./connection_config.json run-plugins DmesgPlugin
73+
```
74+
75+
##### Example connection_config.json
76+
77+
```json
78+
{
79+
"InBandConnectionManager": {
80+
"hostname": "remote_host.example.com",
81+
"port": 22,
82+
"username": "myuser",
83+
"password": "mypassword",
84+
"key_filename": "/path/to/private/key"
85+
}
86+
}
87+
```
88+
89+
**Notes:**
90+
- If using SSH keys, specify `key_filename` instead of `password`.
91+
- The remote user must have permissions to run the requested plugins and access required files. If needed, use the `--skip-sudo` argument to skip plugins requiring sudo.
92+
6193
### Subcommmands
6294
6395
Plugins to run can be specified in two ways, using a plugin JSON config file or using the

‎nodescraper/connection/inband/inbandremote.py‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -57,6 +57,7 @@ def __init__(
5757
) -> None:
5858
self.ssh_params = ssh_params
5959
self.client = paramiko.SSHClient()
60+
self.client.load_system_host_keys()
6061
self.client.set_missing_host_key_policy(self.host_key_policy())
6162

6263
def connect_ssh(self):

‎nodescraper/interfaces/dataanalyzertask.py‎

Lines changed: 3 additions & 3 deletions
Original file line numberDiff line numberDiff line change
@@ -119,19 +119,19 @@ def __init_subclass__(cls, **kwargs: dict[str, Any]) -> None:
119119
raise TypeError(f"No data model set for {cls.__name__}")
120120

121121
if hasattr(cls, "analyze_data"):
122-
cls.analyze_data = analyze_decorator(cls.analyze_data)
122+
setattr(cls, "analyze_data", analyze_decorator(cls.analyze_data)) # noqa
123123

124124
@abc.abstractmethod
125125
def analyze_data(
126126
self,
127127
data: TDataModel,
128-
args: Optional[TAnalyzeArg | dict],
128+
args: Optional[TAnalyzeArg],
129129
) -> TaskResult:
130130
"""Analyze the provided data and return a TaskResult
131131
132132
Args:
133133
data (TDataModel): data to analyze
134-
args (Optional[TAnalyzeArg | dict]): Optional arguments for analysis, can be a model or dict
134+
args (Optional[TAnalyzeArg]): Optional arguments for analysis. Dicts will be handled in the decorator"
135135
136136
Returns:
137137
TaskResult: Task result containing the analysis outcome

‎nodescraper/interfaces/task.py‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -83,7 +83,7 @@ def max_event_priority_level(self, input_value: str | EventPriority):
8383
elif isinstance(input_value, int):
8484
value = EventPriority(input_value)
8585
elif isinstance(input_value, EventPriority):
86-
value: EventPriority = input_value
86+
value: EventPriority = input_value # type:ignore
8787
else:
8888
raise ValueError(f"Invalid type for max_event_priority_level: {type(input_value)}")
8989

‎nodescraper/models/event.py‎

Lines changed: 3 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -73,7 +73,8 @@ def validate_timestamp(cls, timestamp: datetime.datetime) -> datetime.datetime:
7373
if timestamp.tzinfo is None or timestamp.tzinfo.utcoffset(timestamp) is None:
7474
raise ValueError("datetime must be timezone aware")
7575

76-
if timestamp.utcoffset() is not None and timestamp.utcoffset().total_seconds() != 0:
76+
utc_offset = timestamp.utcoffset()
77+
if utc_offset is not None and utc_offset.total_seconds() != 0:
7778
timestamp = timestamp.astimezone(datetime.timezone.utc)
7879

7980
return timestamp
@@ -90,7 +91,7 @@ def validate_category(cls, category: str | Enum) -> str:
9091
if isinstance(category, Enum):
9192
category = category.value
9293

93-
category = category.strip().upper()
94+
category = str(category).strip().upper()
9495
category = re.sub(r"[\s-]", "_", category)
9596
return category
9697

‎nodescraper/pluginexecutor.py‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -194,8 +194,6 @@ def run_queue(self) -> list[PluginResult]:
194194
self.logger.exception("Unexpected exception running plugin queue: %s", str(e))
195195
finally:
196196
self.logger.info("Closing connections")
197-
for connection_manager in self.connection_library.values():
198-
connection_manager.disconnect()
199197

200198
if self.plugin_config.result_collators:
201199
self.logger.info("Running result collators")
@@ -217,6 +215,8 @@ def run_queue(self) -> list[PluginResult]:
217215
],
218216
**collator_args,
219217
)
218+
for connection_manager in self.connection_library.values():
219+
connection_manager.disconnect()
220220

221221
return plugin_results
222222

‎nodescraper/plugins/inband/dmesg/collector_args.py‎

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -28,4 +28,11 @@
2828

2929

3030
class DmesgCollectorArgs(CollectorArgs):
31+
"""Collector args
32+
33+
Args:
34+
CollectorArgs (CollectorArgs): specific dmesg collector args
35+
"""
36+
37+
collect_rotated_logs: bool = False
3138
skip_sudo: bool = False

0 commit comments

Comments
 (0)