You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
This tool analyzes Azure Sentinel Solutions to extract and map data connector definitions to their ingestion tables, producing comprehensive CSV reports for solution metadata analysis.
3
+
This directory contains two complementary tools for analyzing Microsoft Sentinel Solutions:
4
+
5
+
1.**`solution_connector_tables.py`** - Extracts and maps data connector definitions to their ingestion tables, producing CSV reports with solution metadata
6
+
2.**`generate_connector_docs.py`** - Generates browsable markdown documentation from the CSV data with AI-rendered setup instructions
4
7
5
8
## Quick Start
6
9
7
-
**Pre-generated CSV files and documentation are already available in this directory:**
8
-
-`solutions_connectors_tables_mapping.csv` - Main mapping of connectors to tables with full metadata
9
-
-`solutions_connectors_tables_issues_and_exceptions_report.csv` - Issues and exceptions report
10
-
-[`connector-docs/`](connector-docs/) - [Microsoft Sentinel Data Connector Reference](connector-docs/README.md) with browsable indexes by solutions, connectors, and tables
10
+
**Pre-generated files are already available in this directory:**
11
+
-[`solutions_connectors_tables_mapping.csv`](solutions_connectors_tables_mapping.csv) - Main mapping of connectors to tables with full metadata
12
+
-[`solutions_connectors_tables_issues_and_exceptions_report.csv`](solutions_connectors_tables_issues_and_exceptions_report.csv) - Issues and exceptions report
11
13
12
-
You can use these files directly without running the script. They are kept up-to-date with the Solutions directory.
14
+
You can use these files directly without running the scripts. They are kept up-to-date with the Solutions directory.
13
15
14
-
To regenerate the files with the latest data:
15
-
```bash
16
-
python solution_connector_tables.py
17
-
```
16
+
---
18
17
19
-
To regenerate the markdown documentation:
20
-
```bash
21
-
python generate_connector_docs.py
22
-
```
18
+
# 1. Solution Connector Tables Analyzer
19
+
20
+
**Script:**`solution_connector_tables.py`
23
21
24
22
## Overview
25
23
26
-
The analyzer scans the Solutions directory to:
24
+
Scans the Solutions directory to:
27
25
- Extract table references from connector JSON files (queries, sample queries, data types)
28
26
- Resolve parser function references to actual tables
29
27
- Flatten solution metadata from SolutionMetadata.json files
@@ -33,9 +31,8 @@ The analyzer scans the Solutions directory to:
33
31
34
32
**Note:** Solutions without data connectors are included in the CSV output with empty `connector_id`, `connector_title`, `connector_description`, `connector_publisher`, `connector_files`, and `Table` fields. This ensures complete solution coverage in the documentation while clearly indicating which solutions do not include data ingestion components.
35
33
36
-
## Installation and Requirements
34
+
## Prerequisites
37
35
38
-
### Prerequisites
39
36
- Python 3.7 or higher
40
37
- No external dependencies required (optional: json5 for enhanced JSON parsing)
41
38
@@ -44,7 +41,8 @@ The analyzer scans the Solutions directory to:
44
41
pip install json5 # For improved JSON parsing with comments and trailing commas
The main CSV file containing one row per unique combination of solution, connector, and table.
96
82
97
-
**Note:** Newlines in the `connector_description`field are replaced with `<br>` tags to ensure proper rendering in GitHub's CSV viewer while preserving formatting information.
83
+
**Note:** Newlines in the `connector_description`and `connector_permissions` fields are replaced with `<br>` tags to ensure proper rendering in GitHub's CSV viewer. The `connector_instruction_steps` field uses standard JSON encoding with `\n` for newlines as it contains JSON-formatted data.
98
84
99
85
#### Column Descriptions
100
86
@@ -117,6 +103,8 @@ The main CSV file containing one row per unique combination of solution, connect
117
103
|`connector_publisher`| Connector publisher name. Empty for solutions without data connectors. |
118
104
|`connector_title`| Connector display title. Empty for solutions without data connectors. |
119
105
|`connector_description`| Connector description (newlines replaced with `<br>` for GitHub CSV rendering). Empty for solutions without data connectors. |
106
+
|`connector_instruction_steps`| Setup and configuration instructions rendered from connector UI definitions using AI, stored as JSON-encoded string with standard JSON `\n` encoding. May contain inaccuracies - verify in portal. Empty for solutions without data connectors. |
107
+
|`connector_permissions`| Required permissions and prerequisites in markdown format (newlines replaced with `<br>`). Empty for solutions without data connectors. |
120
108
|`connector_files`| Semicolon-separated list of GitHub URLs to connector definition files. Empty for solutions without data connectors. |
121
109
|`is_unique`|`true` if table appears in only one connector file, `false` otherwise |
122
110
|`table_detection_methods`| (Optional, with --show-detection-methods) Semicolon-separated list of methods used to detect this table |
Generates browsable markdown documentation from the CSV data produced by `solution_connector_tables.py`. The documentation includes:
255
+
256
+
- Three index pages (solutions, connectors, tables)
257
+
- Individual pages for each solution with connector details
258
+
- Individual pages for each connector with usage information
259
+
-**AI-rendered setup instructions** extracted from connector UI definitions
260
+
261
+
## Output
262
+
263
+
The script generates the **Microsoft Sentinel Data Connector Reference** documentation in the `connector-docs/` directory:
264
+
265
+
-**[Solutions Index](connector-docs/solutions-index.md)** - All solutions organized alphabetically (with and without connectors)
266
+
-**[Connectors Index](connector-docs/connectors-index.md)** - All unique connectors with metadata
267
+
-**[Tables Index](connector-docs/tables-index.md)** - All unique tables with solution references
268
+
-**Individual Solution Pages** - Detailed pages for each solution with connector and table information (in [`solutions/`](connector-docs/solutions/) directory)
269
+
-**Individual Connector Pages** - Detailed pages for each connector with usage information (in [`connectors/`](connector-docs/connectors/) directory)
270
+
271
+
See the [connector-docs README](connector-docs/README.md) for full documentation.
272
+
273
+
## Prerequisites
274
+
275
+
- Python 3.7 or higher
276
+
- Pre-generated CSV file from `solution_connector_tables.py`
277
+
- No external dependencies required
278
+
279
+
## Running the Script
280
+
281
+
From the `Tools/Solutions Analyzer` directory:
266
282
267
-
After modifications, test with:
268
283
```bash
269
-
cd"Tools/Solutions Analyzer"
270
-
python solution_connector_tables.py
284
+
python generate_connector_docs.py
285
+
```
286
+
287
+
The script reads `solutions_connectors_tables_mapping.csv` and generates all documentation in the `connector-docs/` directory.
288
+
289
+
## Output Structure
290
+
291
+
The generated documentation is organized as:
292
+
293
+
```
294
+
connector-docs/
295
+
├── README.md # Documentation guide
296
+
├── solutions-index.md # Alphabetical list of all solutions
297
+
├── connectors-index.md # Alphabetical list of all connectors
298
+
├── tables-index.md # Alphabetical list of all tables
- Setup instructions for each connector (AI-rendered)
315
+
- Required permissions and prerequisites
316
+
- Tables ingested by each connector
317
+
- Links to connector definition files
318
+
319
+
**Connector Pages** include:
320
+
- Connector description and metadata
321
+
-**AI-rendered setup instructions** from connector UI definitions with step-by-step guidance
322
+
- Required permissions and prerequisites
323
+
- List of solutions using this connector
324
+
- Tables ingested by the connector
325
+
- Links to GitHub connector definition files
326
+
327
+
**Index Pages** provide:
328
+
- Alphabetical navigation
329
+
- Quick statistics
330
+
- Cross-references between solutions, connectors, and tables
331
+
332
+
## AI-Rendered Setup Instructions
333
+
334
+
The "Setup Instructions" sections in the generated connector documentation are **automatically rendered from connector UI definition files using AI**. These instructions interpret the UI-centric JSON structures that define the Azure Portal configuration interface and convert them into readable step-by-step guidance.
335
+
336
+
### ⚠️ Important Disclaimer
337
+
338
+
**These AI-rendered instructions may not be fully accurate.** They are generated by interpreting UI definition metadata and should always be verified against the actual Microsoft Sentinel portal before implementation. The instructions provide a helpful starting point but are not a substitute for official documentation or hands-on portal verification.
274
339
275
-
When adding new detection methods or modifying the logic:
276
-
1. Update the `table_detection_methods` tracking in `record_table()` function
277
-
2. Test with `--show-detection-methods` flag to verify detection sources
278
-
3. Update this README with new detection methods or column descriptions
1.**JSON Parsing**: The script extracts `instructionSteps` arrays from connector definition files in the Solutions directory
345
+
2.**UI Type Detection**: Each instruction step has a `type` property (e.g., `DataConnectorsGrid`, `ContextPane`, `GCPGrid`) that determines how it should be interpreted
346
+
3.**AI-Powered Rendering**: Specialized handlers for each UI type convert the JSON structure into descriptive markdown:
347
+
- Form fields (textboxes, dropdowns) are described with their purposes and validation requirements
348
+
- Management grids and data selectors are explained with their configuration options
349
+
- Portal-only interfaces are identified and marked with clear indicators
350
+
4.**Markdown Formatting**: The rendered content is formatted with emoji indicators, step numbers, and disclaimers
351
+
352
+
### UI Types Supported
353
+
354
+
The script includes specialized handlers for connector UI configuration types based on the [official Microsoft Sentinel data connector UI definitions reference](https://learn.microsoft.com/en-us/azure/sentinel/data-connector-ui-definitions-reference#instructionsteps):
355
+
356
+
**Standard Instruction Types:**
357
+
358
+
-**OAuthForm**: OAuth authentication forms with client credentials
359
+
-**Textbox**: Input fields for text, passwords, numbers, and email addresses
360
+
-**Dropdown**: Selection lists with single or multi-select options
361
+
-**Markdown**: Formatted text content with links and formatting
362
+
-**CopyableLabel**: Text fields with copy-to-clipboard functionality
363
+
-**InfoMessage**: Inline information messages with contextual help
-**Setup Instructions** - Step-by-step configuration guidance rendered from connector UI definitions using AI
33
+
- ⚠️ **Note**: Instructions are automatically rendered from the user interface definition files using AI and may not be fully accurate. Always verify configuration steps in the Microsoft Sentinel portal.
34
+
- Solutions that include this connector
35
+
- Links to connector definition files on GitHub
36
+
23
37
## 📊 Quick Statistics
24
38
25
39
For current statistics, see the [Solutions Index](solutions-index.md) which displays up-to-date counts of solutions (with and without connectors), connectors, and tables.
@@ -64,6 +78,17 @@ This documentation is automatically generated from the Solutions Analyzer tool,
64
78
65
79
The analyzer identifies table references in connector configurations and parser logic to create comprehensive mappings.
66
80
81
+
### AI-Generated Instructions
82
+
83
+
**Setup Instructions** in connector documentation are automatically extracted from connector UI definition files using AI:
- Converts JSON UI definitions to readable markdown format
86
+
- Generates step-by-step configuration guidance
87
+
- Describes form fields, dropdowns, and management interfaces
88
+
- Marks portal-only features with visual indicators
89
+
90
+
⚠️ **Important**: AI-generated instructions may not be fully accurate. Always verify all configuration steps in the Microsoft Sentinel portal before implementation.
91
+
67
92
## 📝 Data Source
68
93
69
94
The documentation is based on analysis of the `solutions_connectors_tables_mapping.csv` file, which contains:
0 commit comments