Skip to content
This repository was archived by the owner on Oct 20, 2023. It is now read-only.

Commit 0b2336c

Browse files
GoldingAustinGoldingAustin
committed
Fix file upload handling duplicate file names
Update parser-guide documentation and run format on creation Fix potential errors initializing sub collections in operators and commands
1 parent bd5dfc4 commit 0b2336c

File tree

32 files changed

+381
-366
lines changed

32 files changed

+381
-366
lines changed

‎applications/client/src/views/Campaigns/Upload/NewCampaignDialog.tsx‎

Lines changed: 17 additions & 15 deletions
Original file line numberDiff line numberDiff line change
@@ -43,21 +43,23 @@ export const NewCampaignDialog = observer<NewCampaignDialogProps>(({ ...props })
4343
id="add-campaign-methods"
4444
renderActiveTabPanelOnly
4545
>
46-
{Array.from(store.graphqlStore.parserInfos.values(), (parserInfo, index) => (
47-
<Tab
48-
cy-test={`create-new-camp-${parserInfo.id}`}
49-
id={index}
50-
key={parserInfo.id}
51-
title={parserInfo?.uploadForm?.tabTitle}
52-
panel={
53-
!parserInfo?.uploadForm?.enabledInBlueTeam && store.appMeta.blueTeam ? (
54-
<BlueTeamSourceWarning css={shadowStyle} />
55-
) : (
56-
<ParserUploadForm parserInfo={parserInfo} onClose={props.onClose} css={shadowStyle} />
57-
)
58-
}
59-
/>
60-
))}
46+
{Array.from(store.graphqlStore.parserInfos.values())
47+
.sort((a) => (a.name.includes('Cobalt') ? -1 : 1))
48+
.map((parserInfo, index) => (
49+
<Tab
50+
cy-test={`create-new-camp-${parserInfo.id}`}
51+
id={index}
52+
key={parserInfo.id}
53+
title={parserInfo?.uploadForm?.tabTitle}
54+
panel={
55+
!parserInfo?.uploadForm?.enabledInBlueTeam && store.appMeta.blueTeam ? (
56+
<BlueTeamSourceWarning css={shadowStyle} />
57+
) : (
58+
<ParserUploadForm parserInfo={parserInfo} onClose={props.onClose} css={shadowStyle} />
59+
)
60+
}
61+
/>
62+
))}
6163
<Tab
6264
cy-test="upload-from-file"
6365
id={CampaignTabs.UPLOAD}

‎applications/client/src/views/Campaigns/Upload/ParserUploadForm.tsx‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -124,7 +124,7 @@ export const ParserUploadForm = observer<ParserUploadFormProps>(({ parserInfo, .
124124
const fileValidationFormData = new FormData();
125125
const allFiles = {};
126126
for (const file of this.originalFiles) {
127-
allFiles[file.name] = createDirectoryFile(file);
127+
allFiles[file.webkitRelativePath] = createDirectoryFile(file);
128128
fileValidationFormData.append(
129129
'file',
130130
new File([file.slice()], file.webkitRelativePath.replace(/\//g, ':'), {

‎applications/server/src/store/command-resolvers.ts‎

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -212,7 +212,9 @@ export class CommandTypeCountResolvers {
212212
hidden: boolean = false
213213
): Promise<CommandTypeCount[]> {
214214
const em = await connectToProjectEmOrFail(campaignId, ctx);
215-
const commands = await em.find(Command, beaconHidden(hidden), { populate: ['commandGroups'] });
215+
const commands = await em.find(Command, beaconHidden(hidden), {
216+
populate: ['commandGroups', 'commandGroups.annotations'],
217+
});
216218
const countObj = commands.reduce<Record<string, CountObjItem>>((acc, current) => {
217219
if (acc[current.inputText]) {
218220
acc[current.inputText] = {

‎docs/parser-guide/Create Parser Guide.md‎

Lines changed: 15 additions & 14 deletions
Original file line numberDiff line numberDiff line change
@@ -1,16 +1,17 @@
11
<!-- TOC -->
2-
* [Intro](#intro)
3-
* [Basic Requirements](#basic-requirements)
4-
* [`info`](#info)
5-
* [`parse-campaign`](#parse-campaign)
6-
* [Parsing a single file](#parsing-a-single-file)
7-
* [Line 1](#line-1)
8-
* [Line 2](#line-2)
9-
* [Line 3 & Output](#line-3--output)
10-
* [Line 4](#line-4)
11-
* [Parsing the log file](#parsing-the-log-file)
12-
* [Make Parser Executable](#make-parser-executable)
13-
* [Conclusion](#conclusion)
2+
3+
- [Intro](#intro)
4+
- [Basic Requirements](#basic-requirements)
5+
- [`info`](#info)
6+
- [`parse-campaign`](#parse-campaign)
7+
- [Parsing a single file](#parsing-a-single-file)
8+
- [Line 1](#line-1)
9+
- [Line 2](#line-2)
10+
- [Line 3 & Output](#line-3--output)
11+
- [Line 4](#line-4)
12+
- [Parsing the log file](#parsing-the-log-file)
13+
- [Make Parser Executable](#make-parser-executable)
14+
- [Conclusion](#conclusion)
1415
<!-- TOC -->
1516

1617
# Intro
@@ -43,7 +44,7 @@ Now let's get into the first command RedEye will send to your parser.
4344

4445
# `info`
4546

46-
The `info` command *informs* RedEye on our parsers configuration and customizes our parsers upload form within RedEye's UI. On startup, RedEye will send the `info` command (example: `custom-redeye-parser info`) to each parser it detects (or is configured to use).
47+
The `info` command _informs_ RedEye on our parsers configuration and customizes our parsers upload form within RedEye's UI. On startup, RedEye will send the `info` command (example: `custom-redeye-parser info`) to each parser it detects (or is configured to use).
4748
The info our parser needs to send includes the parser ID, display name, and upload form configuration. Here's the `info` command implemented for the `custom-redeye-parser` with some comments to explain each property:
4849

4950
```typescript
@@ -408,7 +409,7 @@ Now that we've gone over how to identify what data to extract and how to format
408409
import fileSystem from 'fs';
409410

410411
function parseLogFile() {
411-
const logPath = process.argv[4].replaceAll('"', ""); // Get the path and remove the quotes around it
412+
const logPath = process.argv[4].replaceAll('"', ''); // Get the path and remove the quotes around it
412413
const logFile = fileSystem.readFileSync(`${logPath}/c2-log-files.txt`, 'utf8'); // Read the log file from the path passed in the command
413414
const lines = logFile.split('\n'); // Split the log file into lines
414415
const output = {

‎docs/parser-guide/command-schemas/parser-validate-files.schema.json‎

Lines changed: 2 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -21,14 +21,14 @@
2121
}
2222
},
2323
"valid": {
24-
"description": "An array of valid file names",
24+
"description": "An array of valid file paths relative to the campaign root directory",
2525
"type": "array",
2626
"items": {
2727
"type": "string"
2828
}
2929
},
3030
"invalid": {
31-
"description": "An array of invalid file names",
31+
"description": "An array of invalid file paths relative to the campaign root directory",
3232
"type": "array",
3333
"items": {
3434
"type": "string"

‎docs/parser-guide/commands-api/info/enums/ServerDelineationTypes.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,22 +13,22 @@
1313

1414
### Database
1515

16-
• **Database** = ``"Database"``
16+
• **Database** = `"Database"`
1717

1818
server data not in any particular file/folder structure
1919

2020
#### Defined in
2121

22-
[upload-form.ts:54](https://github.com/cisagov/RedEye/blob/9f9475cf/parsers/parser-core/src/parser-info/upload-form.ts#L54)
22+
[upload-form.ts:54](https://github.com/cisagov/RedEye/blob/bd5dfc45/parsers/parser-core/src/parser-info/upload-form.ts#L54)
2323

24-
___
24+
---
2525

2626
### Folder
2727

28-
• **Folder** = ``"Folder"``
28+
• **Folder** = `"Folder"`
2929

3030
server data seperated into distinct folders
3131

3232
#### Defined in
3333

34-
[upload-form.ts:52](https://github.com/cisagov/RedEye/blob/9f9475cf/parsers/parser-core/src/parser-info/upload-form.ts#L52)
34+
[upload-form.ts:52](https://github.com/cisagov/RedEye/blob/bd5dfc45/parsers/parser-core/src/parser-info/upload-form.ts#L52)

‎docs/parser-guide/commands-api/info/enums/UploadType.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -13,23 +13,23 @@
1313

1414
### Directory
1515

16-
• **Directory** = ``"Directory"``
16+
• **Directory** = `"Directory"`
1717

1818
upload a directory
1919

2020
#### Defined in
2121

22-
[file-upload.ts:30](https://github.com/cisagov/RedEye/blob/9f9475cf/parsers/parser-core/src/parser-info/file-upload.ts#L30)
22+
[file-upload.ts:30](https://github.com/cisagov/RedEye/blob/bd5dfc45/parsers/parser-core/src/parser-info/file-upload.ts#L30)
2323

24-
___
24+
---
2525

2626
### File
2727

28-
• **File** = ``"File"``
28+
• **File** = `"File"`
2929

3030
upload a single file or a selection of multiple files
3131
Use this if data is in a single file like a json or csv or a selection of files like .pcap files
3232

3333
#### Defined in
3434

35-
[file-upload.ts:28](https://github.com/cisagov/RedEye/blob/9f9475cf/parsers/parser-core/src/parser-info/file-upload.ts#L28)
35+
[file-upload.ts:28](https://github.com/cisagov/RedEye/blob/bd5dfc45/parsers/parser-core/src/parser-info/file-upload.ts#L28)

‎docs/parser-guide/commands-api/info/enums/ValidationMode.md‎

Lines changed: 8 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -14,34 +14,34 @@
1414

1515
### FileExtensions
1616

17-
• **FileExtensions** = ``"FileExtensions"``
17+
• **FileExtensions** = `"FileExtensions"`
1818

1919
validate uploaded files in client by file extensions
2020

2121
#### Defined in
2222

23-
[upload-form.ts:61](https://github.com/cisagov/RedEye/blob/9f9475cf/parsers/parser-core/src/parser-info/upload-form.ts#L61)
23+
[upload-form.ts:61](https://github.com/cisagov/RedEye/blob/bd5dfc45/parsers/parser-core/src/parser-info/upload-form.ts#L61)
2424

25-
___
25+
---
2626

2727
### None
2828

29-
• **None** = ``"None"``
29+
• **None** = `"None"`
3030

3131
no validation
3232

3333
#### Defined in
3434

35-
[upload-form.ts:59](https://github.com/cisagov/RedEye/blob/9f9475cf/parsers/parser-core/src/parser-info/upload-form.ts#L59)
35+
[upload-form.ts:59](https://github.com/cisagov/RedEye/blob/bd5dfc45/parsers/parser-core/src/parser-info/upload-form.ts#L59)
3636

37-
___
37+
---
3838

3939
### Parser
4040

41-
• **Parser** = ``"Parser"``
41+
• **Parser** = `"Parser"`
4242

4343
validate uploaded files in server with parser, parser must implement "validate-files" command
4444

4545
#### Defined in
4646

47-
[upload-form.ts:63](https://github.com/cisagov/RedEye/blob/9f9475cf/parsers/parser-core/src/parser-info/upload-form.ts#L63)
47+
[upload-form.ts:63](https://github.com/cisagov/RedEye/blob/bd5dfc45/parsers/parser-core/src/parser-info/upload-form.ts#L63)

‎docs/parser-guide/commands-api/info/index.md‎

Lines changed: 5 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -25,21 +25,21 @@
2525

2626
### UploadValidation
2727

28-
Ƭ **UploadValidation**: { `validate`: [`None`](enums/ValidationMode.md#none) \| [`Parser`](enums/ValidationMode.md#parser) } \| { `acceptedExtensions`: `string`[] ; `validate`: [`FileExtensions`](enums/ValidationMode.md#fileextensions) }
28+
Ƭ **UploadValidation**: { `validate`: [`None`](enums/ValidationMode.md#none) \| [`Parser`](enums/ValidationMode.md#parser) } \| { `acceptedExtensions`: `string`[] ; `validate`: [`FileExtensions`](enums/ValidationMode.md#fileextensions) }
2929

3030
The validation mode for the upload form
3131

3232
**`Example`**
3333

3434
```ts
3535
// No validation, allow uploading any folder or files
36-
validate = { validate: ValidationMode.None }
36+
validate = { validate: ValidationMode.None };
3737
// Only allow files with specific file extensions
38-
validate = { validate: ValidationMode.FileExtensions, acceptedExtensions: ['txt', 'png', 'jpg'] }
38+
validate = { validate: ValidationMode.FileExtensions, acceptedExtensions: ['txt', 'png', 'jpg'] };
3939
// The parser has implemented the 'validate-files' command and will validate the folder of files
40-
validate = { validate: ValidationMode.Parser }
40+
validate = { validate: ValidationMode.Parser };
4141
```
4242

4343
#### Defined in
4444

45-
[upload-form.ts:76](https://github.com/cisagov/RedEye/blob/9f9475cf/parsers/parser-core/src/parser-info/upload-form.ts#L76)
45+
[upload-form.ts:76](https://github.com/cisagov/RedEye/blob/bd5dfc45/parsers/parser-core/src/parser-info/upload-form.ts#L76)

‎docs/parser-guide/commands-api/info/interfaces/FileDisplay.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -19,4 +19,4 @@ A user may want to change the name of a server to something more descriptive
1919

2020
#### Defined in
2121

22-
[upload-form.ts:85](https://github.com/cisagov/RedEye/blob/9f9475cf/parsers/parser-core/src/parser-info/upload-form.ts#L85)
22+
[upload-form.ts:85](https://github.com/cisagov/RedEye/blob/bd5dfc45/parsers/parser-core/src/parser-info/upload-form.ts#L85)

0 commit comments

Comments
 (0)