Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions .github/workflows/test-language-detector.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
name: Test Language Detector

on:
workflow_dispatch:
inputs:
repo:
description: 'Repository to test language detection on'
required: true
type: string
default: 'consensys/linea-monorepo'
languages_config:
description: 'Custom language config (JSON)'
required: false
type: string
default: ''

jobs:
test-language-detector:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Test Language Detector Action
id: test-detector
uses: ./packages/language-detector
with:
repo: ${{ inputs.repo }}
languages_config: ${{ inputs.languages_config }}

- name: Show Results
run: |
echo "🎯 Language Detection Results"
echo "============================"
echo "Languages: ${{ steps.test-detector.outputs.languages }}"
echo
echo "Matrix:"
echo '${{ steps.test-detector.outputs.matrix }}' | jq '.'

test-scripts-directly:
runs-on: ubuntu-latest
steps:
- name: Checkout
uses: actions/checkout@v4

- name: Run Script Tests
run: |
cd packages/language-detector
chmod +x test-scripts.sh
./test-scripts.sh
23 changes: 18 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -29,6 +29,7 @@ jobs:
```

The workflow will:

1. Auto-detect languages in your repository
2. Load repo-specific config from `repo-configs/` (or use defaults)
3. Run CodeQL and Semgrep scans in parallel
Expand All @@ -50,13 +51,16 @@ const config = {
build_mode: 'manual',
build_command: './gradlew build',
version: '21',
distribution: 'temurin'
}
distribution: 'temurin',
},
],
queries: [
{ name: 'Security queries', uses: './query-suites/base.qls' },
{ name: 'Custom queries', uses: './custom-queries/query-suites/custom-queries.qls' }
]
{
name: 'Custom queries',
uses: './custom-queries/query-suites/custom-queries.qls',
},
],
};

export default config;
Expand Down Expand Up @@ -93,7 +97,7 @@ jobs:
uses: metamask/security-codescanner-monorepo/.github/workflows/security-scan.yml@dev-branch
with:
repo: ${{ github.repository }}
ref: dev-branch # Must explicitly pass the branch name
ref: dev-branch # Must explicitly pass the branch name
```

**Note**: The `@branch` in the `uses:` statement only affects which workflow file is used. The `ref` input ensures all internal monorepo checkouts use the same branch.
Expand Down Expand Up @@ -184,6 +188,7 @@ yarn workspaces foreach run <command>
### Supported Languages

**CodeQL:**

- JavaScript/TypeScript → `javascript-typescript`
- Python → `python`
- Java/Kotlin → `java-kotlin`
Expand All @@ -197,43 +202,51 @@ yarn workspaces foreach run <command>
## 🎯 Key Features

### ✅ Automatic Language Detection

- Detects languages via GitHub API
- Maps to appropriate scanners
- Configurable per-repository

### ✅ Optimized Execution

- Parallel scanning per language
- Matrix-based job strategy
- Fail-fast for ignored languages

### ✅ Flexible Configuration

- File-based configs (single source of truth)
- Workflow input overrides
- Per-language build settings

### ✅ Security First

- Minimal token permissions (`contents: read`, `security-events: write`)
- Input validation and sanitization
- See [SECURITY.md](./SECURITY.md) for threat model

## 🔍 Troubleshooting

### Language not detected

- Check GitHub's language detection (repo insights → languages)
- Ensure language is in `LANGUAGE_MAPPING` in `language-detector/src/job-configurator.js`
- Add manual `languages_config` in workflow input

### Build failures

- Verify `build_command` in repo config
- Check if correct `version` and `distribution` are specified
- Review CodeQL build logs in Actions

### Config not loading

- Repo config filename must match repo name: `owner/repo` → `repo.js`
- Ensure config file exports with `export default config`
- Check config-loader logs in workflow output

### Permissions errors

- Add required permissions to calling workflow:
```yaml
permissions:
Expand Down
85 changes: 50 additions & 35 deletions packages/codeql-action/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ Custom CodeQL analysis action with repository-specific configurations and custom
## Overview

This action provides flexible CodeQL scanning with:

- **Automatic language detection** via GitHub API
- **Repository-specific configurations** in `repo-configs/`
- **Custom query suites** for specialized security analysis
Expand All @@ -21,16 +22,16 @@ The action works as part of the security scanning workflow:

## Inputs

| Input | Required | Description |
|-------|----------|-------------|
| `repo` | ✅ | Repository name (format: `owner/repo`) |
| `language` | ✅ | Language to scan (e.g., `javascript-typescript`, `java-kotlin`, `python`) |
| `paths_ignored` | ❌ | Newline-delimited paths to ignore |
| `rules_excluded` | ❌ | Newline-delimited CodeQL rule IDs to exclude |
| `build_mode` | ❌ | Build mode: `none`, `autobuild`, or `manual` |
| `build_command` | ❌ | Build command for `manual` build mode |
| `version` | ❌ | Language/runtime version (e.g., `21` for Java, `3.10` for Python) |
| `distribution` | ❌ | Distribution (e.g., `temurin`, `zulu` for Java) |
| Input | Required | Description |
| ---------------- | -------- | ------------------------------------------------------------------------- |
| `repo` | ✅ | Repository name (format: `owner/repo`) |
| `language` | ✅ | Language to scan (e.g., `javascript-typescript`, `java-kotlin`, `python`) |
| `paths_ignored` | ❌ | Newline-delimited paths to ignore |
| `rules_excluded` | ❌ | Newline-delimited CodeQL rule IDs to exclude |
| `build_mode` | ❌ | Build mode: `none`, `autobuild`, or `manual` |
| `build_command` | ❌ | Build command for `manual` build mode |
| `version` | ❌ | Language/runtime version (e.g., `21` for Java, `3.10` for Python) |
| `distribution` | ❌ | Distribution (e.g., `temurin`, `zulu` for Java) |

## Usage

Expand Down Expand Up @@ -87,23 +88,26 @@ const config = {
build_mode: 'manual',
build_command: './gradlew :coordinator:app:build',
version: '21',
distribution: 'temurin'
distribution: 'temurin',
},
{
language: 'javascript-typescript',
// Uses default config (no build needed)
},
{
language: 'cpp',
ignore: true // Skip C++ scanning
}
ignore: true, // Skip C++ scanning
},
],

// CodeQL query suites
queries: [
{ name: 'Base security queries', uses: './query-suites/base.qls' },
{ name: 'Custom queries', uses: './custom-queries/query-suites/custom-queries.qls' }
]
{
name: 'Custom queries',
uses: './custom-queries/query-suites/custom-queries.qls',
},
],
};

export default config;
Expand All @@ -121,45 +125,49 @@ The action includes sensible defaults for common languages:

```javascript
const DEFAULT_CONFIGS = {
'javascript': { language: 'javascript-typescript' },
'typescript': { language: 'javascript-typescript' },
'python': { language: 'python' },
'go': { language: 'go' },
'java': {
javascript: { language: 'javascript-typescript' },
typescript: { language: 'javascript-typescript' },
python: { language: 'python' },
go: { language: 'go' },
java: {
language: 'java-kotlin',
build_mode: 'manual',
build_command: './mvnw compile'
build_command: './mvnw compile',
},
'cpp': { language: 'cpp' },
'csharp': { language: 'csharp' },
'ruby': { language: 'ruby' }
cpp: { language: 'cpp' },
csharp: { language: 'csharp' },
ruby: { language: 'ruby' },
};
```

## Supported Languages

| GitHub Language | CodeQL Language | Build Required |
|-----------------|-----------------|----------------|
| JavaScript | `javascript-typescript` | No |
| TypeScript | `javascript-typescript` | No |
| Python | `python` | No |
| Java | `java-kotlin` | Yes (defaults to `./mvnw compile`) |
| Kotlin | `java-kotlin` | Yes |
| Go | `go` | No |
| C/C++ | `cpp` | Yes |
| C# | `csharp` | Yes |
| Ruby | `ruby` | No |
| GitHub Language | CodeQL Language | Build Required |
| --------------- | ----------------------- | ---------------------------------- |
| JavaScript | `javascript-typescript` | No |
| TypeScript | `javascript-typescript` | No |
| Python | `python` | No |
| Java | `java-kotlin` | Yes (defaults to `./mvnw compile`) |
| Kotlin | `java-kotlin` | Yes |
| Go | `go` | No |
| C/C++ | `cpp` | Yes |
| C# | `csharp` | Yes |
| Ruby | `ruby` | No |

## Build Modes

### `none`

No build needed (interpreted languages like JavaScript, Python)

### `autobuild`

CodeQL automatically detects and runs build (works for simple projects)

### `manual`

Specify exact build command:

```javascript
{
language: 'java-kotlin',
Expand All @@ -173,38 +181,45 @@ Specify exact build command:
Query suites define which CodeQL queries to run:

**Built-in suites:**

- `./query-suites/base.qls` - Standard security queries
- `./query-suites/linea-monorepo.qls` - Project-specific queries

**Custom queries:**

- Checked out from `metamask/CodeQL-Queries` repository
- Available at `./custom-queries/query-suites/custom-queries.qls`

## Troubleshooting

### Config not loading

- Filename must match repo: `owner/repo` → `repo.js`
- Must use ESM: `export default config`
- Check logs: `[config-loader] Loading config for repository: ...`

### Build failures

- Verify `build_command` works locally
- Check Java version matches `version` input
- Review build step logs in Actions

### Language not detected

- Check GitHub language stats (repo → Insights → Languages)
- Add language manually via `languages_config` in repo config
- Verify language mapping in `language-detector/src/job-configurator.js`

### SARIF upload errors

- Ensure workflow has `security-events: write` permission
- Check SARIF file is generated in `${{ steps.codeql-analysis.outputs.sarif-output }}`
- Review CodeQL analysis logs

## Security

See [SECURITY.md](../../SECURITY.md) for:

- Threat model and security boundaries
- Input validation approach
- Token permissions model
Expand Down
5 changes: 4 additions & 1 deletion packages/codeql-action/__tests__/config-loader.test.js
Original file line number Diff line number Diff line change
Expand Up @@ -66,7 +66,10 @@ describe('config-loader', () => {
try {
// Write a config that will cause an error when evaluated (but is valid JS)
const badConfigPath = path.join(tempDir, 'errorconfig.js');
fs.writeFileSync(badConfigPath, 'throw new Error("Config error"); export default {};');
fs.writeFileSync(
badConfigPath,
'throw new Error("Config error"); export default {};',
);

const config = await loadRepoConfig('owner/errorconfig', tempDir);

Expand Down
Loading
Loading