Skip to content

edgedelta/ed-o365-audit-eventhub

BranchesTags

Repository files navigation

Office 365 Audit Log Integration with Azure Event Hub and EdgeDelta

This solution collects Office 365 audit logs and forwards them to Azure Event Hub using Azure Container Instances for production deployment, with a local bash script for testing and development.

Solution Overview

This repository provides:

  1. Azure Container Instance Deployment - Production-ready scheduled log collection
  2. Local Bash Script (get_o365_logs.sh) - Development and testing tool

Required Configuration Variables

IMPORTANT: Before proceeding, define these variables with your actual values:

# Azure Resource Configuration
export RESOURCE_GROUP="your-resource-group"
export LOCATION="eastus"
export ACR_NAME="your-org-registry$(date +%s | tail -c 6)"  # Must be globally unique

# Office 365 API Credentials (from Azure AD App Registration)
export TENANT_ID="your-tenant-id-here"
export CLIENT_ID="your-client-id-here"
export CLIENT_SECRET="your-client-secret-here"

# Azure Event Hub Configuration
export EVENTHUB_NAMESPACE="your-eventhub-namespace"
export EVENTHUB_NAME="your-eventhub-name"
export EVENTHUB_AUTH_RULE="your-auth-rule-name"

# Container Configuration
export CONTAINER_NAME="o365-audit-collector"

This document provides a comprehensive guide covering setup, configuration, execution, and the underlying concepts of how these Microsoft services interact.

How It Works: An Overview

To understand how the script works, it's helpful to use an analogy. Imagine Microsoft 365 is a massive, high-security office building where all your company's digital work happens.

  • Entra ID (Azure AD) is the Security & ID Card Office in the lobby. Its only job is to manage identity. It issues ID cards (Client ID) and secret passwords (Client Secret) to applications (like our script) and grants them keys (API Permissions) to specific doors.

  • Microsoft Purview is the Central Surveillance & Compliance Department. This department manages the building's entire security camera system (the Unified Audit Log). By default, to save resources, all cameras are turned OFF. Your first step is to go to Purview and flip the master switch to "Start recording user and admin activity."

  • The Office 365 Management API is the Building's Front Desk. It's the single, public-facing point of contact for requesting data. When our script approaches the front desk, it checks its access token (provided by Entra ID) to see what it's allowed to ask for.

The Flow of Data

The following diagram illustrates the journey our script takes to get the logs:

graph TD
    subgraph YourEnv ["Your Environment"]
        A["Shell Script<br/>(get_o365_logs.sh)"]
    end

    subgraph MicrosoftCloud ["Microsoft Cloud"]
        B["Entra ID (Azure AD)<br/>Security & ID Card Office"]
        C["Office 365 Management API<br/>Building Front Desk"]
        D["Microsoft Purview<br/>Central Surveillance Room"]
        E["Microsoft 365 Services<br/>(SharePoint, Exchange, etc.)<br/>Office Floors"]
    end

    A -->|"Step 1 - Request Token"| B
    B -->|"Step 2 - Return Token"| A
    A -->|"Step 3 - Request Logs"| C
    C -->|"Step 4 - Check Audit Status"| D
    D -->|"Step 5 - Return Log Data"| C
    E -->|"User & Admin Activity"| D
    C -->|"Step 6 - Return Logs"| A

    classDef yourEnv fill:#e0f7fa,stroke:#008080,stroke-width:2px
    classDef microsoftService fill:#f5f5f5,stroke:#888,stroke-width:2px
    
    class A,E yourEnv
    class B,C,D microsoftService
Loading
  1. The script authenticates with Entra ID using its credentials.
  2. Entra ID validates the credentials and returns a temporary Access Token.
  3. The script presents this token to the O365 Management API.
  4. The API asks Purview if auditing is enabled for the tenant.
  5. Purview, which has been collecting logs from all M365 services, confirms it is active and provides the data.
  6. The API passes the log data back to the script, which saves it as JSON files.

Prerequisites

Before running the script, ensure the following requirements are met.

1. Azure App Registration

You need an application registered in Microsoft Entra ID with the following API permissions granted and consented to by an administrator:

  • ActivityFeed.Read
  • ActivityFeed.ReadDlp (Optional, for DLP logs)
  • ServiceHealth.Read (Optional)

2. Unified Audit Logging Enabled

As mentioned, you must enable audit logging for your organization.

  • Navigate to the Microsoft Purview compliance portal.
  • If a banner is present, click "Start recording user and admin activity."
  • IMPORTANT: After enabling, you may need to wait 24-48 hours for the service to be fully provisioned across all of Microsoft's backend systems. The script will fail with a Tenant ... does not exist error until this process is complete.

3. jq Command-Line Tool

The script uses jq to parse JSON responses from the API. You must have it installed.

# On macOS
brew install jq

# On Debian/Ubuntu
sudo apt-get install jq

Configuration

Open get_o365_logs.sh and edit the configuration variables at the top of the file.

  1. Credentials:

    • TENANT_ID: Your Directory (tenant) ID from the Entra app overview page.
    • CLIENT_ID: Your Application (client) ID from the Entra app overview page.
    • CLIENT_SECRET: The value of the client secret you generated in Entra.

  2. API Endpoint:

    • Most users will use the default Commercial / GCC endpoint. If your organization is on a government plan, you must comment out the default lines and uncomment the block for your specific plan (GCC High or DoD).

  3. Content Type:

    • CONTENT_TYPE: The type of log you want to retrieve. Common values include:
      • Audit.General
      • Audit.AzureActiveDirectory
      • Audit.Exchange
      • Audit.SharePoint
      • DLP.All

Usage

  1. Make the script executable: 
    chmod +x get_o365_logs.sh
  2. Run the script from its directory: 
    ./get_o365_logs.sh

The script will fetch the available log data for the last 24 hours and save it as a series of .json files in the current directory.

Troubleshooting

The most common error you will encounter is: Error: Starting subscription failed with HTTP status 400 Response: {\"error\":{\"code\":\"...\",\"message\":\"... Tenant ... does not exist.\"}}

This error almost always means that you need to wait longer for the Unified Audit Log service to finish provisioning after you enabled it in Purview.

To check the status, go to the Purview Audit Search page and run a manual search. If the search works there, your script should also work. If the search page shows an error or a banner about provisioning, you must continue to wait.

Local Development and Testing

Bash Script for Testing

Use Case: Local testing, development, and one-time log collection

The get_o365_logs.sh script downloads Office 365 audit logs to local JSON files for testing.

Configuration:

# Edit configuration in get_o365_logs.sh or set environment variables
export TENANT_ID=\"your-tenant-id\"
export CLIENT_ID=\"your-client-id\" 
export CLIENT_SECRET=\"your-client-secret\"

Execution:

chmod +x get_o365_logs.sh
./get_o365_logs.sh

Output: JSON files with audit logs in the current directory

Supported Log Types

All implementations collect the following Office 365 audit log types:

  • Audit.AzureActiveDirectory - Azure AD sign-ins, user management
  • Audit.Exchange - Email, calendar, contacts activity
  • Audit.SharePoint - SharePoint and OneDrive activity
  • Audit.General - Teams, Power BI, and other services
  • DLP.All - Data Loss Prevention events

Sample Log Data

The solution collects rich audit data including:

{
  \"CreationTime\": \"2025-06-16T16:45:01\",
  \"Operation\": \"TeamsSessionStarted\", 
  \"UserId\": \"user@domain.com\",
  \"Workload\": \"MicrosoftTeams\",
  \"ClientIP\": \"200.251.55.246\",
  \"ExtraProperties\": [
    {\"Key\": \"ClientName\", \"Value\": \"skypeteams\"},
    {\"Key\": \"Country\", \"Value\": \"br\"}
  ]
}

Data Flow Architecture

The solution follows this data flow:

  1. Container authenticates with Office 365 Management API using Azure AD credentials
  2. Fetches available audit content from Management Activity API
  3. Downloads log data in JSON format from multiple content types
  4. Batches events and sends to Azure Event Hub
  5. Event Hub forwards data to downstream consumers (EdgeDelta, SIEM, etc.)

Integration with EdgeDelta

Once logs are in Azure Event Hub, configure EdgeDelta to consume them using the Azure Event Hub input processor:

inputs:
  - name: o365_audit_logs
    type: azure_event_hub
    connection_string: \"${EVENT_HUB_CONNECTION_STR}\"
    event_hub_name: \"${EVENT_HUB_NAME}\"
    consumer_group: \"$Default\"

This enables:

  • Real-time monitoring and alerting
  • Security analytics and threat detection
  • Compliance reporting
  • User behavior analytics

Security Best Practices

Secrets Management

  • Azure Key Vault: Store all secrets (client secrets, connection strings) in Key Vault
  • Managed Identity: Use managed identity for Key Vault access instead of service principal keys
  • Least Privilege: Grant minimal required permissions to managed identity
  • Secret Rotation: Regularly rotate client secrets and connection strings
  • Network Security: Restrict Key Vault access to specific networks when possible

Monitoring and Alerting

  • Monitor container execution logs via Azure Container Insights
  • Set up Event Hub metrics monitoring for throughput and errors
  • Configure alerts for authentication failures and API rate limiting
  • Track container restart patterns and resource utilization

Data Retention and Compliance

  • Configure Event Hub retention policies (1-7 days default)
  • Implement archival strategies for long-term storage
  • Consider data residency requirements for international deployments
  • Ensure compliance with organizational data retention policies

Production Deployment with Azure Container Instances

Azure Container Instances provides a reliable, serverless approach for production Office 365 log collection without storage dependencies.

Prerequisites

  1. Azure CLI and Docker installed
  2. Office 365 API credentials (see prerequisites above)
  3. Azure Event Hub for log forwarding
  4. Configuration variables defined (see Required Configuration Variables above)

Step 1: Create Azure Container Registry

# Create Azure Container Registry
az acr create \\
  --resource-group $RESOURCE_GROUP \\
  --name $ACR_NAME \\
  --sku Basic \\
  --location $LOCATION \\
  --admin-enabled true

# Get ACR login server
ACR_LOGIN_SERVER=$(az acr show --name $ACR_NAME --resource-group $RESOURCE_GROUP --query \"loginServer\" -o tsv)

Step 2: Build and Push Container Image

# Build the container image for x86_64 (Azure compatibility)
docker buildx build --platform linux/amd64 -t $ACR_LOGIN_SERVER/o365-audit-collector:latest . --push

# Login to ACR (if needed)
az acr login --name $ACR_NAME

Step 3: Configure Event Hub Permissions

# Update Event Hub authorization rule to include Send permissions
az eventhubs eventhub authorization-rule update \\
  --namespace-name $EVENTHUB_NAMESPACE \\
  --eventhub-name $EVENTHUB_NAME \\
  --name $EVENTHUB_AUTH_RULE \\
  --resource-group $RESOURCE_GROUP \\
  --rights Send Listen

# Get connection string for container configuration
EVENT_HUB_CONNECTION_STR=$(az eventhubs eventhub authorization-rule keys list \\
  --namespace-name $EVENTHUB_NAMESPACE \\
  --eventhub-name $EVENTHUB_NAME \\
  --name $EVENTHUB_AUTH_RULE \\
  --resource-group $RESOURCE_GROUP \\
  --query \"primaryConnectionString\" -o tsv)

Step 4: Create Azure Key Vault for Secrets Management

IMPORTANT: Use Azure Key Vault for secure secrets management in production:

# Create Key Vault
KEYVAULT_NAME=\"kv-o365-logs-$(date +%s | tail -c 6)\"
az keyvault create \\
  --name $KEYVAULT_NAME \\
  --resource-group $RESOURCE_GROUP \\
  --location $LOCATION

# Store secrets in Key Vault
az keyvault secret set --vault-name $KEYVAULT_NAME --name \"client-secret\" --value \"$CLIENT_SECRET\"
az keyvault secret set --vault-name $KEYVAULT_NAME --name \"eventhub-connection-string\" --value \"$EVENT_HUB_CONNECTION_STR\"

# Create managed identity for container
az identity create \\
  --name \"${CONTAINER_NAME}-identity\" \\
  --resource-group $RESOURCE_GROUP

# Get managed identity details
IDENTITY_ID=$(az identity show --name \"${CONTAINER_NAME}-identity\" --resource-group $RESOURCE_GROUP --query \"id\" -o tsv)
IDENTITY_CLIENT_ID=$(az identity show --name \"${CONTAINER_NAME}-identity\" --resource-group $RESOURCE_GROUP --query \"clientId\" -o tsv)

# Grant Key Vault access to managed identity
az keyvault set-policy \\
  --name $KEYVAULT_NAME \\
  --object-id $(az identity show --name \"${CONTAINER_NAME}-identity\" --resource-group $RESOURCE_GROUP --query \"principalId\" -o tsv) \\
  --secret-permissions get

Step 5: Deploy Container Instance with Managed Identity

# Get ACR credentials
ACR_USERNAME=$(az acr credential show --name $ACR_NAME --query \"username\" -o tsv)
ACR_PASSWORD=$(az acr credential show --name $ACR_NAME --query \"passwords[0].value\" -o tsv)

# Deploy container instance with managed identity
az container create \\
  --resource-group $RESOURCE_GROUP \\
  --name $CONTAINER_NAME \\
  --image $ACR_LOGIN_SERVER/o365-audit-collector:latest \\
  --registry-login-server $ACR_LOGIN_SERVER \\
  --registry-username $ACR_USERNAME \\
  --registry-password $ACR_PASSWORD \\
  --assign-identity $IDENTITY_ID \\
  --restart-policy OnFailure \\
  --os-type Linux \\
  --cpu 0.5 \\
  --memory 1 \\
  --environment-variables \\
    TENANT_ID=\"$TENANT_ID\" \\
    CLIENT_ID=\"$CLIENT_ID\" \\
    EVENT_HUB_NAME=\"$EVENTHUB_NAME\" \\
    AZURE_CLIENT_ID=\"$IDENTITY_CLIENT_ID\" \\
    KEYVAULT_NAME=\"$KEYVAULT_NAME\" \\
  --secure-environment-variables \\
    CLIENT_SECRET=\"$CLIENT_SECRET\" \\
    EVENT_HUB_CONNECTION_STR=\"$EVENT_HUB_CONNECTION_STR\"

Step 6: Verify Deployment

# Check container status
az container show \\
  --resource-group $RESOURCE_GROUP \\
  --name $CONTAINER_NAME \\
  --query \"{Name:name, State:instanceView.state, RestartPolicy:restartPolicy}\" \\
  --output table

# View container logs
az container logs \\
  --resource-group $RESOURCE_GROUP \\
  --name $CONTAINER_NAME \\
  --follow

Expected Output

When working correctly, you should see logs similar to:

2025-06-20 17:52:21,353 - INFO - Starting Office 365 log collection
2025-06-20 17:52:21,703 - INFO - Successfully retrieved access token
2025-06-20 17:52:27,705 - INFO - Found 1 content blobs for Audit.AzureActiveDirectory
2025-06-20 17:52:29,286 - INFO - Sent final batch of 9 records to Event Hub
2025-06-20 17:52:32,421 - INFO - Sent final batch of 20 records to Event Hub
2025-06-20 17:52:34,715 - INFO - Sent final batch of 29 records to Event Hub
2025-06-20 17:52:35,870 - INFO - Total events sent to Event Hub: 58
2025-06-20 17:52:35,870 - INFO - Execution time: 14.52 seconds

Manual Execution

To run the container manually:

# Start the container (if stopped)
az container start --resource-group $RESOURCE_GROUP --name $CONTAINER_NAME

# Restart the container
az container restart --resource-group $RESOURCE_GROUP --name $CONTAINER_NAME

Container Architecture

The Azure Container Instance approach provides:

  • No storage dependencies - Eliminates Azure Functions storage restrictions
  • Reliable execution - Container runs to completion and stops
  • Easy debugging - Full container logs available via Azure CLI
  • Cost effective - Pay only for execution time
  • Scalable - Can be easily scheduled or triggered externally

Production Deployment Checklist

  • Office 365 audit logging enabled in Microsoft Purview
  • Azure AD application created with proper API permissions
  • Event Hub namespace and hub created
  • Event Hub authorization rule has Send + Listen permissions
  • Container Registry created and accessible
  • Container image built for linux/amd64 platform
  • Container instance deployed with proper environment variables
  • Container execution validated with real data flow
  • Scheduling mechanism configured (Logic App, Azure Automation, etc.)

Scheduling Options

Option 1: Azure Logic App (Recommended)

Create a Logic App with a recurrence trigger to restart the container on schedule:

{
    \"definition\": {
        \"$schema\": \"https://schema.management.azure.com/providers/Microsoft.Logic/schemas/2016-06-01/workflowdefinition.json#\",
        \"triggers\": {
            \"Recurrence\": {
                \"recurrence\": {
                    \"frequency\": \"Minute\",
                    \"interval\": 5
                },
                \"type\": \"Recurrence\"
            }
        },
        \"actions\": {
            \"Restart_Container\": {
                \"type\": \"Http\",
                \"inputs\": {
                    \"method\": \"POST\",
                    \"uri\": \"https://management.azure.com/subscriptions/{subscription-id}/resourceGroups/{resource-group}/providers/Microsoft.ContainerInstance/containerGroups/{container-name}/restart\",
                    \"authentication\": {
                        \"type\": \"ManagedServiceIdentity\"
                    }
                }
            }
        }
    }
}

Option 2: Azure Automation Runbook

Create a PowerShell runbook that runs on schedule:

# Restart Azure Container Instance
$resourceGroupName = \"your-resource-group\"
$containerName = \"o365-audit-collector\"

try {
    Connect-AzAccount -Identity
    Restart-AzContainerGroup -ResourceGroupName $resourceGroupName -Name $containerName
    Write-Output \"Container $containerName restarted successfully\"
} catch {
    Write-Error \"Failed to restart container: $($_.Exception.Message)\"
}

Support and Troubleshooting

Common Issues

  1. "Tenant does not exist" error: Wait 24-48 hours after enabling audit logging in Purview
  2. Authentication failures: Verify client secret hasn't expired and app permissions are granted
  3. Event Hub send failures: Check authorization rule has Send + Listen permissions
  4. Container architecture errors: Ensure image is built for linux/amd64 platform

Logs and Monitoring

  • Container logs: az container logs --resource-group $RESOURCE_GROUP --name $CONTAINER_NAME
  • Event Hub metrics: Monitor in Azure Portal under Event Hub namespace
  • Container metrics: Available in Azure Portal under Container Instances

License

This project is provided as-is for demonstration and educational purposes. Please review and adapt the code according to your organization's security and compliance requirements.

About

Office 365 audit logs to Azure Event Hub integration with EdgeDelta

Resources

Readme
Activity
Custom properties

Stars

0 stars

Watchers

0 watching

Forks

0 forks
Report repository

Releases

No releases published

Packages

 
 
 

Contributors

Languages