Azure DevOps Migration Platform – Reference Architecture (with Observability)

[MrHinsh]

Azure DevOps Migration Platform – Reference Architecture (with Observability)

✨ Overview

A resilient, resumable, and observable migration platform designed to move work items, Git repos, queries, and metadata from TFS or Azure DevOps to Azure DevOps using a tiered, agent-based architecture. Includes streaming logs to Azure Blob Storage and end-to-end observability via OpenTelemetry.

---

🔹 Core Components

1. devopsmigration.exe (CLI)

• Prepares, validates, and queues jobs.

• Interacts with the control plane.

• Emits OpenTelemetry logs, traces, and basic metrics.

2. Control Plane (ASP.NET Core API)

• Accepts job definitions.

• Queues jobs based on service tier.

• Assigns agents and manages state.

• Emits OpenTelemetry traces, logs, and metrics.

3. Agent (Docker / Self-hosted CLI)

• Polls control plane for jobs.

• Executes migrations from source to target.

• Streams logs to Azure Blob Storage.

• Emits detailed telemetry for each processor.

• Supports resumability via .resume.json and .map.db.

4. TfsExport.exe

• Legacy tool written in .NET Framework 4.x.

• Uses deprecated TFS Object Model APIs.

• Exports TFS or on-prem Azure DevOps instance to a local filesystem .zip package.

• Output includes full project data and history, covering:

  • Work items and revisions

  • Links and attachments

  • Area and iteration paths

  • Team and query definitions

  • Git repositories as .bundle files

• The .zip export structure follows a predictable format for agent ingestion and is uploaded to Azure Blob Storage.

Example .zip Structure:

project-export.zip
├── workitems/
│   ├── 2024/
│   │   ├── 01/
│   │   │   ├── 03/
│   │   │   │   ├── 13/
│   │   │   │   │   ├── 00/
│   │   │   │   │   │   ├── 2024-01-03-13-00-00-34563-1/
│   │   │   │   │   │   │   ├── revision.json
│   │   │   │   │   │   │   ├── links.json
│   │   │   │   │   │   │   └── attachments/
│   │   │   │   │   │   │       ├── attachment1.png
│   │   │   │   │   │   │       └── attachment2.docx
├── area-paths.json
├── iteration-paths.json
├── queries.json
├── teams.json
├── repos/
│   ├── RepoA.bundle

• Each revision is stored in its own folder, named using YYYY-MM-DD-HH-mm-SS-{WorkItemId}-{Revision}.

• To support scalability, these folders are hierarchically partitioned by timestamp (Year/Month/Day/Hour/Minute).

• Each revision folder includes:

  • revision.json – the revision's data

  • links.json – links specific to the revision

  • attachments/ – files added or modified in the revision

• This format enables strict historical ordering and efficient processing at massive scale.

5. Azure Blob Storage

• Stores logs and resume data per job.

• Append blobs for logs.

• Block blobs for checkpoint and map files.

• Stores logs and resume data per job.

• Append blobs for logs.

• Block blobs for checkpoint and map files.

---

📊 Job Lifecycle

1. Preparation

   • CLI validates config and packages job data.

2. Queueing

   • Job sent to control plane with trace context.

3. Assignment

   • Job placed in tier-specific queue.

   • Available agent is assigned.

4. Execution

   • Agent downloads job and executes processors.

   • Streams logs to blob and OpenTelemetry endpoint.

   • Checkpoints written regularly.

5. Completion or Preemption

   • Job finishes or is paused.

   • State saved for resume.

---

🌌 Observability (OpenTelemetry)

Telemetry Types

Component	Traces	Logs	Metrics
CLI	Job submission, validation	Structured CLI logs	Invocation count
Control Plane	Job routing, API span	Lifecycle and errors	Queue depth, active jobs
Agent	Processor spans, retries	Streamed JSON logs	Work items processed, durations

---

📊 Diagram (Text-Based)

CLI (devopsmigration.exe)
     |
     ▼
[ Control Plane API ]──▶ [ Azure Blob Storage ]
     |                          ▲
     ▼                          |
[ Tiered Queue ]               Logs + Checkpoints
     |                          |
     ▼                          ▼
[ Agent Pool ] ──▶ [ Azure DevOps | Filesystem Source ]
                         |
                         ▼
                [ Target Azure DevOps Org ]

---

🔒 Security Considerations

• Blob access via SAS tokens or MSI

• Control plane secured with Azure AD auth

• Logs scrubbed for PII before external transmission

---

🧼 Optional Enhancements

• Event Grid for job lifecycle events

• Key Vault for job-specific credentials

• Dashboard visualisation via Grafana + Prometheus

• Resilience via durable queues and retry policies

---

Would you like an architectural diagram image or a Visio/Draw.io export?