Skip to content

Commit da137cc

Browse files
przlplxprzlplx
authored
AB#6858: Create Troubleshooting Guide: Windows Server Data Deduplication (#9969)
* Create troubleshootwindows-server-data-deduplication * Create troubleshootwindows-server-data-deduplication.md * Update troubleshootwindows-server-data-deduplication.md * Update troubleshootwindows-server-data-deduplication.md * Delete support/windows-server/backup-and-storage/troubleshootwindows-server-data-deduplication * Revise title and description in troubleshooting guide Updated the title and description for clarity and added links to the references section. * Change 'Introduction' to 'Summary' in documentation Updated the section title from 'Introduction' to 'Summary' for clarity.
1 parent d4bb699 commit da137cc

File tree

1 file changed

+284
-0
lines changed

1 file changed

+284
-0
lines changed

‎support/windows-server/backup-and-storage/troubleshootwindows-server-data-deduplication.md‎

Lines changed: 284 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,284 @@
1+
---
2+
title: Troubleshoot Windows Server Data Deduplication
3+
description: Provides troubleshooting methods for issues that affect Windows Server data deduplication.
4+
ms.date: 10/08/2025
5+
manager: dcscontentpm
6+
audience: itpro
7+
ms.topic: troubleshooting
8+
ms.reviewer: kaushika
9+
ms.custom:
10+
- sap:Backup, Recovery, Disk, and Storage\Deduplication
11+
- pcy:WinComm Storage High Avail
12+
appliesto:
13+
- <a href=https://learn.microsoft.com/windows/release-health/windows-server-release-info target=_blank>Supported versions of Windows Server</a>
14+
---
15+
16+
# Troubleshoot Windows Server data deduplication
17+
18+
## Summary
19+
20+
Windows Server data deduplication is a widely used feature for optimizing disk space by removing redundant data blocks and storing only unique information. Although deduplication offers significant storage savings, it also introduces complexity that can cause a variety of failure modes, such as job failures, space reclamation issues, file access problems, high chunkstore usage, and errors in migration scenarios.
21+
22+
This article provides a methodical approach to diagnosing and resolving common data deduplication problems in Windows Server 2022, 2019, and 2016 across NTFS and ReFS file systems.
23+
24+
## Troubleshooting checklist
25+
26+
Use this checklist for systematic troubleshooting:
27+
28+
- **Validate disk space and free space**
29+
- Make sure that there is at least 10 free free space on deduplicated volumes. Chunk storage should not exceed 80-90 percent usage.
30+
- **Check system resources**
31+
- Verify that there is sufficient RAM (minimum 1 GB or 1 TB logical data) and adequate CPU, and that memory and CPU thresholds aren’t exceeded.
32+
- **Review deduplication job status**
33+
- Use PowerShell to check whether jobs are running, stuck, or failing.
34+
- **Examine event logs**
35+
- Look for relevant deduplication, system, and application log events.
36+
- **Inspect file system health**
37+
- Run chkdsk /scan, and repair if necessary (for NTFS).
38+
- **Check the deduplication role or feature state**
39+
- Verify that the deduplication role is installed and enabled on volumes.
40+
- **Assess recent changes**
41+
- Review any OS upgrades, server migrations, or new security and backup software deployments.
42+
- **Verify deduplication settings**
43+
- Make sure that deduplication scheduling, volume support, and memory or core allocations align with best practices.
44+
- **Backup and disaster recovery**
45+
- Make sure that backups are up to date. Do not repair or migrate servers without working backups.
46+
47+
## Common issues and solutions
48+
49+
The following sections detail the most common failure modes and provide step-by-step solutions.
50+
51+
### Insufficient disk space and chunk store growth
52+
53+
#### Symptoms
54+
55+
- Deduplicated volume is critically low on free space despite file deletions.
56+
- Chunkstore folder (System Volume Information) is excessively large.
57+
- Storage operations (backups, file copies) fail.
58+
59+
#### Resolution
60+
61+
1. Manually run maintenance jobs:
62+
63+
```powershell
64+
Start-DedupJob -Volume <DriveLetter> -Type GarbageCollection -Full
65+
Start-DedupJob -Volume <DriveLetter> -Type Scrubbing -Full
66+
```
67+
68+
2. Remove Orphaned Shadow Copies/Checkpoints:
69+
70+
```console
71+
vssadmin list shadows
72+
devnodeclean
73+
```
74+
75+
3. Consider Expanding the Volume if still low on space.
76+
4. If ReFS/Cluster Size Limitations:
77+
- Back up, reformat with 64K cluster, and restore data.
78+
5. Investigate file patterns: Use TreeSize or a similar value to find space hogs not visible in user data.
79+
6. If maintenance jobs fail, see section F for file system corruption repair.
80+
81+
### Deduplication job failures and stuck jobs
82+
83+
#### Symptoms
84+
85+
- Deduplication job (Optimization, Garbage Collection, Scrubbing) starts but stays at 0% or aborts.
86+
- Jobs fail with errors such as:
87+
- “Not enough memory resources are available to complete this operation.”
88+
- “The operation was cancelled.”
89+
- Event IDs: 4105, 4106, 4140, 4142, 4185, and so on.
90+
91+
#### Resolution
92+
93+
1. Review job status:
94+
95+
```powershell
96+
Get-DedupJob
97+
```
98+
99+
2. Increase available resources:
100+
- Free up RAM or CPU by stopping unnecessary processes.
101+
- Increase physical or virtual memory, if possible.
102+
- Adjust job memory threshold:
103+
104+
```powershell
105+
Set-ItemProperty -Path HKLM:\System\CurrentControlSet\Services\ddpsvc\Settings -Name WlmMemoryOverPercentThreshold -Value 100
106+
```
107+
108+
3. Reschedule jobs for offpeak hours to avoid contention.
109+
4. Reduce memory parameter for jobs:
110+
111+
```powershell
112+
Start-DedupJob -Volume <DriveLetter> -Type Optimization -Memory 25
113+
```
114+
115+
5. Restart the server if jobs are persistently stuck.
116+
6. Check for scheduling conflicts:
117+
- Use Get-DedupSchedule and adjust as needed.
118+
119+
### File system corruption and deduplication metadata errors
120+
121+
#### Symptoms
122+
123+
- Event logs show corrupt user data containers, failed chunk store references, or errors such as:
124+
- "User data container is corrupt in the chunk store" (Event ID: 12837)
125+
- "Redirection table is too small" (for example, 0x8056530c)
126+
- Maintenance jobs fail repeatedly.
127+
- Deletions don’t reclaim space.
128+
129+
#### Resolution
130+
131+
1. Check and repair the file system:
132+
133+
```console
134+
chkdsk <DriveLetter>: /f /scan
135+
```
136+
137+
2. If issues persist:
138+
- Back up the data by using Windows Server Backup.
139+
- Create a new volume, restore the backup, and enable deduplication.
140+
3. As a last resort:
141+
- Migrate the data to new storage by using supported tools (not Robocopy) for deduplicated data.
142+
143+
### File access or migration problems
144+
145+
#### Symptoms
146+
147+
- Files cannot be opened or moved after a Windows upgrade or migration.
148+
- “ERROR_CANT_ACCESS_FILE" (0x80070780) when accessing files after an upgrade.
149+
- Manual file copies made to new servers expand the data (“rehydration”) and exhaust the destination storage.
150+
151+
#### Resolution
152+
153+
1. Reinstall the deduplication feature after an OS upgrade:
154+
155+
```powershell
156+
Install-WindowsFeature -Name FS-Data-Deduplication
157+
Enable-DedupVolume -Volume <DriveLetter>
158+
```
159+
160+
2. Use a supported backup and restore process or direct disk moves for migration:
161+
- Avoid Robocopy for deduplicated data.
162+
- Use Windows Backup, attach disks directly to new server, or image entire volumes.
163+
3. If data must Be moved and deduplication must be preserved:
164+
- Copy in small batches by using File Explorer to allow deduplication jobs to process incrementally.
165+
166+
### Misreporting or incomplete deduplication status
167+
168+
#### Symptoms
169+
170+
- Deduplication status not shown in Server Manager/Admin Center but available in PowerShell.
171+
- No errors—but functional confusion for administrators.
172+
173+
#### Resolution
174+
175+
1. Use PowerShell for an accurate status:
176+
177+
```powershell
178+
Get-DedupVolume
179+
```
180+
181+
2. Check S2D and clustered environments:
182+
183+
- Verify the volume group ownership and status.
184+
- If you experience a known code defect, escalate to Microsoft Support. Provide a BI document if the business impact is high.
185+
186+
### Low deduplication rate or unexpected space usage
187+
188+
#### Symptoms
189+
190+
- Deduplication rate much lower than anticipated (less than 30 percent).
191+
- Disk usage remains high despite deduplication.
192+
193+
#### Resolution
194+
195+
1. Analyze the data profile:
196+
- Identify the prevalence of already compressed or encrypted files (.jpg, .zip, .mp4).
197+
- Deduplication is less effective on such data types.
198+
2. Review deduplication logs for skipped files.
199+
3. Run manual optimization job and monitor space savings.
200+
4. Adjust deduplication configuration for file types and patterns.
201+
202+
### Interaction with third-party software and features
203+
204+
#### Symptoms
205+
206+
- Missing files or file corruption on deduplicated volumes.
207+
- Deduplication repeatedly disables itself.
208+
- Errors connected to endpoint security (e.g., CrowdStrike) or storage vendor deduplication.
209+
210+
#### Resolution
211+
212+
1. Check active file filters:
213+
214+
```console
215+
fltmc
216+
```
217+
218+
2. Exclude Deduplication Folders from Security/Antivirus Scans.
219+
3. Consult Vendor Documentation Before Double Deduplication (SAN + Windows).
220+
4. Restore From Backups if Data Corruption Has Occurred.
221+
222+
## Common issues quick reference table
223+
224+
| Symptom | Root cause | Resolution | Key commands and scripts |
225+
| --- | --- | --- | --- |
226+
| Dedup job stuck/fails, Event IDs 4105, 4140, 4142, etc. | Resource limits; job config | Free up memory, adjust job params, reboot, run jobs off-peak | Start-DedupJob, Set-ItemProperty |
227+
| Files inaccessible (0x80070780) after migration/upgrade | Missing/inactive dedup feature | Reinstall dedup role, enable on volume | Install-WindowsFeature, Enable-DedupVolume |
228+
| No space reclaimed after delete | Chunkstore overgrowth, no GC | Manually run GarbageCollection and Scrubbing jobs, expand volume | Start-DedupJob -Type GarbageCollection -Full |
229+
| Excessive chunkstore/spikes in disk usage | High unique data, old data | Run GC, review file patterns, expand disk, consider archiving/migration | Get-DedupMetadata |
230+
| Dedup status not shown in GUI | S2D/Cluster code defect | Use PowerShell for status, escalate with BI if needed | Get-DedupVolume |
231+
| Low deduplication rate | Incompressible/encrypted data | Review data profile, reset expectations, check config | Get-DedupStatus |
232+
| Space not reclaimed; chkdsk finds errors | File system corruption | Repair file system, migrate data, rebuild storage | chkdsk, robocopy, Backup/Restore |
233+
| Unusual backup/snapshot growth | Block-level dedup changes | Reconsider dedup usage on backup targets, adjust backup schedules | — |
234+
| Robocopy rehydrates data during migration | Not dedup-aware | Use Windows Backup or disk moves, enable dedup after copy; avoid Robocopy | Windows Backup, robocopy (w/ warnings) |
235+
236+
## Data collection
237+
238+
Before you contact Microsoft Support, you can gather the following information about your issue.
239+
240+
- Deduplication job and status:
241+
242+
```powershell
243+
Get-DedupJob
244+
Get-DedupStatus | fl \*
245+
Get-DedupVolume | fl \*
246+
Get-DedupMetadata <DriveLetter>
247+
```
248+
- Job and system event logs:
249+
250+
```powershell
251+
get-winevent -LogName 'Microsoft-Windows-Deduplication/Operational' -MaxEvents 10000 | export-csv dedup_operational.csv
252+
get-winevent -LogName 'Microsoft-Windows-Deduplication/Diagnostic' -MaxEvents 10000 | export-csv dedup_diag.csv
253+
get-winevent -LogName 'Microsoft-Windows-Deduplication/Scrubbing' -MaxEvents 10000 | export-csv dedup_scrubbing.csv
254+
```
255+
- CHKDSK output:
256+
257+
```console
258+
chkdsk <DriveLetter>: /f /scan
259+
```
260+
- Memory and CPU usage:
261+
262+
```console
263+
Get-Process | Sort-Object -Descending WorkingSet | Select-Object -First 20
264+
```
265+
- Directory and allocation reports:
266+
267+
```console
268+
dir <volume> /s /a > output.txt
269+
fsutil volume allocationReport <drive>: output.txt
270+
```
271+
- Shadow copy status and devices:
272+
273+
```console
274+
vssadmin list shadows
275+
devnodeclean /?
276+
```
277+
Collect these outputs before and after troubleshooting steps for comparison. Send logs securely to Microsoft, if it's necessary.
278+
279+
## References
280+
281+
- [Microsoft official data deduplication documentation](/windows-server/storage/data-deduplication/overview)
282+
- [Backup and restore with deduplication](/previous-versions/windows/desktop/dedup/backup-and-restore-of-data-deduplication-enabled-volumes)
283+
- [BitLocker and deduplication](/windows-server/storage/storage-spaces/volume-encryption-deduplication)
284+
- [Cluster S2D and deduplication best practices](/windows-server/storage/storage-spaces/overview)

0 commit comments

Comments
 (0)