Merge pull request #1391 from mvphelps/betterMetrics

Better metrics

Author: NiklasGustafsson

RELEASENOTES.md

@@ -12,6 +12,7 @@ __Bug Fixes__:
 #1383 `torch.linalg.vector_norm`: Make `ord`-argument optional, as specified in docs<br/>
 #1385 PackedSequence now participates in the DisposeScope system at the same level as Tensor objects.<br/>
 #1387 Attaching tensor to a DisposeScope no longer makes Statistics.DetachedFromScopeCount go negative.<br/>
+#1390 DisposeScopeManager.Statistics now includes DisposedOutsideScopeCount and AttachedToScopeCount. ThreadTotalLiveCount is now exact instead of approximate. ToString gives a useful debug string, and documentation is added for how to troubleshoot memory leaks. Also DisposeScopeManager.Statistics.TensorStatistics and DisposeScopeManager.Statistics.PackedSequenceStatistics provide separate metrics for these objects.
 
 # NuGet Version 0.103.0
 

docfx/articles/memory leak troubleshooting.md

@@ -0,0 +1,81 @@
+# Memory Leak Troubleshooting
+
+If suspect you are leaking memory this is your guide. First be sure to be familiar with the [Memory Management Techniques](memory.md).
+
+## Verifying you have a leak
+
+The `DisposeScopeManager.Statistics` property defines thread level statistics of objects captured
+in TorchSharp as objects are created and moved between DisposeScopes. Normally deal directly
+with only this property.
+
+To see where code may be leaking objects, it is easiest to modify the training loop.
+Use a DisposeScope, reset the global statistics to have a known starting point, then take
+some action and look at the statistics to see what's still around.
+
+```csharp
+	//Training Loop, 10 epochs
+	for (int i = 0; i < 10; i++) {
+		//Clear the statistics
+		DisposeScopeManager.Statistics.Reset();
+		//Take action. In this case it is inside a DisposeScope, so
+		//when this code block is done, there should be no new live objects.
+		using (NewDisposeScope()) {
+			var eval = model.call(x);
+			// ... other model execution code
+			optimizer.step();
+		}
+		//Examine what happened
+		Console.WriteLine(DisposeScopeManager.Statistics);
+	}
+```
+
+If on every iteration the number of live objects is increasing, there is a leak. In the following
+example note that the number of live objects increases by 200 every iteration. It can also be
+seen these objects were created on a DisposeScope, but were eventually detached. In this specific
+case, look for where the code is detaching the tensors, and then determine how
+to correctly manage the lifetime of these objects.
+```csharp
+ThreadTotalLiveCount: 548; CreatedOutsideScopeCount: 0; DisposedOutsideScopeCount: 0; CreatedInScopeCount: 200; DisposedInScopeCount: 2; AttachedToScopeCount: 0; DetachedFromScopeCount: 200"
+ThreadTotalLiveCount: 748; CreatedOutsideScopeCount: 0; DisposedOutsideScopeCount: 0; CreatedInScopeCount: 200; DisposedInScopeCount: 2; AttachedToScopeCount: 0; DetachedFromScopeCount: 200"
+ThreadTotalLiveCount: 948; CreatedOutsideScopeCount: 0; DisposedOutsideScopeCount: 0; CreatedInScopeCount: 200; DisposedInScopeCount: 2; AttachedToScopeCount: 0; DetachedFromScopeCount: 200"
+```
+
+It is not necessary to leave this code in place for production implementations after fixing a leak. It may be removed so the code looks more Pythonic if needed.
+
+## Identifying the leak
+This is where the leg work is. Look at each line of code where Tensor or PackedSequence objects are created.
+Ensure they are eventually disposed either manually or by a DisposeScope. One can also print the statistics to
+the debugger while stepping code for an interactive approach.
+
+Be aware that TorchSharp also creates tensors for itself and uses them in various
+ways. Just because one finds a tensor that is created by TorchSharp isn't being disposed, it likely isn't caused
+by TorchSharp. A good example is the Adam optimizer. It creates tensors internally to manage it's parameters,
+and detaches them from any DisposeScope that is in use. If it didn't, it would fail doing gradients and back
+propagation as it's tensors would have been disposed. These are eventually cleaned up when the optimizer is
+properly disposed after training. Faliure of the client code to dispose is the most likely cause of memory leaks.
+
+## Working with RNNs
+One may want to drill down to `DisposeScopeManager.Statistics.TensorStatistics` or
+`DisposeScopeManager.Statistics.PackedSequenceStatistics`, these track Tensors and
+PackedSequence usages independently.
+
+Additionally, a PackedSequence uses some tensors internally. These tensors show up in the creation statistics,
+and are immediately detached from any scope if there is one in context and will
+increment the DetachedFromScopeCount property. When a PackedSequence is disposed, it will also Dispose
+it's tensors. The differences in counts can be seen in the following, which represents output within an IDE
+debug window where all three levels of statistics were observed at the same execution time. Note the first two
+sum to the totals on the last line.
+
+```
+  DisposeScopeManager.Statistics.TensorStatistics.ToString()
+  "ThreadTotalLiveCount: 4; CreatedOutsideScopeCount: 0; DisposedOutsideScopeCount: 0; CreatedInScopeCount: 6; DisposedInScopeCount: 2; AttachedToScopeCount: 0; DetachedFromScopeCount: 4"
+
+  DisposeScopeManager.Statistics.PackedSequenceStatistics.ToString()
+  "ThreadTotalLiveCount: 1; CreatedOutsideScopeCount: 0; DisposedOutsideScopeCount: 0; CreatedInScopeCount: 1; DisposedInScopeCount: 0; AttachedToScopeCount: 0; DetachedFromScopeCount: 0"
+
+  DisposeScopeManager.Statistics.ToString()
+  "ThreadTotalLiveCount: 5; CreatedOutsideScopeCount: 0; DisposedOutsideScopeCount: 0; CreatedInScopeCount: 7; DisposedInScopeCount: 2; AttachedToScopeCount: 0; DetachedFromScopeCount: 4"
+
+```
+
+

docfx/articles/memory.md

@@ -10,6 +10,8 @@ In both cases, you may want to experiment with using a smaller batch size -- tem
 
 Note DiffSharp (which uses TorchSharp) relies on techniques 1.
 
+Also refer to [Memory Leak Troubleshooting](memory leak troubleshooting.md) for help on fixing any leaks.
+
 > Most of the examples included will use technique #1, doing frequent explicit calls to GC.Collect() in the training code -- if not after each batch in the training loop, at least after each epoch.
 
 ## Technique 1. Automatic disposal via Garbage Collection
@@ -44,7 +46,7 @@ __Note__: Even with this approach, it is a good idea to place a call to `GC.Coll
 
 It is important to understand that all TorchSharp "tensors" (type Tensor) are actually "tensor aliases", referring to a C++ tensor. When a C++ tensor is created and returned to .NET as a tensor alias, and the reference count on the C++ tensor is incremented. When you call `Dispose()` on the TorchSharp tensor alias (that is, type Tensor), it is decremented. If the tensor alias is finalized instead, the decrement happens implicitly.
 
-To enable this technique, all operations that return one or more TorchSharp `Tensor`s should return "fresh" Tensor aliases (though that doesn't always mean freshly copied C++ tensors). This is true even for in-place, destructive operations like `add_()`, which overwrites the underlying native tensor with data, but still returns a fresh tensor alias to that same tensor.  
+To enable this technique, all operations that return one or more TorchSharp `Tensor`s should return "fresh" Tensor aliases (though that doesn't always mean freshly copied C++ tensors). This is true even for in-place, destructive operations like `add_()`, which overwrites the underlying native tensor with data, but still returns a fresh tensor alias to that same tensor.
 
 Thus, when you write methods and functions that take and produce type Tensor, for example in the `forward()` method of a model, you should always make sure to return a fresh alias. Most of the time, this happens automatically, because the last action of your code will normally be to call another tensor function, which itself will be returning a fresh alias, but there are cases when it's not, especially when returning input tensors or tensors stored in some lookaside table.
 
@@ -55,7 +57,7 @@ Tensor flatten(Tensor input) {
     if (input.shape.Length == 1)
         return input.alias();
     else
-        return input.reshape(input.numel());        
+        return input.reshape(input.numel());
 }
 ```
 
@@ -100,10 +102,10 @@ let myTensorFunction0(input: Tensor) =
     input.alias()
 
 let myTensorFunction1() =
-    if today then 
-       table[4].alias()  
+    if today then
+       table[4].alias()
     else
-       table[5].alias()  
+       table[5].alias()
 
 let myTensorFunction2(input: Tensor) =
     input.add(tensor(1))
@@ -124,9 +126,9 @@ let myTensorFunction5(go: bool, input: Tensor) =
         tmp2.add(tensor(1))
     else
         input.alias()
-    
+
 let myTensorFunction5(go: bool, input: Tensor) =
-    if go then 
+    if go then
         use tmp1 = input.add_(tensor(1))  // NOTE: even for in-place mutations
         use tmp2 = input.add_(tensor(1))  // NOTE: even for in-place mutations
         tmp2.add(tensor(1))
@@ -173,13 +175,13 @@ use d = torch.NewDisposeScope()
 total_acc <- total_acc + (predicted_labels.argmax(1) == labels).sum().cpu().item<long>()
 ```
 
-If you need to dispose some tensors before the scope is disposed, you can use `DisposeEverything()`, or `DisposeEverythingBut(...)` if you want to exclude a few tensors from disposal. These can be useful when tensor lifetimes aren't cleanly nested in dynamic scopes. 
+If you need to dispose some tensors before the scope is disposed, you can use `DisposeEverything()`, or `DisposeEverythingBut(...)` if you want to exclude a few tensors from disposal. These can be useful when tensor lifetimes aren't cleanly nested in dynamic scopes.
 
 __NOTE: It is absolutely essential for the proper functioning of dynamic dispose scopes that the scope is created with a 'using' statemen (C#) or 'use' expression (F#).__
 
 It's important to note that these scopes are dynamic -- if any functions are called, the tensors inside them are also registered and disposed, unless there's a nested scope within those functions.
 
-It is advisable to place a dispose scope around your training and test code, and in any library code that can be called from contexts that do not have dispose scopes. 
+It is advisable to place a dispose scope around your training and test code, and in any library code that can be called from contexts that do not have dispose scopes.
 
 That said, you should use dispose scope very carefully: having _too few_ scope raises the pressure on native memory, which is particularly bad for GPUs. Having too _many_ scopes, managing too few temporaries, will add runtime overhead to computations. For example, it may be better to put a scope outside an inner loop that contains multiple computations than to place it inside the loop. There is no single best answer.
 

src/TorchSharp/DisposeScope.cs

@@ -154,9 +154,13 @@ public void MoveToOther(DisposeScope? scope, IEnumerable<IDisposable> disposable
         {
             if (this._disposeScopeManager is null)
                 throw new ObjectDisposedException(this.GetType().FullName);
-            foreach (var disposable in disposables) {
-                if (Disposables.Remove(disposable)) {
-                    AddToOther(scope, disposable);
+            if (scope == null) {
+                Detach(disposables);
+            } else {
+                foreach (var disposable in disposables) {
+                    if (Disposables.Remove(disposable)) {
+                        AddToOther(scope, disposable);
+                    }
                 }
             }
         }
@@ -209,11 +213,11 @@ public void Detach(IEnumerable<IDisposable> disposables)
                 throw new ObjectDisposedException(this.GetType().FullName);
             foreach (var disposable in disposables) {
                 if (Disposables.Remove(disposable)) {
-                    _disposeScopeManager.StatisticsInstance.DetachedFromScopeCount++;
                     if (disposable is torch.Tensor tensor) {
+                        _disposeScopeManager.StatisticsInstance.TensorStatistics.DetachedFromScopeCount++;
                         tensor.OwningDisposeScope = null;
-                    }
-                    else if (disposable is torch.nn.utils.rnn.PackedSequence sequence) {
+                    } else if (disposable is torch.nn.utils.rnn.PackedSequence sequence) {
+                        _disposeScopeManager.StatisticsInstance.PackedSequenceStatistics.DetachedFromScopeCount++;
                         sequence.OwningDisposeScope = null;
                     }
                 }
@@ -237,7 +241,13 @@ public IReadOnlyList<IDisposable> Attach(IEnumerable<IDisposable> disposables)
 
             var result = new List<IDisposable>();
             foreach (var disposable in disposables) {
-                AddToOther(this, disposable);
+                if (AddToOther(this, disposable)) {
+                    if (disposable is torch.Tensor tensor) {
+                        _disposeScopeManager.StatisticsInstance.TensorStatistics.AttachedToScopeCount++;
+                    } else if (disposable is torch.nn.utils.rnn.PackedSequence sequence) {
+                        _disposeScopeManager.StatisticsInstance.PackedSequenceStatistics.AttachedToScopeCount++;
+                    }
+                }
                 result.Add(disposable);
             }
 
@@ -267,22 +277,6 @@ public void DisposeEverythingBut(IEnumerable<IDisposable> inKeep)
                     continue;
                 }
 
-                if (disposable is torch.Tensor tensor) {
-                    // No need to have the disposable call back to the scope
-                    tensor.OwningDisposeScope = null;
-                    if (!tensor.IsInvalid) {
-                        _disposeScopeManager.StatisticsInstance.DisposedInScopeCount++;
-                    }
-                } else if (disposable is torch.nn.utils.rnn.PackedSequence sequence) {
-                    // No need to have the disposable call back to the scope
-                    sequence.OwningDisposeScope = null;
-                    if (!sequence.IsInvalid) {
-                        _disposeScopeManager.StatisticsInstance.DisposedInScopeCount++;
-                    }
-                } else {
-                    _disposeScopeManager.StatisticsInstance.DisposedInScopeCount++;
-                }
-
                 disposable.Dispose();
             }
         }
@@ -358,7 +352,7 @@ public void MarkAsDisposed(IDisposable disposable)
         {
             if (this._disposeScopeManager is null)
                 throw new ObjectDisposedException(this.GetType().FullName);
-            _disposeScopeManager.StatisticsInstance.DisposedInScopeCount++;
+
             Disposables.Remove(disposable);
             if (disposable is torch.Tensor tensor) {
                 tensor.OwningDisposeScope = null;
@@ -375,33 +369,45 @@ public void MarkAsDisposed(IDisposable disposable)
         /// <returns></returns>
         public bool Contains(IDisposable disposable) => Disposables.Contains(disposable);
 
-        private void AddToOther(DisposeScope? scope, IDisposable disposable)
+        private bool AddToOther(DisposeScope scope, IDisposable disposable)
         {
             if (this._disposeScopeManager is null)
                 throw new ObjectDisposedException(this.GetType().FullName);
-            if (scope != null) {
-                scope.Disposables.Add(disposable);
+
+            DisposeScope? oldScope;
+            if (disposable is torch.Tensor t) {
+                oldScope = t.OwningDisposeScope;
+            } else if (disposable is torch.nn.utils.rnn.PackedSequence sequence) {
+                oldScope = sequence.OwningDisposeScope;
             } else {
-                _disposeScopeManager.StatisticsInstance.DetachedFromScopeCount++;
+                throw new InvalidOperationException("DisposeScope can only manage Tensor or PackedSequence");
+            }
+
+            if (scope == oldScope) return false;
+
+            scope.Disposables.Add(disposable);
+            if (oldScope != null) {
+                oldScope.Disposables.Remove(disposable);
             }
 
             if (disposable is torch.Tensor tensor) {
                 tensor.OwningDisposeScope = scope;
-            }
-            else if (disposable is torch.nn.utils.rnn.PackedSequence sequence) {
+            } else if (disposable is torch.nn.utils.rnn.PackedSequence sequence) {
                 sequence.OwningDisposeScope = scope;
             }
+
+            return true;
         }
 
         internal HashSet<IDisposable> DetachAllAndDispose()
         {
             var disposables = this.Disposables;
             foreach (var disposable in this.Disposables) {
-                this._disposeScopeManager!.StatisticsInstance.DetachedFromScopeCount++;
                 if (disposable is torch.Tensor tensor) {
+                    this._disposeScopeManager!.StatisticsInstance.TensorStatistics.DetachedFromScopeCount++;
                     tensor.OwningDisposeScope = null;
-                }
-                else if (disposable is torch.nn.utils.rnn.PackedSequence sequence) {
+                } else if (disposable is torch.nn.utils.rnn.PackedSequence sequence) {
+                    this._disposeScopeManager!.StatisticsInstance.PackedSequenceStatistics.DetachedFromScopeCount++;
                     sequence.OwningDisposeScope = null;
                 }
             }

src/TorchSharp/DisposeScopeManager.cs

@@ -18,18 +18,42 @@ public class DisposeScopeManager
         internal ThreadDisposeScopeStatistics StatisticsInstance { get; } = new ThreadDisposeScopeStatistics();
         internal DisposeScope? CurrentDisposeScope { get; private set; } = null;
 
-        internal DisposeScope? RegisterOnCurrentDisposeScope(IDisposable tensor)
+        internal DisposeScope? RegisterOnCurrentDisposeScope(IDisposable item)
         {
             if (this.CurrentDisposeScope is null) {
-                StatisticsInstance.CreatedOutsideScopeCount++;
+                if (item is torch.Tensor t) {
+                    StatisticsInstance.TensorStatistics.CreatedOutsideScopeCount++;
+                } else if (item is torch.nn.utils.rnn.PackedSequence p) {
+                    StatisticsInstance.PackedSequenceStatistics.CreatedOutsideScopeCount++;
+                }
                 return null;
+            } else {
+                if (item is torch.Tensor t) {
+                    StatisticsInstance.TensorStatistics.CreatedInScopeCount++;
+                } else if (item is torch.nn.utils.rnn.PackedSequence p) {
+                    StatisticsInstance.PackedSequenceStatistics.CreatedInScopeCount++;
+                }
+                this.CurrentDisposeScope.Disposables.Add(item);
+                return CurrentDisposeScope;
             }
-
-            StatisticsInstance.CreatedInScopeCount++;
-            this.CurrentDisposeScope.Disposables.Add(tensor);
-            return CurrentDisposeScope;
         }
 
+        internal void DisposingOnCurrentScope(torch.Tensor item)
+        {
+            if (item.OwningDisposeScope == null) {
+                StatisticsInstance.TensorStatistics.DisposedOutsideScopeCount++;
+            } else {
+                StatisticsInstance.TensorStatistics.DisposedInScopeCount++;
+            }
+        }
+        internal void DisposingOnCurrentScope(torch.nn.utils.rnn.PackedSequence item)
+        {
+            if (item.OwningDisposeScope == null) {
+                StatisticsInstance.PackedSequenceStatistics.DisposedOutsideScopeCount++;
+            } else {
+                StatisticsInstance.PackedSequenceStatistics.DisposedInScopeCount++;
+            }
+        }
         internal void RemoveDisposeScope(DisposeScope disposeScope)
         {
             var scope = this.CurrentDisposeScope;