[sil-combine] Canonicalize owned forwarding insts without non-debug non-consuming uses by sinking to their uses.

There are a bunch of optimizations in SILCombine where we try to fold an
ownership forwarding instruction A into another ownership forwarding instruction
B without deleting A. Consider the upcasts in the example below:

```
  %0 = upcast %x : $X->Y
  %1 = upcast %0 : $Y->Z
```

These sorts of optimizations fold the first instruction into the second like so:

```
  %0 = upcast %x : $X->Y
  %1 = upcast %x : $X->Z
```

This creates a problem when we are dealing with owned values since we have just
introduced two consumes for %x. To work around this, we have two options:

1. Introduce extra copies.

2. We recognize the situations where we can guarantee that we can delete the
   first upcast.

The first choice I believe is not a choice since breaking a forwarding chain of
ownership in favor of extra copies is a less canonical form. That leaves us with
the second form. What are the necessary/sufficient conditions for deleting the
first upcast. Simply it is that the upcast cannot have any non-debug,
non-consuming uses! In such a case, we know that along all paths through the
program the value has exactly one non-debug use, one of its consuming uses. If
when optimizing upcasts we could recognize that pattern, duplicate the inst
along paths not through our 2nd upcast and thus delete the original upcast
fixing the ownership error!

While this is all nice and good there is a problem with this: it doesn't
scale. As I was writing a few optimizations like this I began to note that I had
to write different versions of this same helper for many of the visitors (they
generally varied by how many forwarding instructions they looked through).

As I pondered the above, I chatted a bit with @atrick and during our
conversation, we both realized that it is much easier to solve this problem in
one block and that the condition above would allow us to sink these instructiosn
into the same block and thus if we could check for this condition and
canonicialize the IR to sink these instructions before we visiting, we could use
a single helper to handle all of these cases.

Author: gottesmm

include/swift/SILOptimizer/Utils/DebugOptUtils.h

@@ -0,0 +1,48 @@
+//===--- DebugOptUtils.h --------------------------------------------------===//
+//
+// This source file is part of the Swift.org open source project
+//
+// Copyright (c) 2014 - 2020 Apple Inc. and the Swift project authors
+// Licensed under Apache License v2.0 with Runtime Library Exception
+//
+// See https://swift.org/LICENSE.txt for license information
+// See https://swift.org/CONTRIBUTORS.txt for the list of Swift project authors
+//
+//===----------------------------------------------------------------------===//
+///
+/// Debug Info related utilities that rely on code in SILOptimizer/ and thus can
+/// not be in include/swift/SIL/DebugUtils.h.
+///
+//===----------------------------------------------------------------------===//
+
+#ifndef SWIFT_SILOPTIMIZER_DEBUGOPTUTILS_H
+#define SWIFT_SILOPTIMIZER_DEBUGOPTUTILS_H
+
+#include "swift/SIL/DebugUtils.h"
+#include "swift/SIL/SILValue.h"
+#include "swift/SILOptimizer/Utils/InstOptUtils.h"
+
+namespace swift {
+
+/// Deletes all of the debug instructions that use \p value.
+inline void deleteAllDebugUses(SILValue value, InstModCallbacks &callbacks) {
+  for (auto ui = value->use_begin(), ue = value->use_end(); ui != ue;) {
+    auto *inst = ui->getUser();
+    ++ui;
+    if (inst->isDebugInstruction()) {
+      callbacks.deleteInst(inst);
+    }
+  }
+}
+
+/// Deletes all of the debug uses of any result of \p inst.
+inline void deleteAllDebugUses(SILInstruction *inst,
+                               InstModCallbacks &callbacks) {
+  for (SILValue v : inst->getResults()) {
+    deleteAllDebugUses(v, callbacks);
+  }
+}
+
+} // namespace swift
+
+#endif

lib/SILOptimizer/SILCombiner/SILCombine.cpp

@@ -19,6 +19,7 @@
 //===----------------------------------------------------------------------===//
 
 #define DEBUG_TYPE "sil-combine"
+
 #include "SILCombiner.h"
 #include "swift/SIL/DebugUtils.h"
 #include "swift/SIL/SILBuilder.h"
@@ -28,6 +29,7 @@
 #include "swift/SILOptimizer/PassManager/Passes.h"
 #include "swift/SILOptimizer/PassManager/Transforms.h"
 #include "swift/SILOptimizer/Utils/CanonicalizeInstruction.h"
+#include "swift/SILOptimizer/Utils/DebugOptUtils.h"
 #include "swift/SILOptimizer/Utils/InstOptUtils.h"
 #include "swift/SILOptimizer/Utils/SILOptFunctionBuilder.h"
 #include "swift/SILOptimizer/Utils/StackNesting.h"
@@ -41,6 +43,11 @@ using namespace swift;
 STATISTIC(NumCombined, "Number of instructions combined");
 STATISTIC(NumDeadInst, "Number of dead insts eliminated");
 
+static llvm::cl::opt<bool> EnableSinkingOwnedForwardingInstToUses(
+    "silcombine-owned-code-sinking",
+    llvm::cl::desc("Enable sinking of owened forwarding insts"),
+    llvm::cl::init(true), llvm::cl::Hidden);
+
 //===----------------------------------------------------------------------===//
 //                              Utility Methods
 //===----------------------------------------------------------------------===//
@@ -145,6 +152,63 @@ class SILCombineCanonicalize final : CanonicalizeInstruction {
   }
 };
 
+bool SILCombiner::trySinkOwnedForwardingInst(SingleValueInstruction *svi) {
+  if (auto *consumingUse = svi->getSingleConsumingUse()) {
+    auto *consumingUser = consumingUse->getUser();
+
+    // If our user is already in the same block, we don't move it further.
+    if (svi->getParent() == consumingUser->getParent())
+      return false;
+
+    // Otherwise, make sure our instruction does not have any non-debug uses
+    // that are non-lifetime ending. If so, we return.
+    if (llvm::any_of(getNonDebugUses(svi),
+                     [](Operand *use) { return !use->isLifetimeEnding(); }))
+      return false;
+
+    // Otherwise, delete all of the debug uses so we don't have to sink them as
+    // well and then return true so we process svi in its new position.
+    deleteAllDebugUses(svi, instModCallbacks);
+    svi->moveBefore(consumingUser);
+    MadeChange = true;
+
+    // NOTE: We return nullptr here so that our caller doesn't delete the
+    // instruction and instead tries to simplify it.
+    return false;
+  }
+
+  // If we have multiple consuming uses, then we know that our
+  // forwarding inst must be live out of the current block and thus we
+  // might be able to duplicate/sink.
+  if (llvm::any_of(getNonDebugUses(svi),
+                   [](Operand *use) { return !use->isLifetimeEnding(); }))
+    return false;
+
+  while (!svi->use_empty()) {
+    auto *sviUse = *svi->use_begin();
+    auto *sviUser = sviUse->getUser();
+
+    if (auto *dvi = dyn_cast<DestroyValueInst>(sviUser)) {
+      dvi->setOperand(svi->getOperand(0));
+      Worklist.add(dvi);
+      continue;
+    }
+
+    if (sviUser->isDebugInstruction()) {
+      eraseInstFromFunction(*sviUser);
+      continue;
+    }
+
+    auto *newSVI = svi->clone(sviUser);
+    Worklist.add(newSVI);
+    sviUse->set(newSVI);
+  }
+
+  eraseInstFromFunction(*svi);
+  MadeChange = true;
+  return true;
+}
+
 bool SILCombiner::doOneIteration(SILFunction &F, unsigned Iteration) {
   MadeChange = false;
 
@@ -184,7 +248,31 @@ bool SILCombiner::doOneIteration(SILFunction &F, unsigned Iteration) {
     }
 
     // If we have reached this point, all attempts to do simple simplifications
-    // have failed. Prepare to SILCombine.
+    // have failed. First if we have an owned forwarding value, we try to
+    // sink. Otherwise, we perform the actual SILCombine operation.
+    if (EnableSinkingOwnedForwardingInstToUses) {
+      // If we have an ownership forwarding single value inst that forwards
+      // through its first argument and it is trivially duplicatable, see if it
+      // only has consuming uses. If so, we can duplicate the instruction into
+      // the consuming use blocks and destroy any destroy_value uses of it that
+      // we see. This makes it easier for SILCombine to fold instructions with
+      // owned paramaters since chains of these values will be in the same
+      // block.
+      if (auto *svi = dyn_cast<SingleValueInstruction>(I)) {
+        if ((isa<FirstArgOwnershipForwardingSingleValueInst>(svi) ||
+             isa<OwnershipForwardingConversionInst>(svi)) &&
+            SILValue(svi).getOwnershipKind() == OwnershipKind::Owned) {
+          // Try to sink the value. If we sank the value and deleted it,
+          // continue. If we didn't optimize or sank but we are still able to
+          // optimize further, we fall through to SILCombine below.
+          if (trySinkOwnedForwardingInst(svi)) {
+            continue;
+          }
+        }
+      }
+    }
+
+    // Then begin... SILCombine.
     Builder.setInsertionPoint(I);
 
 #ifndef NDEBUG

lib/SILOptimizer/SILCombiner/SILCombiner.h

@@ -333,6 +333,12 @@ class SILCombiner :
   bool tryOptimizeKeypathKVCString(ApplyInst *AI, FuncDecl *calleeFn,
                                   KeyPathInst *kp);
 
+  /// Sinks owned forwarding instructions to their uses if they do not have
+  /// non-debug non-consuming uses. Deletes any debug_values and destroy_values
+  /// when this is done. Returns true if we deleted svi and thus we should not
+  /// try to visit it.
+  bool trySinkOwnedForwardingInst(SingleValueInstruction *svi);
+
   // Optimize concatenation of string literals.
   // Constant-fold concatenation of string literals known at compile-time.
   SILInstruction *optimizeConcatenationOfStringLiterals(ApplyInst *AI);

test/SILOptimizer/sil_combine_ossa.sil

@@ -9,6 +9,9 @@ import Builtin
 import Swift
 
 class Klass {}
+
+sil @use_klass_guaranteed : $@convention(thin) (@guaranteed Klass) -> ()
+
 class RawBuffer {}
 class HeapBufferStorage<T, U> : RawBuffer {}
 
@@ -47,6 +50,8 @@ struct MyInt {
 sil [global_init] @global_init_fun : $@convention(thin) () -> Builtin.RawPointer
 
 sil [ossa] @user : $@convention(thin) (@owned Builtin.NativeObject) -> ()
+sil [ossa] @use_nativeobject_owned : $@convention(thin) (@owned Builtin.NativeObject) -> ()
+sil [ossa] @use_nativeobject_guaranteed : $@convention(thin) (@guaranteed Builtin.NativeObject) -> ()
 
 sil [ossa] @unknown : $@convention(thin) () -> ()
 
@@ -4111,3 +4116,97 @@ bb0(%0 : @owned $Klass):
   destroy_value %0 : $Klass
   return %2 : $Optional<Klass>
 }
+
+// We do not sink if the forwarding inst is in the same block as the consuming use.
+//
+// CHECK-LABEL: sil [ossa] @sink_forwarding_owned_uses_no_sink_same_block : $@convention(thin) (@owned Klass, @guaranteed Klass) -> @owned Builtin.NativeObject {
+// CHECK: bb0(
+// CHECK-NEXT: unchecked_ref_cast
+// CHECK-NEXT: function_ref
+// CHECK: } // end sil function 'sink_forwarding_owned_uses_no_sink_same_block'
+sil [ossa] @sink_forwarding_owned_uses_no_sink_same_block : $@convention(thin) (@owned Klass, @guaranteed Klass) -> @owned Builtin.NativeObject {
+bb0(%0a : @owned $Klass, %0b : @guaranteed $Klass):
+  %1 = unchecked_ref_cast %0a : $Klass to $Builtin.NativeObject
+  %2 = function_ref @use_klass_guaranteed : $@convention(thin) (@guaranteed Klass) -> ()
+  apply %2(%0b) : $@convention(thin) (@guaranteed Klass) -> ()
+  return %1 : $Builtin.NativeObject
+}
+
+// In this case, since we have different blocks, we sink.
+//
+// CHECK-LABEL: sil [ossa] @sink_forwarding_owned_uses_sink_diff_block_single_use : $@convention(thin) (@owned Klass, @guaranteed Klass) -> @owned Builtin.NativeObject {
+// CHECK: bb0(
+// CHECK-NEXT: br bb1
+// CHECK: bb1:
+// CHECK-NEXT: unchecked_ref_cast
+// CHECK-NEXT: return
+// CHECK: } // end sil function 'sink_forwarding_owned_uses_sink_diff_block_single_use'
+sil [ossa] @sink_forwarding_owned_uses_sink_diff_block_single_use : $@convention(thin) (@owned Klass, @guaranteed Klass) -> @owned Builtin.NativeObject {
+bb0(%0a : @owned $Klass, %0b : @guaranteed $Klass):
+  %1 = unchecked_ref_cast %0a : $Klass to $Builtin.NativeObject
+  br bb1
+
+bb1:
+  return %1 : $Builtin.NativeObject
+}
+
+// CHECK-LABEL: sil [ossa] @sink_forwarding_owned_uses_sink_diff_block_single_use_noopt : $@convention(thin) (@owned Klass, @guaranteed Klass) -> @owned Builtin.NativeObject {
+// CHECK: bb0(
+// CHECK-NEXT: unchecked_ref_cast
+// CHECK-NEXT: function_ref
+// CHECK-NEXT: function_ref
+// CHECK-NEXT: apply
+// CHECK-NEXT: br bb1
+// CHECK: bb1:
+// CHECK-NEXT: return
+// CHECK: } // end sil function 'sink_forwarding_owned_uses_sink_diff_block_single_use_noopt'
+sil [ossa] @sink_forwarding_owned_uses_sink_diff_block_single_use_noopt : $@convention(thin) (@owned Klass, @guaranteed Klass) -> @owned Builtin.NativeObject {
+bb0(%0a : @owned $Klass, %0b : @guaranteed $Klass):
+  %1 = unchecked_ref_cast %0a : $Klass to $Builtin.NativeObject
+  %f = function_ref @use_nativeobject_guaranteed : $@convention(thin) (@guaranteed Builtin.NativeObject) -> ()
+  apply %f(%1) : $@convention(thin) (@guaranteed Builtin.NativeObject) -> ()
+  br bb1
+
+bb1:
+  return %1 : $Builtin.NativeObject
+}
+
+// In this case, since we have different blocks, we sink.
+//
+// CHECK-LABEL: sil [ossa] @sink_forwarding_owned_uses_sink_diff_block_single_use_destroy : $@convention(thin) (@owned Klass, @guaranteed Klass) -> () {
+// CHECK: bb0(
+// CHECK-NEXT: cond_br undef, bb1, bb2
+//
+// CHECK: bb1:
+// CHECK-NEXT: function_ref
+// CHECK-NEXT: function_ref
+// CHECK-NEXT: unchecked_ref_cast
+// CHECK-NEXT: apply
+// CHECK-NEXT: br bb3
+//
+// CHECK: bb2:
+// CHECK-NEXT: destroy_value
+// CHECK-NEXT: br bb3
+//
+// CHECK: bb3:
+// CHECK-NEXT: tuple
+// CHECK-NEXT: return
+// CHECK: } // end sil function 'sink_forwarding_owned_uses_sink_diff_block_single_use_destroy'
+sil [ossa] @sink_forwarding_owned_uses_sink_diff_block_single_use_destroy : $@convention(thin) (@owned Klass, @guaranteed Klass) -> () {
+bb0(%0a : @owned $Klass, %0b : @guaranteed $Klass):
+  %1 = unchecked_ref_cast %0a : $Klass to $Builtin.NativeObject
+  cond_br undef, bb1a, bb1b
+
+bb1a:
+  %f = function_ref @use_nativeobject_owned : $@convention(thin) (@owned Builtin.NativeObject) -> ()
+  apply %f(%1) : $@convention(thin) (@owned Builtin.NativeObject) -> ()
+  br bb1
+
+bb1b:
+  destroy_value %1 : $Builtin.NativeObject
+  br bb1
+
+bb1:
+  %9999 = tuple()
+  return %9999 : $()
+}