Fix ReLU fusion when conv/linear has > 1 user (pytorch#6894)

Summary:
X-link: pytorch/pytorch#140846


Bug in quantizer when Conv + ReLU is fused even when the preceeding conv has more than one user. Conv and ReLU can not be fused in this case because the result of Conv must be used elsewhere.

XNNPACK Delegate naturally handles this by inserting a clamp node for ReLU.

Reviewed By: digantdesai

Differential Revision: D65989599

Author: mcr229

backends/xnnpack/test/ops/conv2d.py

@@ -394,3 +394,26 @@ def get_inputs(self):
             quant_config=get_symmetric_quantization_config(),
             conv_count=2,
         )
+
+    def test_qs8_conv2d_relu_multi_users(self):
+        class Conv2dReluMultiUsers(torch.nn.Module):
+            def __init__(self):
+                super().__init__()
+                self.conv1 = torch.nn.Conv2d(1, 1, 1)
+                self.conv2 = torch.nn.Conv2d(1, 64, 1)
+                self.relu = torch.nn.ReLU()
+
+            def forward(self, x):
+                conv_default = self.conv1(x)
+                y = self.relu(conv_default)
+                conv_default_2 = self.conv2(y)
+                return conv_default + conv_default_2
+
+            def get_inputs(self):
+                return (torch.randn(1, 1, 1, 1),)
+
+        self._test(
+            Conv2dReluMultiUsers(),
+            quant_config=get_symmetric_quantization_config(),
+            conv_count=2,
+        )