improving hfsort+ algorithm

Summary:
A few improvements for hfsort+ algorithm. The goal of the diff is (i) to make the resulting function order more i-cache "friendly" and (ii) fix a bug with incorrect input edge weights. A specific list of changes is as follows:
- The "samples" field of CallGraph.Node should be at least the sum of incoming edge weights. Fixed with a new method CallGraph::adjustArcWeights()
- A new optimization pass for hfsort+ in which pairs of functions that call each other with very high probability (>=0.99) are always merged. This improves the resulting i-cache but may worsen i-TLB. See a new method HFSortPlus::runPassOne()
- Adjusted optimization goal to make the resulting ordering more i-cache "friendly", see HFSortPlus::expectedCalls and HFSortPlus::mergeGain
- Functions w/o samples are now reordered too (they're placed at the end of the list of hot functions). These functions do appear in the call graph, as some of their basic blocks have samples in the LBR dataset. See HfSortPlus::initializeClusters

(cherry picked from FBD6248850)

Author: spupyrev

bolt/Passes/CallGraph.cpp

@@ -94,7 +94,6 @@ const CallGraph::Arc &CallGraph::incArcWeight(NodeId Src, NodeId Dst, double W,
 }
 
 void CallGraph::normalizeArcWeights() {
-  // Normalize arc weights
   for (NodeId FuncId = 0; FuncId < numNodes(); ++FuncId) {
     auto& Func = getNode(FuncId);
     for (auto Caller : Func.predecessors()) {
@@ -108,5 +107,18 @@ void CallGraph::normalizeArcWeights() {
   }
 }
 
+void CallGraph::adjustArcWeights() {
+  for (NodeId FuncId = 0; FuncId < numNodes(); ++FuncId) {
+    auto& Func = getNode(FuncId);
+    uint64_t InWeight = 0;
+    for (auto Caller : Func.predecessors()) {
+      auto Arc = findArc(Caller, FuncId);
+      InWeight += (uint64_t)Arc->weight();
+    }
+    if (Func.samples() < InWeight)
+      setSamples(FuncId, InWeight);
+  }
+}
+
 }
 }

bolt/Passes/CallGraph.h

@@ -153,11 +153,21 @@ class CallGraph {
     return double(Arcs.size()) / (Nodes.size()*Nodes.size());
   }
 
+  // Initialize NormalizedWeight field for every arc
   void normalizeArcWeights();
+  // Make sure that the sum of incoming arc weights is at least the number of
+  // samples for every node
+  void adjustArcWeights();
 
   template <typename L>
   void printDot(char* fileName, L getLabel) const;
+  
 private:
+  void setSamples(const NodeId Id, uint64_t Samples) {
+    assert(Id < Nodes.size());
+    Nodes[Id].Samples = Samples;
+  }
+
   std::vector<Node> Nodes;
   ArcsType Arcs;
 };

bolt/Passes/HFSort.h

@@ -103,7 +103,7 @@ std::vector<Cluster> clusterize(const CallGraph &Cg);
 /*
  * Optimize function placement for iTLB cache and i-cache.
  */
-std::vector<Cluster> hfsortPlus(const CallGraph &Cg,
+std::vector<Cluster> hfsortPlus(CallGraph &Cg,
                                 bool UseGainCache = true,
                                 bool UseShortCallCache = true);