Add a random UUID to worker identities, to prevent collisions (#1135)

The fallback worker identity of {pid}@{host}@{tasklist} is insufficient in practice.
E.g. within Uber, our docker setup can somewhat regularly lead to multiple instances
of a worker ending up on the same physical host (which is not changed by docker) and
with the same PID (as all the docker containers start up the same way -> they all
get the same PIDs).
Other causes are possible too, e.g. if a worker crashes and is restarted it could
share the same host+pid, even though its caches (and anything else we actually care
about for the identity) are lost.

At the least-problematic-but-confusing level, this can lead to a misleadingly-short
list of workers on tasklists in the web UI, as duplicates are collapsed.

At the most-problematic level, this can lead to uneven load and sub-par request
routing, as only the first/last/??? request to poll the server will receive data
intended for a specific worker.

Ideally that wouldn't be an issue, e.g. the server could keep better track of polls
and route more precisely, but 1) it's difficult to reliably identify polls since
the server only has the poller-provided data (which mis-identifies itself), and 2)
even if it can be fixed on the server, improving identity uniqueness helps both fix
this now and reduces or eliminates the impact of any future identity confusions.

---

Alternatively, we could add a process-global random UUID that auto-inits, which
would give us IDs that can be correlated across tasklists.  While I think this is
acceptable as well, I'm loathe to add another global variable, and the benefit of
cross-task identity correlating is pretty minor or nonexistent.

If someone actually cares about that, they can pass an explicit ID that they
generate however they like.

Author: Groxx

internal/internal_utils.go

@@ -32,6 +32,7 @@ import (
 	"syscall"
 	"time"
 
+	"github.com/pborman/uuid"
 	"github.com/uber-go/tally"
 	s "go.uber.org/cadence/.gen/go/shared"
 	"go.uber.org/cadence/internal/common"
@@ -196,8 +197,12 @@ func newChannelContextHelper(
 }
 
 // GetWorkerIdentity gets a default identity for the worker.
+//
+// This contains a random UUID, generated each time it is called, to prevent identity collisions when workers share
+// other host/pid/etc information.  These alone are not guaranteed to be unique, especially when Docker is involved.
+// Take care to retrieve this only once per worker.
 func getWorkerIdentity(tasklistName string) string {
-	return fmt.Sprintf("%d@%s@%s", os.Getpid(), getHostName(), tasklistName)
+	return fmt.Sprintf("%d@%s@%s@%s", os.Getpid(), getHostName(), tasklistName, uuid.New())
 }
 
 func getHostName() string {