Fix issue with mongodb database not being recreated for each test (#3228)

Several mongodb tests have been failing in the ServerCentral environment
for a while with errors that look like they're unable to find a world
with a given id.  I wasn't getting this error in a local setup, so I
compared the two databases.  I noticed that on ServerCentral, the worlds
all had "_id" and "randomNumber" fields, but they had no "id" field (no
underscore) like they did in my local setup.

Our mongo create.js script is supposed to create all the worlds with both
kinds of id, so for the database on ServerCentral to look different, that
must have not been happening when it was supposed to.  It was supposed to
happen (1) when mongo is first installed and also when (2) whenever a
framework depends on mongo and mongo is already installed.  It looked
like it was not happening on (2).

Pulling it together, the theory was:

 - Some frameworks (such as nodejs-mongodb) were querying for worlds by
   "id" instead of "_id".

 - Some framework(s) (such as redstone-mongodb) were removing the "id"
   fields from worlds and persisting them.

 - The worlds were not reset to a known good state between each test, so
   if one framework clobbered the "id" fields, other frameworks that ran
   later in the same run would fail because of it.

I confirmed that in the logs on ServerCentral, there is content like this
right around where create.js is supposed to be executed:

  Setup nodejs-mongodb: mongod start/running, process 15837
  Setup nodejs-mongodb: MongoDB shell version: 3.2.13
  Setup nodejs-mongodb: connecting to: test
  Setup nodejs-mongodb: 2018-01-23T19:10:33.408-0600 W NETWORK  [thread1] Failed to connect to 127.0.0.1:27017, in(checking socket for error after poll), reason: errno:111 Connection refused
  Setup nodejs-mongodb: 2018-01-23T19:10:33.408-0600 E QUERY    [thread1] Error: couldn't connect to server 127.0.0.1:27017, connection attempt failed :
  Setup nodejs-mongodb: connect@src/mongo/shell/mongo.js:229:14
  Setup nodejs-mongodb: @(connect):1:6
  Setup nodejs-mongodb:
  Setup nodejs-mongodb: exception: connect failed

We were not giving mongod enough time to start up.  Elsewhere in the same
file we did a wait loop after "sudo service mongod start", so the fix is
to repeat that wait loop in this other place.

I also took the opportunity to fix the $i loop variable that was not
being printed correctly in the logs in those "Waiting for MongoDB ($i/15)"
messages.

The fact that some frameworks are apparently looking for worlds by "id"
instead of "_id" raises the question of whether both approaches are
equally efficient, or whether mongo provides better indexing for one or
the other.  That's a research topic for another day.

Author: michaelhixson

toolset/setup/linux/databases/mongodb/mongodb.sh

@@ -9,7 +9,7 @@ scp $FWROOT/toolset/setup/linux/databases/mongodb/mongodb.conf $DBHOST:~/
 scp $FWROOT/toolset/setup/linux/databases/mongodb/create.js $DBHOST:~/
 
 # install mongo on database machine
-ssh $DBHOST 'bash' <<EOF
+ssh $DBHOST 'bash' <<"EOF"
 echo "Setting up MongoDB database"
 sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv EA312927
 echo 'deb http://repo.mongodb.org/apt/ubuntu trusty/mongodb-org/3.2 multiverse' | sudo tee /etc/apt/sources.list.d/mongodb-org-3.2.list
@@ -29,7 +29,7 @@ sudo service mongod start
 
 for i in {1..15}; do
   nc -z localhost 27017 && break || sleep 1;
-  echo "Waiting for MongoDB ($i/15}"
+  echo "Waiting for MongoDB ($i/15)"
 done
 nc -z localhost 27017
 if [ $? -eq 0 ]; then
@@ -40,8 +40,12 @@ else
 fi
 EOF
 
-echo -e "ssh \$DBHOST 'bash' <<EOF" > $IROOT/mongodb.installed
+echo -e "ssh \$DBHOST 'bash' <<\"EOF\"" > $IROOT/mongodb.installed
 echo -e "sudo service mongod start || echo 'mongod service already started'" >> $IROOT/mongodb.installed
+echo -e "for i in {1..15}; do" >> $IROOT/mongodb.installed
+echo -e "  nc -z localhost 27017 && break || sleep 1;" >> $IROOT/mongodb.installed
+echo -e "  echo \"Waiting for MongoDB (\$i/15)\"" >> $IROOT/mongodb.installed
+echo -e "done" >> $IROOT/mongodb.installed
 echo -e "mongo < create.js" >> $IROOT/mongodb.installed
 echo -e "EOF" >> $IROOT/mongodb.installed