Issue in cray build checker.

[gpaulsen]

There appears to be some issue with the Cray Build Checker.

For example, the PR #6672, has a single commit: e00d0abe56b2163c9aedbecf0e973c9609804c61

From the github records, it was never force pushed to after the PR was opened.
The Ci failed with this one line:

Pull Request Build Checker — Testing failed! Sad panda wants you to click on...

Which refers to: https://jenkins.open-mpi.org/jenkins/job/open-mpi.pull_request/4751/

Here, there are two passes and a failure:
open-mpi.build.autogen_options - PASSED
open-mpi.build.configure_options - PASSED
open-mpi.build.platforms - FAILED
Which refers to: https://jenkins.open-mpi.org/jenkins/job/open-mpi.build.platforms/4847/
The console output is: https://jenkins.open-mpi.org/jenkins/job/open-mpi.build.platforms/4847/console

Started by upstream project "open-mpi.pull_request" build number 4751
originally caused by:
 GitHub pull request #6672 of commit e00d0abe56b2163c9aedbecf0e973c9609804c61, no merge conflicts.
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on Ubuntu 16.04 (i-0f70c624a298746d8) (clang38 clang36 clang37 ubuntu_16.04 gcc48 gcc47 ec2 gcc49 management_job linux gcc5) in workspace /home/ubuntu/workspace/open-mpi.build.platforms
[WS-CLEANUP] Deleting project workspace...
[WS-CLEANUP] Deferred wipeout is used...
No credentials specified
Cloning the remote Git repository
Cloning repository https://github.com/open-mpi/ompi/
 > git init /home/ubuntu/workspace/open-mpi.build.platforms/src # timeout=10
Fetching upstream changes from https://github.com/open-mpi/ompi/
 > git --version # timeout=10
 > git fetch --tags --progress https://github.com/open-mpi/ompi/ +refs/heads/*:refs/remotes/origin/*
 > git config remote.origin.url https://github.com/open-mpi/ompi/ # timeout=10
 > git config --add remote.origin.fetch +refs/heads/*:refs/remotes/origin/* # timeout=10
 > git config remote.origin.url https://github.com/open-mpi/ompi/ # timeout=10
Fetching upstream changes from https://github.com/open-mpi/ompi/
 > git fetch --tags --progress https://github.com/open-mpi/ompi/ +refs/heads/*:refs/remotes/origin/* +refs/pull/*:refs/remotes/origin/pr/*
 > git rev-parse refs/remotes/origin/pr/6672/merge^{commit} # timeout=10
 > git rev-parse refs/remotes/origin/origin/pr/6672/merge^{commit} # timeout=10
JENKINS-19022: warning: possible memory leak due to Git plugin usage; see: https://wiki.jenkins-ci.org/display/JENKINS/Remove+Git+Plugin+BuildsByBranch+BuildData
Checking out Revision 06e5b0bd071fad26a3c1dca0e9967e406f091476 (refs/remotes/origin/pr/6672/merge)
 > git config core.sparsecheckout # timeout=10
 > git checkout -f 06e5b0bd071fad26a3c1dca0e9967e406f091476
Commit message: "Merge e00d0abe56b2163c9aedbecf0e973c9609804c61 into 8c9a3d1d1f05b92b209b1149b5fe9d9b2b4662dc"
First time build. Skipping changelog.
Triggering open-mpi.build.platforms » cray_cle
Triggering open-mpi.build.platforms » opensuse_leap_42
Triggering open-mpi.build.platforms » ARMv8
open-mpi.build.platforms » cray_cle completed with result FAILURE
Configuration open-mpi.build.platforms » opensuse_leap_42 is still in the queue: Waiting for next available executor on ‘ompi.cs.uh.edu’
open-mpi.build.platforms » opensuse_leap_42 completed with result SUCCESS
open-mpi.build.platforms » ARMv8 completed with result SUCCESS
Finished: FAILURE

This appears to point to the cray_cle, which is here: https://jenkins.open-mpi.org/jenkins/job/open-mpi.build.platforms/Platform=cray_cle/4847/console

Searching for 'error' results in two errors:

/usr/bin/m4:config/c_get_alignment.m4:1: read error WEIRD

And:

================================================
Open MPI autogen: completed successfully.  w00t!
================================================

--> running ./configure --prefix="/global/homes/h/hpp/jenkins_cori/workspace/open-mpi.build.platforms/Platform/cray_cle/install" 
configure: error: cannot find install-sh, install.sh, or shtool in config "."/config
./configure --prefix="/global/homes/h/hpp/jenkins_cori/workspace/open-mpi.build.platforms/Platform/cray_cle/install"  failed, ABORTING !
config.log content :
This file contains any messages produced by compilers while
running configure, to aid debugging if configure makes a mistake.

It was created by Open MPI configure gitclone, which was
generated by GNU Autoconf 2.69.  Invocation command line was

  $ ./configure --prefix=/global/homes/h/hpp/jenkins_cori/workspace/open-mpi.build.platforms/Platform/cray_cle/install

## --------- ##
## Platform. ##
## --------- ##

hostname = cori08
uname -m = x86_64
uname -r = 4.4.162-94.72-default
uname -s = Linux
uname -v = #1 SMP Mon Nov 12 18:57:45 UTC 2018 (9de753f)

/usr/bin/uname -p = x86_64
/bin/uname -X     = unknown

...

## ----------- ##
## Core tests. ##
## ----------- ##

configure:5468: error: cannot find install-sh, install.sh, or shtool in config "."/config

[bwbarrett]

Reassigning to Howard. I don't own any Cray/HPE systems (too soon?).

[gpaulsen]

Oh, sorry. I thought all of the "build checker" CI was done on AWS.

[hppritcha]

looks like this was some transient hickup on cori. closing. reopen if this occurs on a regular basis.