Potentially needs a backport from @atmyers in ECP-WarpX#1 :
We apparently don't consider a non-zero exit code a failure unless self.run_as_script is True. When doing plotfile comparisons, this isn't a big issue, since the comparison itself will fail as the test did not produce any output. However, when we run through travis we turn off comparisons, so we need to check the exit code.
An alternative fix would be to consider all non-zero exit codes a failure. In principle I prefer this, but I don't know the reasoning behind the original behavior here, so I'd like to be conservative.