Randomized tests need to be repeatable. Use this: http://labs.carrotsearch.com/randomizedtesting.html Lucene/Solr uses this, the Spatial4j lib uses this, the SolrTextTagger uses this. It's wonderful and simple.