Advanced examples

This page contains some more advanced examples of using Crawljax.

Using a Browser pool

Crawling even a small blog site can take several minutes. This is due to the huge number of states even the simplest blogging sites like Octopress and Wordpress have. Luckily, Crawljax can run multiple browsers in parallel, offering a huge speed-up in the overal process.

To configure a Browser pool, specify the number of threads and browsers you'd like to use in a ThreadConfiguration. You can then add the ThreadConfiguration to your CrawljaxConfiguration.

Here's an example:

ThreadConfiguration threadConfiguration = new ThreadConfiguration();
threadConfiguration.setNumberThreads(10);
threadConfiguration.setNumberBrowsers(5);
threadConfiguration.setBrowserBooting(true);

The number of Threads is set higher then the number of browsers because some of the analysis in the background can be run in parallel. The Number of browsers is the actual number of browsers that will be used to do the crawling. Setting the BrowserBooting to true makes the BrowserPool start the Browsers immediately so the Crawler has doesn't have to wait for them to boot when it needs them.