Skip to content

CHANGES.md

Latest commit

 

History

History
250 lines (184 loc) · 9.45 KB

CHANGES.md

File metadata and controls

250 lines (184 loc) · 9.45 KB

Change Log

Unreleased

Nothing notable yet.

0.9.0 - 2017-08-06

Added

  • #178: You can now see additional output during launch and other operations with the new --debug option.
  • #185: Added a new mount point under /media/tmp that can be used when /tmp is not big enough.
  • #186: You can now tag your clusters with arbitrary tags on launch using the new --ec2-tag option. (Remember: As with all options, you can also set this via flintrock configure.)
  • #191: You can now specify the size of the root EBS volume with the new --ec2-min-root-ebs-size-gb option.
  • #181: You can now set the number of executors per worker with --spark-executor-instances.

Changed

  • #195: After launching a new cluster, Flintrock now shows the master address and login command.
  • #196, #197: Fixed some bugs that were preventing Flintrock from launching Spark clusters at a specific commit.
  • #204: Flintrock now automatically retries starting the Spark and HDFS masters if it encounters common issues with bringing the cluster up. This greatly improves launch and restart reliability.
  • #208: Flintrock now provides a hint with possible causes for certain SSH errors.

0.8.0 - 2017-02-11

Added

  • #180: Accessing data on S3 from your Flintrock cluster is now much easier! Just configure Flintrock to use Hadoop 2.7+ (which is the default) and an appropriate IAM role, and you'll be able to access paths on S3 using the new s3a:// prefix. Check the README for more information.
  • #176, #187: Flintrock now supports users with non-standard home directories.

Changed

  • #168: Flintrock now does a better job of cleaning up after interrupted operations.
  • #179, #184: Flintrock can now clean up malformed Flintrock clusters.
  • 6b426ae: We fixed an issue affecting some users of Flintrock's standalone package that caused Flintrock to intermittently throw ImportErrors.

0.7.0 - 2016-11-15

Added

  • #146: Flintrock now ensures that launched clusters have Java 8 or higher installed.
  • #149: You can now specify an EC2 user data script to use on launch with the new --ec2-user-data option.

Changed

  • #154, #155, #156: Flintrock now provides friendly error messages when it encounters common configuration or setup problems.

0.6.0 - 2016-08-28

Added

  • #115: Flintrock can now resize existing clusters with the new add-slaves and remove-slaves commands.

Changed

  • #115: If you lost your master somehow, Flintrock can now still destroy the cluster.
  • #115: You can no longer launch clusters with 0 slaves. The implementation was broken. We may fix and add this capability back in the future.

0.5.0 - 2016-07-20

Added

  • #118: You can now specify --hdfs-download-source (or the equivalent in your config file) to tell Flintrock to download Hadoop from a specific URL when launching your cluster.
  • #125: You can now specify --spark-download-source (or the equivalent in your config file) to tell Flintrock to download Spark from a specific URL when launching your cluster.
  • #112: You can now specify --ec2-security-group to associate additional security groups with your cluster on launch.

Changed

  • #103, #114: Flintrock now opens port 6066 and 7077 so local clients like Apache Zeppelin can connect directly to the Spark master on the cluster.
  • #122: Flintrock now automatically adds executables like spark-submit, pyspark, and hdfs to the default PATH, so they're available to call right when you login to the cluster.

0.4.0 - 2016-03-27

Added

  • #98, #99: You can now specify latest for --spark-git-commit and Flintrock will automatically build Spark on your cluster at the latest commit. This feature is only available for Spark repos hosted on GitHub.
  • #94: Flintrock now supports launching clusters into non-default VPCs.

Changed

  • #86: Flintrock now correctly catches when spot requests fail and bubbles up an appropriate error message.
  • #93, #97: Fixed the ability to build Spark from git. (It was broken for recent commits.)
  • #96, #100: Flintrock launches should now work correctly whether the default Python on the cluster is Python 2.7 or Python 3.4+.

0.3.0 - 2016-02-14

Changed

  • eca59fc, 3cf6ee6: Tweaked a few things so that Flintrock can launch 200+ node clusters without hitting certain limits.

0.2.0 - 2016-02-07

Added

  • b00fd12: Added --assume-yes option to the launch command. Use --assume-yes to tell Flintrock to automatically destroy the cluster if there are problems during launch.

Changed

  • #69: Automatically retry Hadoop download from flaky Apache mirrors.
  • 0df7004: Delete unneeded security group after a cluster is destroyed.
  • 244f734: Default HDFS not to install. Going forward, Spark will be the only service that Flintrock installs by default. Defaults can easily be changed via Flintrock's config file.
  • de33412: Flintrock installs services, not modules. The terminology has been updated accordingly throughout the code and docs. Update your config file to use services instead of modules. Warning: Flintrock will have problems managing existing clusters that were launched with versions of Flintrock from before this change.
  • #73: Major refactoring of Flintrock internals.
  • #74: Flintrock now catches common configuration problems upfront and provides simple error messages, instead of barfing out errors from EC2 or launching broken clusters.
  • bf766ba: Fixed a bug in how Flintrock polls SSH availability from Linux. Cluster launches now work from Linux as intended.

0.1.0 - 2015-12-11

  • Initial release.