Version 2.3.2

This is mostly a bug-fix release.

---

• Changed behaviour (read bugfix) in local locking #1842
• Changed behaviour (read bugfix) in disabling workers #1839
• Many bugfixes to the scheduler, particularly related to the Batch running functionality.

Here are the changes commit by commit.

Version 2.3.1

This release mainly fixes bugs introduced in the two latest releases and also
some older bugs.

---

• Bugfix regarding sftp #1825
• Bugfix regarding error emails with smtp #1821
• Bugfix regarding spark tasks #1819
• Bugfixes regarding visualiser #1817 #1818

Here are the changes commit by commit.

Version 2.3.0

There's been over a month since the last release. This new release includes a
bunch of new features. What I like the most is that they all come with full and
proper documentation!

luigi:

• Fixed lockfile bugfix introduced in 2.2.0 #1809 Thanks @nugend
• Retcodes improved #1771 Thanks @fabriziodemaria
• Added NumericalParameter
  and ChoiceParameter Thanks @brcopeland
• Per-task configurable retry policies! #1791 Thanks @javrasya
• Luigi can now batch tasks! #1784 Thanks @daveFNbuck

luigi.scheduler:

• We now have a new definition of the UPSTREAM statuses. New intuition is that
  UPSTREAM_FAILED means that that task cannot run because all downstream
  tasks have failed or worse (like being disabled). As an effect there will be
  much much fewer tasks considered to have an upstream status. #1789

There have been a lot of other features, cleanups and bugfixes! See all commits
here.

Version 2.2.0

There's been 3 months since the latest release. Making Google's results on readthedocs outdated and gives a stale feeling to luigi. Enjoy updated and hopefully bugfree software. :)

At least read these

• Luigi finally has user-land configurable task status messages #1625
• Parameters: From now on, you must not pass None as the default for a
  parameter. Usually, passing the empty string '' is a sufficient
  replacement. If you don't do this luigi will print a deprecation warning.
  #1624.
• Logging for server: Things are greatly improved
  now as of #1633
  and #1636.
  Here's my jotted down usage info about it here
  #1752 (comment).

Main changes

luigi:

• More fine grained eventhandlers #1698
• Range: Finally a proper way to pass along parameters: #1675
• From this release, we'll also bump the debian verion number. #1718
• Print your dependency tree as ascii art! #1680
• We now have a template for PR's! #1655

luigi.contrib:

• AWS: You can now set the session token: https://github.com/spotify/luigi/pull/1702/files
• Salesforce: Add support for multiple results #1686
• FTP: Configurable port #1689
• MSSQL support: #1650
• Streaming mapreduce: Allow additional archives #1649
• Streaming mapreduce: Recognize the Google File System formats #1664
• Streaming mapreduce: mrrunner.py is not hardcoded as the binary being run #1565

Various goodies

• Cleanup: #1734
• Cleanup: #1735
• Cleanup: #1723
• Cleanup: #1677
• Cleanup: #1640
• Cleanup: #1722
• Speedup: #1737
• Speedup: #1681
• Docs: #1729
• Docs: #1662
• Docs: #1716
• Bugfix: #1708
• Bugfix: #1643
• Improved error message: #1692

Contribution spirit

A few great examples that show how improvements are well-receieved no matter how small they are. Yet these "small" changes helps hundreads of people reading the docs of luigi. #1672 #1642

Other changes

There were even more changes which we didn't include in these release notes. Like every contribution, whether merged or not we are happy for getting them. So please keep contributing! :)

Version 2.1.1 (Includes security fix)

Last release was only 2 business days ago (as opposed to the 5 months since the one before it). But this release got rushed as of a security fix!

In addition to doc fixes:

Additions

• List and Tupleparameters are now added
• More Bigquery Options
• The hive-cmd can now contain spaces.

Security bugfix

• The server now have an explicit whitelist of external commands.
  • Previous potential harm: Malicious hackers can run arbitrary code if they have file system (even external mounts!)+network access on the machine running luigid (executed by the user that you run luigid with).

We wait for a while with saying how to use this exploit, giving time to people to apply the bugfix.

Version 2.1.0

Finally, a new PyPI release in a rather long while. Thanks @Tarrasch and @erikbern, and all the contributors!

Added

luigi:

• Notifications: more emails and proper coloring (#1471), improved SMTP handling
• EnumParameter (#1479), DictParameter (#1574)
• Support for Python 3.5 (#1494)
• Process locking on Alpine Linux (#1530) and Windows (#1557)
• Visualizer: resources tab (#1566), GUI functionality to disable a worker (#1564)

luigi.contrib:

• ExternalBigqueryTask (#1434), BigqueryCreateViewTask (#1465)
• Luigi tasks for Dataproc, Google's managed Hadoop MapReduce, Spark, Pig, and Hive service (#1601)
• ExternalProgramTask, ExternalPythonProgramTask - commonalities for running any external application or script (#1520)
• Support for SFTP (#1585)
• Sped-up Hive client using Metastore (#1533)
• OpenerTarget, a single Luigi target to open multiple file system types (#1555)
• Query base task (giving rise also to luigi.contrib.redshift.RedshiftQuery and luigi.postgres.PostgresQuery) (#1493)
• RedshiftUnloadTask (#1527)
• UploadToSalesforceTask (#1404)
• Support for S3 assumed role (#1596)

Changed

luigi:

• Semi-opaque, hashed task_id (as opposed to TaskName(param1=value1, param2=foo bar)) (#1444)
• More explicit way to handle timelike parameters (date vs datetime) (#1473)
• Optimizations in scheduler algorithm

Removed

luigi:

• Old deprecated (2014) stuff around scheduler and its state (#1592)

luigi.contrib:

• Deprecated classes SparkJob, Spark1xBackwardCompat, Spark1xJob, PySpark1xJob (#1442)

Fixed

luigi.contrib:

• Ensure that FTP RemoteTarget successfully creates temporary files (meaning, in a directory relative to output) (#1515)
• Remove superfluous init_mapper()/init_reducer() calls in LocalJobRunner (#1475)
• Humanly format HadoopJobError (#1528)
• Broken Redshift table creation (#1453)
• Improved Salesforce reliability (#1597, #1600)
• Missing call to post_copy() (#1502)

...and a slew of other additions, fixes, improvements and documentation.

Version 2.0.1

This version contains mostly bugfixes, in particular making Luigi 2.0 working on Windows

Thanks @daveFNbuck @econchick @Tarrasch @dlstadther @DeaconDesperado @jpiper @nresare @dhurlburtusa @stephenpascoe @mbruggmann and some more people I probably forgot!

Pull requests

• #1441 don't print nested stack traces
• #1439 spark: use list2command
• #1447 fix unix socket
• #1446 fix daemon
• #1448 support symlinks in hdfs
• #1431 #1433 worker as a context manager
• #1429 keyboardinterrupt fix
• #1428 make it work without tornado
• #1425 positional parameter qualification
• #1422 fixed broken task trigger
• #1411 mention misspelled task name
• #1418 bigquery fix
• #1417 visualizer fix
• #1413 kwargs for s3
• #1345 retry get_work
• #1315 limit graph size
• #1391 amazon sns support
• #1392 pyspark parameter fix
• #1395 tornado fix
• #1397 spark package fix
• #1400 serialize TaskParameter
• #1402 kwargs for boto s3
• #1388 disable email from cfg
• #1341 get_work fix
• #1340 stop subprocesses on exception
• #1352 fix bug in scheduler
• #1357 fix issues in requires during scheduling
• #1367 fix visualizer w weird task names
• #1372 s3target.path
• #1314 redshift updates
• #1370 gmail authentication for notifications
• #1366 pig fix
• #1331 path warning fix
• #1338 ssh forwarding agent
• #1337 ssh multiplexing
• #1363 windows signal fix
• #1298 tracking fix
• #1346 fix dep graph on firefox

2.0.0

Highlights:

• Improved documentation.
• New visualiser page! #1086, thanks @stephenpascoe
• Execution summaries at end of runs! #1091, thanks @nicolehedblom
• Smarter layout algorithm for the SVG dependency graphs, thanks @daveFNbuck
• Command line interface now has a useful --help, see #1276
• Luigi finally has semantic exit codes! #1264

Contrib:

• Stablization of GCP tools
• AWS EC2 support. #1317, thanks @jfeala
• Sungrid support. #1124, thanks @jfeala (again :))

Scheduling:

• UNIX sockets are now supported, hence also encryption. #1129, thanks @graingert
• Smarter scheduling! #1272
• Speedup in scheduler. #1046
• A --take-lock option to smoother roll out code changes. #1137

Deprecations:

• There have been a lot of deprecations of features regarding Parameters. Before upgrading, please consider taking a look at the new module documentation for parameters.

1.3.0

Notable changes:

• is_global finally removed! (It still exists to not break old code, but it only sets positional=False)
• Initial support for Google Cloud Storage and Google BigQuery. Thanks @mikekap! #999 #1002
• Massive luigid speedup for get_work() response times. #986
• Various bugfixes.
• Standardized FileSystem/Target interface. Again thanks @mikekap! #997
• SQLAlchemy works for multiple engines. Thanks @dylanjbarth! #991

v1.2.1

Same as 1.2.0, only that I didn't forget to update the setup.py file this time.