2.8.3

Added:

luigi

• Add BaseTIS to the company list #2607
• Add Hopper to the company list #2614
• give a few default values to opts when setting up logging #2612
• Add range functionality for monthly cadence. #2601

luigi.contrib

• Added port to PostgresTarget #2615
• Support for Azure Blob Storage Target #2585
• Add Datadog contrib for monitoring purpose #2434

Fixed:

luigi

• Docs: Fixed a mistake with @inherits syntax in luigi/util.py #2613
• Check type of column before migrating schemas for task db history for postgres dialect (fixes #2563) #2564

luigi.contrib

• S3: Fix call to message from TypeError not working with Python 3.6 #2617
• Use proper API call in bigtable.py's make_dataset #2618
• Sqla: Fix the table name when reflect is True in sqla.CopyToTable (fixes #2604) #2605

Changed:

luigi.contrib

• Changed to buffered reads when using GCSTarget #2588

2.8.2 (Hotfix)

• fix logging setup in Python 2.7 (#2593)

2.8.1(broken)

Note: Broken due to a runtime error in LuigiConfigParser. See #2592.

Added:

luigi

• Add some docs to interface.run #2582
• Configure logging via TOML config #2483

luigi.contrib

• Added port property to CopyToTable #2561
• Make it so we can do from luigi.contrib.hdfs import HdfsFlagTarget #2594
• contrib: Add ExternalDailySnapshot #2591

Fixed:

luigi

• (docstring) Update task.py #2589
• Docs: Fixed "Github" to fit to the rest of the doc #2596
• Fix inspect.getargspec() DeprecationWarning in PY3 #2579

luigi.contrib

• Fix Travis Moto Test Failures #2586
• Reduce TravisCI Test Runtime #2541

Changed:

luigi

• Make Worker parameter task_process_context an OptionalParameter #2574

luigi.contrib

• S3Client improvements #2569

2.8.0

This is a minor version bump, due to:

• Dropping Python 3.4 and 3.5 from CI, which means no automated tests to ensure compatibility for those versions
• [Security Patch] CORS being disabled by default. A new section of configuration [cors] is introduced to enable custom settings. For details, refer to user group topic: https://groups.google.com/forum/#!topic/luigi-user/ZgfRTpBsVUY

Added

luigi:

• Add Python 3.7 compatibility (#2466)
  This also drops 3.4 and 3.5 from CI.
• Interpolate environment variables in .cfg config files (#2527)

luigi.contrib:

• Add CopyToTable task for MySQL (#2553)
• Add HdfsFlagTarget (#2559)

luigi.contrib:

Fixed

luigi:

• Fix ReadTheDocs build (#2546)
• Make capture_output non-positional in ExternalProgramTask (#2547)

luigi.contrib:

• Fix S3Client's _path_to_bucket_and_key to support keys with question marks (#2534)
• Fix S3Client.remove - add max batch size (#2529)
• Small fix to logging in contrib/ecs.py (#2556)
• FIX HdfsAtomicWriteDirPipe.close() when using snakebite and the file do not exist. (#2549)

Changed:

luigi:

• [ImgBot] optimizes images (#2555)

luigi.contrib:

• Remove s3 bucket validation prior to file upload (#2528)
• Refactor s3 copy into sub-methods (#2508)

2.7.9

Added

luigi.contrib:

• Added optional choice for hdfs clients (#2487)
• s3client check for deprecated host keyword and raise error with the details (#2493)
• Add a "capture_output" parameter to ExternalProgramTask (#2430)

Fixed

luigi:

• Fix exception when toml lib is not installed (#2506)
• Replace direct attribute accessing by using built-n function getattr (#2509)
• set upper bound of python-daemon (#2536)

luigi.contrib:

• Fix S3Client.copy return value consistency (#2488)
• Update MockTarget mode to accept r* or w* (#2519)

Version 2.7.8 (hotfix)

• revert tornado upgrade (#2504)
  Upgrading tornado unfortunately breaks older version of Python

Version 2.7.7 (partly broken)

Added

luigi:

• Add Data Revenue to the blogged list (#2472)
• Add default reviewers in CODEOWNERS (#2465)
• Optional TOML configs support (#2457)
• Add support for multiple requires and inherits arguments (#2475)
• Add a visiblity level for luigi.Parameters (#2278)
• Make logging of RPC retries configurable #2486
• Added a new event 'progress' (#2498)

luigi.contrib:

• Additions to provide support for the Load Sharing Facility (LSF) job scheduler (#2373)
• Added default port behaviour for Redshift (#2474)
• Add metadata columns to the RDBMS contrib (#2440)
• Use passed password when create a redis connection (#2489)

Changed

luigi:

• Update supplementary github files to improve repo organization and maintenance (#2463)
• Use task_id in Task.eq comparison (#2462)
• Replace luigi.Task by RunOnceTask in scheduler_visualisation_test (#2476)
• (Breaking change) Bump tornado milestone version (#2490)
  This changes requires Python version 2.7.9+ and 3.4+

luigi.contrib:

• S3 client refactor (#2482)
• Update moto to 1.x milestone version (#2471)

Fixed

luigi:

• Fix Scheduler.add_task to overwrite accepts_messages attribute. (#2469)
• Fix race condition (#2477)
• Fix attribute forwarding for tasks with dynamic dependencies (#2478)

luigi.contrib:

• Fix transfer config import (#2458)

Removed

luigi:

• Remove long-deprecated scheduler config variable alternatives (#2491)

Version 2.7.6

Added

luigi:

• Add a configuration parameter to force multiprocessing (#2401)
• Add a configuration parameter to enable/disable the pause button (#2399)
• Send messages from scheduler to tasks (via "Send message" UI button) (#2426)
• Allow to inject a context manager around TaskProcess.run (via task_process_context configuration parameter) (#2449)

luigi.contrib:

• S3: use Boto3 for the S3Client (#2423, #2149)
• GCS: add method to push files using multiprocessing (#2376)
• HDFS: add get_merge to snakebite client (#2410)
• Redshift: add schema to DB if it doesn't exist (#2439)
• Redshift: add table constraints support (#2435)

Fixed

luigi:

• Allow long parameters in task history DB SQL result store (#2404)
• Fix MissingParameterException when generating execution summary (#2415)
• Fix luigid crash due to configuration file parsing (#2394)
• Allow explicit parsing of BoolParameters (via luigi.BoolParameter.parsing variable) (#2427)
• Make ChoiceParameter check if option is valid within .normalize (#2454)
• ...and a good deal of documentation fixes and similar.

luigi.contrib:

• BigQuery: fix bulk_complete failing when argument is a generator (#2441)
• Kubernetes: prevent KeyError in KubernetesJobTask (#2433)
• Kubernetes: don't set activeDeadlineSeconds by default (#2452)

Version 2.7.5

Fixed

luigi

• append is vulnerable of xss #2391

Version 2.7.4

Added

luigi:

• Add google-auth-httplib2 as dependency (#2384)
• Add new company user (#2388)
• Allow release of resources during task running. (#2346)

luigi.contrib:

• Add parameterized backoff limit (#2375)

Fixed

luigi:

• Typo in running luigi documentation (#2387)
• Skip running coverage on unreasonable files
• Pass kwargs to discovery.build() when instantiating GSCClient. (#2291)
• Removed message 'No Instance(s) Available.' in Windows when starting … #2294