Recent Releases of luigi

luigi - 3.6.0

What's Changed

  • Fix logging of sensitive information in pai.py by @Ali-Razmjoo in https://github.com/spotify/luigi/pull/3310
  • Fix overly permissive file permissions in luigi/lock.py by @Ali-Razmjoo in https://github.com/spotify/luigi/pull/3308
  • Fix unsafe shell command in luigi/contrib/lsf.py by @Ali-Razmjoo in https://github.com/spotify/luigi/pull/3307
  • Fix arbitrary file write during tarfile extraction in luigi/contrib/lsfrunner.py and luigi/contrib/sgerunner.py by @Ali-Razmjoo in https://github.com/spotify/luigi/pull/3309
  • add ChoiceListParameter by @kitagry in https://github.com/spotify/luigi/pull/3305
  • Implement mypy plugin for luigi.Task by @hiro-o918 in https://github.com/spotify/luigi/pull/3315
  • /history/bytaskid : a method to link each task's history uniquely to the SQL database by @tashrifbillah in https://github.com/spotify/luigi/pull/3321
  • Replace os.rename with os.replace by @yyyyuki in https://github.com/spotify/luigi/pull/3322
  • Support Dropbox rootnamespaceid for Team Spaces by @smrohrer in https://github.com/spotify/luigi/pull/3316
  • Validate str type before tuple cast by @ikyasam18 in https://github.com/spotify/luigi/pull/3323
  • fix(luigi/contrib/postgres.py): replace copyfrom with copyexpert by @delhomer in https://github.com/spotify/luigi/pull/3324
  • Drop Support for Python 3.5 and 3.6 by @dlstadther in https://github.com/spotify/luigi/pull/3325
  • Version 3.6.0 by @dlstadther in https://github.com/spotify/luigi/pull/3328

New Contributors

  • @Ali-Razmjoo made their first contribution in https://github.com/spotify/luigi/pull/3310
  • @yyyyuki made their first contribution in https://github.com/spotify/luigi/pull/3322
  • @ikyasam18 made their first contribution in https://github.com/spotify/luigi/pull/3323
  • @delhomer made their first contribution in https://github.com/spotify/luigi/pull/3324

Full Changelog: https://github.com/spotify/luigi/compare/v3.5.2...v3.6.0

- Python
Published by dlstadther about 1 year ago

luigi - 3.5.2

What's Changed

  • Updated luigi.contrib.azureblob to 12.x.y series of azure.storage.blob by @bcamel in https://github.com/spotify/luigi/pull/3290
  • Fix batch email configuration docs by @ccmaymay in https://github.com/spotify/luigi/pull/3296

New Contributors

  • @bcamel made their first contribution in https://github.com/spotify/luigi/pull/3290
  • @ccmaymay made their first contribution in https://github.com/spotify/luigi/pull/3296
  • @hiro-o918 made their first contribution in https://github.com/spotify/luigi/pull/3297
  • @shadchin made their first contribution in https://github.com/spotify/luigi/pull/3298
  • @radeklat made their first contribution in https://github.com/spotify/luigi/pull/3277

Full Changelog: https://github.com/spotify/luigi/compare/3.5.1...v3.5.2

- Python
Published by RRap0so over 1 year ago

luigi - 3.5.1

What's Changed

  • don't convert str to tuple in TupleParameter by @kitagry in https://github.com/spotify/luigi/pull/3275
  • Fix error message in case of unfulfilled dependencies with single output by @GianlucaFicarelli in https://github.com/spotify/luigi/pull/3281
  • Add option to remove event handler from luigi.Task by @starhel in https://github.com/spotify/luigi/pull/3282
  • Fix execution summary for optional parameters by @starhel in https://github.com/spotify/luigi/pull/3286
  • Support Python3.12 by @ukky17 in https://github.com/spotify/luigi/pull/3266
  • Fixed visualization of SVG graph for tasks directly dependent on each… by @starhel in https://github.com/spotify/luigi/pull/3287
  • Version 3.5.1 by @andresgomezfrr in https://github.com/spotify/luigi/pull/3291

New Contributors

  • @kitagry made their first contribution in https://github.com/spotify/luigi/pull/3275
  • @GianlucaFicarelli made their first contribution in https://github.com/spotify/luigi/pull/3281

Full Changelog: https://github.com/spotify/luigi/compare/3.5.0...3.5.1

- Python
Published by andresgomezfrr almost 2 years ago

luigi - 3.5.0

What's Changed

  • worker: Log which outputs are missing when task is unexpectedly incomplete by @progval in https://github.com/spotify/luigi/pull/3258
  • Update dropbox.py for compatibility with Dropbox SDK v11.X.X by @smrohrer in https://github.com/spotify/luigi/pull/3259
  • Add python 3.11 support by @ukky17 in https://github.com/spotify/luigi/pull/3265
  • Migrate readthedocs to version 2 by @pablocasares in https://github.com/spotify/luigi/pull/3269
  • Added Veridas to companies using Luigi by @ironbar in https://github.com/spotify/luigi/pull/3268
  • Release 3.5.0 by @RRap0so in https://github.com/spotify/luigi/pull/3274

New Contributors

  • @smrohrer made their first contribution in https://github.com/spotify/luigi/pull/3259
  • @ukky17 made their first contribution in https://github.com/spotify/luigi/pull/3265
  • @pablocasares made their first contribution in https://github.com/spotify/luigi/pull/3269
  • @ironbar made their first contribution in https://github.com/spotify/luigi/pull/3268

Full Changelog: https://github.com/spotify/luigi/compare/3.4.0...3.5.0

- Python
Published by pablocasares about 2 years ago

luigi - 3.4.0

What's Changed

Added

  • Add CodeQL workflow by @jorgectf in https://github.com/spotify/luigi/pull/3244

Fixed

  • Fix compatibility with cythonized tasks. by @MichaelGrupp in https://github.com/spotify/luigi/pull/3246
  • fix flake8 error in master by @dlstadther in https://github.com/spotify/luigi/pull/3248
  • Do not escape
    tags in task tooltips, so that the line breaks are… by @ewallace-RI in https://github.com/spotify/luigi/pull/3250
  • Silence false positive UnconsumedParameterWarning by @starhel in https://github.com/spotify/luigi/pull/3235
  • Fix pymssql import by @grihabor in https://github.com/spotify/luigi/pull/3252

Full Changelog: https://github.com/spotify/luigi/compare/3.3.0...3.4.0

- Python
Published by andresgomezfrr over 2 years ago

luigi - 3.3.0

Added

  • Add hostconfigoptions to allow configuration of DockerTask for GPU resources #3234

Fixed

  • Fix bigquery retry #3238
  • Fix for applying logging configurations using the a TOML configuration file #3237

- Python
Published by sonjaer almost 3 years ago

luigi - 3.2.1

Fixed

luigi

  • Fix XSS vulnerability #3230
  • Fix typo in range_test.py #3228
  • Using defaultschedulerurl as a mounting point for not root pat… #3216
  • Deduplicate outputs in luigi.task.flatten_output (#3106) #3174

- Python
Published by honnix almost 3 years ago

luigi - 3.2.0

Added

luigi

  • Improve control over dynamic requirements #3179
  • Can pass a JSON schema to DictParameter and ListParameter #3217
  • support Python 3.10 #3140

Fixed

luigi

  • Update random seed for python 3.11 compatibility #3204
  • Fix md5 hash error for FIPS in python 3.9 #3209
  • Catch io.UnsupportedOperation exception #3215
  • Fix schema validation and add custom validators #3220

- Python
Published by sonjaer about 3 years ago

luigi - 3.1.1

3.1.1

Added

luigi

  • Add worker config to cache task completion results. (#3178)

Fixed

luigi

  • Close requests.Socket in RemoteScheduler before exiting (#3173) (#3175)
  • Use int-parameters for random.randrange() (#3177)
  • Alternate (probably more correct) fix for #3182 (supercedes #3183) #3185

- Python
Published by RRap0so over 3 years ago

luigi - 3.1.0

3.1.0

Added

luigi

  • Documentation guidance around release version increments #3074
  • Add support for naming tasks in @requires #3077
  • Add traceback_max_length parameter for error email notifications #3086
  • Document cause of Unfulfilled dependency error #3105
  • Add additional OptionalParameter datatype options #3079
  • UI: Add rerun command snippet on "show error" modal #3117
  • Add support for updating default config parser after loading luigi #3135
  • Allow batch_email.email_interval to be set in config as email-internal-minutes #3125
  • Allow TimeDeltaParameter to accept input as "seconds" #3125
  • Add EnumListParameter to top-level attribute imports #3144
  • Enable metrics_custom_import for MetricCollectors #3146
  • Improve warning when a parameter is not consumed by a task #3170

luigi.contrib

  • Add configure_job BigQuery property #3098
  • Add parquet support to BigQuery #3099
  • Add network retry logic to BigQuery #3088
  • Add run_task_kwargs property to ECS #3083
  • Add pickle_protocol attribute and configuration option to Spark #3001
  • Add pg8000 driver support to Postgres #3142

Fixed

luigi

  • Fix default value for task.disable_window #3081
  • Fix deconstructor of LocalTarget when is_tmp attribute dne #3085
  • Improved documentation reference to luigi.format.Nop import #3047
  • Fix Python 3.10 deprecation warnings #3150
  • Remove unnecessary extra call to cls.get_task_namespace() #3129
  • Fix documentation typo in notifications_test.py #3151
  • Fix docs ci #3158
  • Fix task history rendering #3153
  • Move max_shown_tasks and max_graph_nodes documentation to correct section #3156
  • Fix ability to subclass Task's Register metaclass #3154
  • Replace legacy TravicCI readme badge with GithubActions #3159

luigi.contrib

  • Fix apache ci
    • #3091
    • #3113
  • Fix documentation typo in sqla #3110
  • Fix spark cluster mode error for missing path #3111
  • Fix connection object passed to rdbms.CopyToTable's _add_metadata_columns() #3011

Removed

luigi

  • Remove @Tarrasch from codeowners #3127

Changed

luigi

  • Update license copyright year #3108
  • Improve error message on parsing default parameter value #3115
  • Group tasks by task class in svg graph #3122
  • Upgrade tenacity version #3147
  • Enable task search to be case insensitive #3157

luigi.contrib

  • Update ExternalPythonProgramTask parameters to OptionalParameter #3130
  • Allow newer versions of prometheus-client package #3163

- Python
Published by RRap0so over 3 years ago

luigi - 3.0.3

Added:

luigi
  • Adding retry on objexists #3022
  • don't use absolute path in redirect for visualizer #2785
luigi.contrib
  • kubernetes: labels are applied to pod #3007

Fixed:

luigi
  • Flush streamforsearchingtrackingurl #3000
luigi.contrib
  • Fix azureblob tests: use json instead of numpy #3032

Changed:

luigi
  • replace naive retry with tenacity #3026

- Python
Published by sonjaer almost 5 years ago

luigi - 3.0.2

3.0.2

Fixed:

luigi
  • Garbage collect task result queue when worker context exits #2973
  • Fixing problem with ListParameter and Dynamic Dependencies #2970

Changed:

luigi
  • Drop Python 3.3 and 3.4 support #2978
luigi.contrib
  • Use updated uri for gcs batch reqs #2998

- Python
Published by honnix over 5 years ago

luigi - 3.0.1

Added:

luigi
  • Worker_timeout can be 0. #2968
  • Return bq job id from biquery.run_job() #2957
  • Documentation for checkcompleteon_run config #2961

- Python
Published by ckiosidis over 5 years ago

luigi - 3.0.0

3.0.0

This is a major release without many feature changes compared to 2.8.13. The reason we decided to give it a major bump is the drop of Python2 support. From this version on, Luigi stops supporting Python2 for the obvious reason. 3.0.0 release includes a series of PRs deprecating Python2, plus a few other changes listed below. Special thanks go to @drowoseque for doing all the great work!

Added:

luigi
  • Show task progress in visualizer workers tab. #2932

Fixed:

luigi
  • Fix TravisCI build #2948
  • Use is_alive in favour of isAlive for Python 3.9 compatibility. #2940

- Python
Published by honnix over 5 years ago

luigi - 2.8.13

Added:

luigi.contrib
  • Presto support in Luigi (#2885)

Fixed:

luigi
  • removed wrong type of Target.init path arg from doc master (#2927)
  • remove StreamingBodyAdaptor that didn't allow choosing the chunk size (#2929)
  • Fix docs explaining write modes for Luigi Targets. Closes #2783 (#2931)
  • All configuration parameters in docs now use underscore in their names for consistency. (#2890)

Changed:

luigi
  • Allowed wider popovers in grapth. (#2093)
  • update documentation to prefer pykube-ng (#2924)

- Python
Published by jesperps almost 6 years ago

luigi - 2.8.12

Added:

luigi
  • EnumListParameter #2801

Fixed:

luigi
  • Import ABC from collections.abc instead of collections for Python 3.9 compatibility #2895

Changed:

luigi.contrib
  • [luigi.contrib.hive] HiveTableTarget inherits HivePartitionTarget #2872
  • [luigi.contrib.pyspark_runner] SparkSession support in PySparkTask #2862

- Python
Published by honnix about 6 years ago

luigi - 3.0.0b2

This the second 3.0.0 beta release including a series of PRs deprecating Python2, plus following:

Special thanks go to @drowoseque for doing all the great work!

Added:

luigi
  • Add internal version info #2760
  • EnumListParameter #2801 (new since 3.0.0b1)
luigi.contrib
  • [luigi.contrib.spark] pyspark python options added #2818

Fixed:

luigi
  • Fix params hashing #2540
  • Check for autoload_range istead of autoload-range
  • autoload_range doc fix #2878 (new since 3.0.0b1)

Removed:

luigi
  • [luigi.file] removed #2832
  • [luigi.mock.MockFile] removed #2839

Changed:

luigi
  • Allow python-daemon >= 2.2.0 if not on windows #2796
  • Make URLLibFetcher aware of basic auth info in scheduler URL. #2791
luigi.contrib
  • [luigi.contrib.externalprogram.ExternalProgramTask] logsoutputpatternto_url provided #2822
  • [luigi.contrib.hive] HiveTableTarget inherits HivePartitionTarget #2872 (new since 3.0.0b1)
  • [luigi.contrib.pyspark_runner] SparkSession support in PySparkTask #2862 (new since 3.0.0b1)

- Python
Published by honnix about 6 years ago

luigi - 2.8.11

Added:

luigi
  • Add internal version info #2760
luigi.contrib
  • [luigi.contrib.spark] pyspark python options added #2818

Fixed:

luigi
  • Fix params hashing #2540
  • Check for autoload_range istead of autoload-range
  • autoload_range doc fix #2878

Removed:

luigi
  • [luigi.file] removed #2832
  • [luigi.mock.MockFile] removed #2839

Changed:

luigi
  • Allow python-daemon >= 2.2.0 if not on windows #2796
  • Make URLLibFetcher aware of basic auth info in scheduler URL. #2791
luigi.contrib
  • [luigi.contrib.externalprogram.ExternalProgramTask] logsoutputpatternto_url provided #2822

- Python
Published by honnix about 6 years ago

luigi - 2.8.10

Added:

luigi
  • Add HEAD endpoint to scheduler server for status/health checks #2789
luigi.contrib
  • [luigi.contrib.hive] WarehouseHiveClient #2826

Fixed:

luigi.contrib
  • Add Python version-agnostic getwriterschema. #2827
  • PySparkTask: handle special characters in name (#2778) #2779 #### Changed:
luigi.contrib
  • [luigi.contrib.spark] trackingurlpattern as a property #2820
  • Add podcreationwait_interal #2813
  • Added optional argument 'awssessiontoken' to S3Client #2798

- Python
Published by honnix about 6 years ago

luigi - 2.8.9

Added:

luigi
  • Adds "Force Commit" button in UI to set tasks to DONE #2751
  • Show task history link in visualizer when recording. #2759

Fixed:

luigi
  • Replace documentation reference to outdated test environment py27-nonhdfs #2762
  • Issue 2644: Tasks can be run several times under certain conditions #2645
luigi & luigi.contrib
  • Ensure ignored tests are picked up by tox #2758

Changed:

luigi
  • Update tornado requirement for new enough python versions #2761
luigi.contrib
  • contrib/ftp: Clean up temporary files #2755

- Python
Published by honnix over 6 years ago

luigi - 2.8.8

Added:

luigi
  • Expandable Namespace Folders for the Visualiser Sidebar #2716
  • Added new companies to the luigi users list: #2730 #2747
luigi.contrib
  • Enable Overriding Poll Interval for Kubernetes Jobs #2724 #### Fixed:
luigi
  • Code example correction #2754

Changed:

luigi
  • Update release process #2727
  • Change GET request to POST requests in luigi/rpc #2732
  • Fix SendGrid email API documentation. #2745

- Python
Published by NatashaL over 6 years ago

luigi - 2.8.7

Added:

luigi
  • Add check_complete_on_run to optionally mark tasks as failed if complete() is false when run finishes (#2710)
  • Add section "Running Luigi on Windows" to docs (#2720)
  • Add Giphy to list of companies using Luigi to docs (#2713)
luigi.contrib
  • Add DropboxTarget for luigi (#2696)

Fixed:

luigi
  • UI: Fix time graph - y axis to account for timezones. (#2711)

Changed:

luigi
  • Bump dependencies used by SendGrid integration. (#2715)
  • Update copyright year in LICENSE (#2723)
luigi.contrib
  • Make RedisTarget compatible with redis-py >= 3 (#2722)

- Python
Published by narape over 6 years ago

luigi - 2.8.6

Fixed:

luigi
  • fix calling method on wrong object #2709 This fixes #2704.

- Python
Published by honnix almost 7 years ago

luigi - 2.8.5

Fixed:

luigi
  • (Doc) Fix example of "summary_length" #2700
  • Fix __init__ error when using TOML config #2702
  • add callback to metric collector #2704
luigi.contrib
  • Fix BigQueryTarget parsing in beam_dataflow module #2705

Changed:

luigi.contrib
  • aws batch : job queue as parameter #2689

- Python
Published by honnix almost 7 years ago

luigi - 2.8.4 (BROKEN)

This release is broken due to #2628 .

Added:

luigi
  • Added support for a detailed LuigiRunResult instead of a plain Boolean (#2630)
  • Add worker option 'maxkeepaliveidleduration (#2654)
  • Added worker-id commandline parameter (#2655)
luigi.contrib
  • Add support for specifying kubernetes namespace (#2629)
  • Add a Task wrapper for MicroSoft OpenPAI (#2531)
  • Provide automatic URL tracking for Spark applications (#2661) (#2669)
  • add beamdataflowtask to luigi/contrib (#2675)
Both
  • Add Prometheus contrib for monitoring purpose (#2628)

Fixed:

luigi
  • setup.py: Support older setuptools (<=20.1.1) (#2623)
  • Fix broken aws tests (#2658)
  • Accept pathlib based path as argument for LocalTarget (#2548)
  • Fix durations in D3 graph (fixes #2620) (#2624)
  • Import collections ABCs from collections.abc, not collections (#2683)
  • Configuration documentation: remove deprecated/wrong [core]max_reschedules entry (#2692)
luigi.contrib
  • fixes #2223 HdfsTarget is not working with snakebite (#2572)
  • Add port field for PostgresQuery (fixes #2625) (#2627)
Both
  • Fix flake errors after moving to python 3 (#2695)

Changed:

luigi
  • Simplify implementation of temporary_path() (#2652)
  • Prevent range-tasks from autoloading (#2656)
  • Replace Python MapReduce example with Spark example. (#2668)
  • Minor improvements to single-worker-timeout support (#2667)
  • Require at least python-dateutil version 2.7.5 instead of only 2.7.5 (fixes #2662) (#2679)
  • RangeMonthly should deal with whole months (#2666)
  • Reconcile underscore/dash config style handling (#2688) #2691
luigi.contrib
  • Add autodetect parameter to BigQueryLoadTask (#2363) (#2575)

- Python
Published by zatine almost 7 years ago

luigi - 2.8.3

Added:

luigi
  • Add BaseTIS to the company list #2607
  • Add Hopper to the company list #2614
  • give a few default values to opts when setting up logging #2612
  • Add range functionality for monthly cadence. #2601
luigi.contrib
  • Added port to PostgresTarget #2615
  • Support for Azure Blob Storage Target #2585
  • Add Datadog contrib for monitoring purpose #2434

Fixed:

luigi
  • Docs: Fixed a mistake with @inherits syntax in luigi/util.py #2613
  • Check type of column before migrating schemas for task db history for postgres dialect (fixes #2563) #2564
luigi.contrib
  • S3: Fix call to message from TypeError not working with Python 3.6 #2617
  • Use proper API call in bigtable.py's make_dataset #2618
  • Sqla: Fix the table name when reflect is True in sqla.CopyToTable (fixes #2604) #2605

Changed:

luigi.contrib
  • Changed to buffered reads when using GCSTarget #2588

- Python
Published by NatashaL about 7 years ago

luigi - 2.8.2 (Hotfix)

  • fix logging setup in Python 2.7 (#2593)

- Python
Published by dlstadther about 7 years ago

luigi - 2.8.1(broken)

Note: Broken due to a runtime error in LuigiConfigParser. See https://github.com/spotify/luigi/issues/2592.

Added:

luigi
  • Add some docs to interface.run #2582
  • Configure logging via TOML config #2483
luigi.contrib
  • Added port property to CopyToTable #2561
  • Make it so we can do from luigi.contrib.hdfs import HdfsFlagTarget #2594
  • contrib: Add ExternalDailySnapshot #2591

Fixed:

luigi
  • (docstring) Update task.py #2589
  • Docs: Fixed "Github" to fit to the rest of the doc #2596
  • Fix inspect.getargspec() DeprecationWarning in PY3 #2579
luigi.contrib
  • Fix Travis Moto Test Failures #2586
  • Reduce TravisCI Test Runtime #2541

Changed:

luigi
  • Make Worker parameter taskprocesscontext an OptionalParameter #2574
luigi.contrib
  • S3Client improvements #2569

- Python
Published by NatashaL about 7 years ago

luigi - 2.8.0

This is a minor version bump, due to:

  • Dropping Python 3.4 and 3.5 from CI, which means no automated tests to ensure compatibility for those versions
  • [Security Patch] CORS being disabled by default. A new section of configuration [cors] is introduced to enable custom settings. For details, refer to user group topic: https://groups.google.com/forum/#!topic/luigi-user/ZgfRTpBsVUY

Added

luigi:
  • Add Python 3.7 compatibility (#2466) This also drops 3.4 and 3.5 from CI.
  • Interpolate environment variables in .cfg config files (#2527)
luigi.contrib:
  • Add CopyToTable task for MySQL (#2553)
  • Add HdfsFlagTarget (#2559)
luigi.contrib:

Fixed

luigi:
  • Fix ReadTheDocs build (#2546)
  • Make capture_output non-positional in ExternalProgramTask (#2547) ###### luigi.contrib:
  • Fix S3Client's pathtobucketand_key to support keys with question marks (#2534)
  • Fix S3Client.remove - add max batch size (#2529)
  • Small fix to logging in contrib/ecs.py (#2556)
  • FIX HdfsAtomicWriteDirPipe.close() when using snakebite and the file do not exist. (#2549)

Changed:

luigi:
  • [ImgBot] optimizes images (#2555)
luigi.contrib:
  • Remove s3 bucket validation prior to file upload (#2528)
  • Refactor s3 copy into sub-methods (#2508)

- Python
Published by honnix over 7 years ago

luigi - 2.7.9

Added

luigi.contrib:
  • Added optional choice for hdfs clients (#2487)
  • s3client check for deprecated host keyword and raise error with the details (#2493)
  • Add a "capture_output" parameter to ExternalProgramTask (#2430)

Fixed

luigi:
  • Fix exception when toml lib is not installed (#2506)
  • Replace direct attribute accessing by using built-n function getattr (#2509)
  • set upper bound of python-daemon (#2536) ###### luigi.contrib:
  • Fix S3Client.copy return value consistency (#2488)
  • Update MockTarget mode to accept r* or w* (#2519)

- Python
Published by honnix over 7 years ago

luigi - Version 2.7.8 (hotfix)

  • revert tornado upgrade (#2504) Upgrading tornado unfortunately breaks older version of Python

- Python
Published by honnix over 7 years ago

luigi - Version 2.7.7 (partly broken)

Added

luigi:
  • Add Data Revenue to the blogged list (#2472)
  • Add default reviewers in CODEOWNERS (#2465)
  • Optional TOML configs support (#2457)
  • Add support for multiple requires and inherits arguments (#2475)
  • Add a visiblity level for luigi.Parameters (#2278)
  • Make logging of RPC retries configurable #2486
  • Added a new event 'progress' (#2498) ###### luigi.contrib:
  • Additions to provide support for the Load Sharing Facility (LSF) job scheduler (#2373)
  • Added default port behaviour for Redshift (#2474)
  • Add metadata columns to the RDBMS contrib (#2440)
  • Use passed password when create a redis connection (#2489)

Changed

luigi:
  • Update supplementary github files to improve repo organization and maintenance (#2463)
  • Use taskid in Task.eq_ comparison (#2462)
  • Replace luigi.Task by RunOnceTask in schedulervisualisationtest (#2476)
  • (Breaking change) Bump tornado milestone version (#2490) This changes requires Python version 2.7.9+ and 3.4+ ###### luigi.contrib:
  • S3 client refactor (#2482)
  • Update moto to 1.x milestone version (#2471)

Fixed

luigi:
  • Fix Scheduler.addtask to overwrite acceptsmessages attribute. (#2469)
  • Fix race condition (#2477)
  • Fix attribute forwarding for tasks with dynamic dependencies (#2478) ###### luigi.contrib:
  • Fix transfer config import (#2458)

Removed

luigi:
  • Remove long-deprecated scheduler config variable alternatives (#2491)

- Python
Published by honnix over 7 years ago

luigi - Version 2.7.6

Added

luigi:
  • Add a configuration parameter to force multiprocessing (#2401)
  • Add a configuration parameter to enable/disable the pause button (#2399)
  • Send messages from scheduler to tasks (via "Send message" UI button) (#2426)
  • Allow to inject a context manager around TaskProcess.run (via task_process_context configuration parameter) (#2449) ###### luigi.contrib:
  • S3: use Boto3 for the S3Client (#2423, #2149)
  • GCS: add method to push files using multiprocessing (#2376)
  • HDFS: add get_merge to snakebite client (#2410)
  • Redshift: add schema to DB if it doesn't exist (#2439)
  • Redshift: add table constraints support (#2435)

Fixed

luigi:
  • Allow long parameters in task history DB SQL result store (#2404)
  • Fix MissingParameterException when generating execution summary (#2415)
  • Fix luigid crash due to configuration file parsing (#2394)
  • Allow explicit parsing of BoolParameters (via luigi.BoolParameter.parsing variable) (#2427)
  • Make ChoiceParameter check if option is valid within .normalize (#2454)
  • ...and a good deal of documentation fixes and similar. ###### luigi.contrib:
  • BigQuery: fix bulk_complete failing when argument is a generator (#2441)
  • Kubernetes: prevent KeyError in KubernetesJobTask (#2433)
  • Kubernetes: don't set activeDeadlineSeconds by default (#2452)

- Python
Published by ulzha over 7 years ago

luigi - Version 2.7.5

Fixed

luigi
  • append is vulnerable of xss #2391

- Python
Published by zatine almost 8 years ago

luigi - Version 2.7.4

Added

luigi:
  • Add google-auth-httplib2 as dependency (#2384)
  • Add new company user (#2388)
  • Allow release of resources during task running. (#2346) ###### luigi.contrib:
  • Add parameterized backoff limit (#2375)

Fixed

luigi:
  • Typo in running luigi documentation (#2387)
  • Skip running coverage on unreasonable files
  • Pass kwargs to discovery.build() when instantiating GSCClient. (#2291)
  • Removed message 'No Instance(s) Available.' in Windows when starting … #2294

- Python
Published by zatine almost 8 years ago

luigi - Version 2.7.3

Added

luigi:
  • Added generated data files to .gitignore (#2367) ###### luigi.contrib:
  • Add possibility to specify Redshift column compression (#2343)

Changed

luigi:
  • Show status message button in worker tab when only progress is set (#2344)
  • Verbose worker error logging (#2353) ###### luigi.contrib:
  • Replace oauth2client by google-auth (#2361)

Fixed

luigi:
  • Fix unicode formatting (#2339) ###### luigi.contrib:
  • Fix contrib.docker_runner exit code check (#2356)

- Python
Published by honnix almost 8 years ago

luigi - Version 2.7.2

Added

luigi:
  • A button to show errors for disabled tasks in the visualizer (#2266)
  • OptionalParameter parameter class (#2295)
luigi.contrib:
  • Support for the Docker Python SDK (#2158)

Changed

luigi:
  • Change the logging status of prune messages from info to debug (#2254) - reduce repetitiveness of logging
  • Speed up the visualizer so that a refresh doesn’t take minutes when number of tasks gets into the millions (#2239)
  • Hide "re-enable" and "forgive failures" buttons on success (#2281)
  • Convert dates to datetimes for DateHourParameter (#2285)
  • Handle multiprocessing with request sessions (#2290) - Fixes bug with "RPCError: Received null response from remote scheduler"
  • Replaced param_args with dynamic property with deprecation message
  • Check task equality using param_kwargs instead of param_args

Removed

luigi.contrib:
  • Copying of Avro field documentation by BigQueryLoadAvro (#2269) - said copying is done by BigQuery natively now

Fixed

luigi:
  • Remove invalid entrypoint luigi.tools.migrate (#2257)
  • Preserve filter on server on reload (#2273)
  • Fix MRO on tasks using util.requires (#2307)
luigi.contrib:
  • Fix Python 3 TypeError in contrib.hive.HiveTableTarget.exists (#2323)

- Python
Published by bergman about 8 years ago

luigi - Version 2.7.1

Luigi 2.7.1 is mostly bug fixes and small feature additions.

  • Standardize Redshift credential usage across Redshift tasks: https://github.com/spotify/luigi/pull/2068
  • BigQueryLoadAvro now handles complex Avro types: https://github.com/spotify/luigi/pull/2224
  • Support for user-specified number of paralleled scheduled processes: https://github.com/spotify/luigi/pull/2205
  • ECS support for non-default cluster: https://github.com/spotify/luigi/pull/2045
  • BigQuery ExtractTask support: https://github.com/spotify/luigi/pull/2134
  • Support for autocommitting queries within Redshift and Postgres, allowing VACUUM statement execution: https://github.com/spotify/luigi/pull/2242
  • High Availability (HA) support with WebHdfsClient using multiple namenodes: https://github.com/spotify/luigi/pull/2230
  • Column mapping support for Redshift S3CopyToTable using the column list: https://github.com/spotify/luigi/pull/2245

There have also been some other feature additions, bugfixes, and docfixes. See all commits here.

- Python
Published by dlstadther over 8 years ago

luigi - Version 2.7.0

Introducing Luigi 2.7.0!

  • Removed deprecations! luigi.{postgres, s3} are removed, use luigi.contrib.{..} instead https://github.com/spotify/luigi/pull/2181
  • As of the bullet above. We now believe all contrib and core modules are well seperated by either being under either luigi.xyx or luigi.contrib.xyz. This should simplify to move modules into seperate repositories, which is future work.
  • Luigi finally has a progress bar! https://github.com/spotify/luigi/pull/2108
  • Scheduling improvements / bug fixes: https://github.com/spotify/luigi/pull/2208 https://github.com/spotify/luigi/pull/2207
  • Luigi can now run PySpark in cluster mode https://github.com/spotify/luigi/pull/2197
  • The batch email notifier shouldn't stall any more on failures. https://github.com/spotify/luigi/pull/2177
  • A global panic "stop scheduling button" https://github.com/spotify/luigi/pull/2145
  • Fixed error emails from workers: https://github.com/spotify/luigi/pull/2191
  • My personal favorite PR, as it removes many deprecation warnings when running the tests: https://github.com/spotify/luigi/pull/2130

There have been a lot of other bugfixes, docsfixes, cleanups and added testcases! See all commits here.

- Python
Published by Tarrasch over 8 years ago

luigi - Version 2.6.2

Luigi 2.6.2 is also mainly about bug-fixes. Including many fixes in the scheduler web interface.

  • One new feature is forgiving failures. https://github.com/spotify/luigi/pull/2091 (thanks @FxIII!)
  • The annoying "is not of type" warnings are now easier to fix/trace, as the var name is also included in the log message https://github.com/spotify/luigi/pull/2107 (thanks @Lucas-C!)

- Python
Published by Tarrasch over 8 years ago

luigi - Version 2.6.1

Luigi 2.6.1 is mainly about bugfixes. Some highlights:

  • Fixed the installation issue that plagued 2.6.0 https://github.com/spotify/luigi/pull/2033 (thanks @Aaronhrndz)
  • Fixes to temporary_path decorator. Awesome! (https://github.com/spotify/luigi/pull/2057 and https://github.com/spotify/luigi/pull/2058, thanks @adaitche!)

There actually is more! Read the whole commit log here!

- Python
Published by Tarrasch almost 9 years ago

luigi - Version 2.6.0

Luigi 2.6.0 comes with many new cool features! - Removed deprecations! luigi.{hadoop, hadoop_jar, hdfs, hive, scalding, webhdfs} are removed, use luigi.contrib.{..} instead https://github.com/spotify/luigi/pull/1995 - Deprecations! luigi.{postgres, s3} are now moved into luigi.contrib. https://github.com/spotify/luigi/pull/1997 - Multiple workers finally works for Windows again! https://github.com/spotify/luigi/pull/1992 - Server can now communicate with the clients! We started small and implemented so --workers can be set. https://github.com/spotify/luigi/pull/1993 - Make the visualizer put your search queries in the URLs hash. So links are now finally shareable! https://github.com/spotify/luigi/pull/1986 https://github.com/spotify/luigi/pull/2002 - A new recommended way to automatically set the task namespace! https://github.com/spotify/luigi/pull/2000 (docs)

There have been a lot of other bugfixes, docsfixes, cleanups and added testcases! See all commits here.

- Python
Published by Tarrasch about 9 years ago

luigi - Version 2.5.0

This releases contains mostly bugfixes, but also changes to the otherwise quite stale luigi core.

Most users will probably not have anything break. But at least read the warnings placed below to see what could've have changed for you.

luigi:
  • Changed behavior warning! BooleanParameter is now removed after a long deprecation. Instead simply use BoolParameter. https://github.com/spotify/luigi/pull/1959
  • Make luigi Task classes more pythonic and functional:
    • Changed behavior warning! task_namespace is now inherited as usual in python and not overridden my metamagic from luigi. https://github.com/spotify/luigi/issues/1950 (Thanks @riga).
    • Changed behavior warning! externalize now goes out of it's way to ensure it doesn't mutate its input anymore, and returns a copy. Allowing for new cool usage patterns. https://github.com/spotify/luigi/pull/1975 (docs) (shameless thanks @Tarrasch :p)
    • Concepts like task namespace and friends are now documented. Curious folks can read the new docs. :)
  • Further bigquery improvements from @Spotify engineers: https://github.com/spotify/luigi/pull/1896 https://github.com/spotify/luigi/pull/1946 (Thanks @ukarlsson and @fabriziodemaria and more)
  • Various bugfixes:
    • Fix serialization of TimeDeltaParameter https://github.com/spotify/luigi/pull/1968 (Thanks @kierkegaard13)
    • Fix execution summary and return codes for succesfully retried tasks https://github.com/spotify/luigi/pull/1951 (Thanks @bwtakacy)

There have been a lot of other bugfixes, docsfixes, cleanups and added testcases! See all commits here.

- Python
Published by Tarrasch about 9 years ago

luigi - Version 2.4.0

This release come with a few new features and some changed behaviors. Hopefully bringing us a tiny step towards scheduling heaven.

luigi:
  • Luigi can now send error emails in easy-to-digest batches. https://github.com/spotify/luigi/pull/1916 Thanks @daveFNBuck
  • New behavior for task.clone() https://github.com/spotify/luigi/pull/1630 Thanks @ChrisBeaumont
  • Various bigquery improvements from @Spotify engineers: https://github.com/spotify/luigi/pull/1888 https://github.com/spotify/luigi/pull/1917
  • Add target.temporary_path() as a new recommended ways to deal with temporary paths in luigi https://github.com/spotify/luigi/pull/1909 (docs)
luigi.scheduler:
  • Reverted the behavior introduced in 2.3.0. https://github.com/spotify/luigi/pull/1926

There have been a lot of other features, cleanups and bugfixes! See all commits here.

- Python
Published by Tarrasch about 9 years ago

luigi - Version 2.3.3

Biggest risk of breakage for people updating early: - File locking strategy just got changed (on Unix) https://github.com/spotify/luigi/pull/1886 Thanks @nmandery

Other things: - Range for Minutes. https://github.com/spotify/luigi/pull/1863 Thanks @j-santander - CORS headers for responses from scheduler. https://github.com/spotify/luigi/pull/1870 Thanks @jessicaaustin - Docs and configurationf for email updated. Finally! https://github.com/spotify/luigi/pull/1871 Thanks @daveFNbuck - Further fixes to Batch running functionality. https://github.com/spotify/luigi/pull/1858 Thanks @daveFNbuck

And many more other small improvements. Thanks to everyone who've contributed!

- Python
Published by Tarrasch over 9 years ago

luigi - Version 2.3.2

This is mostly a bug-fix release.


  • Changed behaviour (read bugfix) in local locking https://github.com/spotify/luigi/pull/1842
  • Changed behaviour (read bugfix) in disabling workers https://github.com/spotify/luigi/pull/1839
  • Many bugfixes to the scheduler, particularly related to the Batch running functionality.

Here are the changes commit by commit.

- Python
Published by Tarrasch over 9 years ago

luigi - Version 2.3.1

This release mainly fixes bugs introduced in the two latest releases and also some older bugs.


  • Bugfix regarding sftp https://github.com/spotify/luigi/pull/1825
  • Bugfix regarding error emails with smtp https://github.com/spotify/luigi/pull/1821
  • Bugfix regarding spark tasks https://github.com/spotify/luigi/pull/1819
  • Bugfixes regarding visualiser https://github.com/spotify/luigi/pull/1817 https://github.com/spotify/luigi/pull/1818

Here are the changes commit by commit.

- Python
Published by Tarrasch over 9 years ago

luigi - Version 2.3.0

There's been over a month since the last release. This new release includes a bunch of new features. What I like the most is that they all come with full and proper documentation!

luigi:
  • Fixed lockfile bugfix introduced in 2.2.0 https://github.com/spotify/luigi/pull/1809 Thanks @nugend
  • Retcodes improved https://github.com/spotify/luigi/pull/1771 Thanks @fabriziodemaria
  • Added NumericalParameter and ChoiceParameter Thanks @brcopeland
  • Per-task configurable retry policies! https://github.com/spotify/luigi/pull/1791 Thanks @javrasya
  • Luigi can now batch tasks! https://github.com/spotify/luigi/pull/1784 Thanks @daveFNbuck
luigi.scheduler:
  • We now have a new definition of the UPSTREAM statuses. New intuition is that UPSTREAM_FAILED means that that task cannot run because all downstream tasks have failed or worse (like being disabled). As an effect there will be much much fewer tasks considered to have an upstream status. https://github.com/spotify/luigi/pull/1789

There have been a lot of other features, cleanups and bugfixes! See all commits here.

- Python
Published by Tarrasch over 9 years ago

luigi - Version 2.2.0

There's been 3 months since the latest release. Making Google's results on readthedocs outdated and gives a stale feeling to luigi. Enjoy updated and hopefully bugfree software. :)

At least read these

  • Luigi finally has user-land configurable task status messages https://github.com/spotify/luigi/pull/1625
  • Parameters: From now on, you must not pass None as the default for a parameter. Usually, passing the empty string '' is a sufficient replacement. If you don't do this luigi will print a deprecation warning. https://github.com/spotify/luigi/pull/1624.
  • Logging for server: Things are greatly improved now as of https://github.com/spotify/luigi/pull/1633 and https://github.com/spotify/luigi/pull/1636. Here's my jotted down usage info about it here https://github.com/spotify/luigi/issues/1752#issuecomment-231266977.

Main changes

luigi:
  • More fine grained eventhandlers https://github.com/spotify/luigi/pull/1698
  • Range: Finally a proper way to pass along parameters: https://github.com/spotify/luigi/pull/1675
  • From this release, we'll also bump the debian verion number. https://github.com/spotify/luigi/pull/1718
  • Print your dependency tree as ascii art! https://github.com/spotify/luigi/pull/1680
  • We now have a template for PR's! https://github.com/spotify/luigi/pull/1655
luigi.contrib:
  • AWS: You can now set the session token: https://github.com/spotify/luigi/pull/1702/files
  • Salesforce: Add support for multiple results https://github.com/spotify/luigi/pull/1686
  • FTP: Configurable port https://github.com/spotify/luigi/pull/1689
  • MSSQL support: https://github.com/spotify/luigi/pull/1650
  • Streaming mapreduce: Allow additional archives https://github.com/spotify/luigi/pull/1649
  • Streaming mapreduce: Recognize the Google File System formats https://github.com/spotify/luigi/pull/1664
  • Streaming mapreduce: mrrunner.py is not hardcoded as the binary being run https://github.com/spotify/luigi/pull/1565

Various goodies

  • Cleanup: https://github.com/spotify/luigi/pull/1734
  • Cleanup: https://github.com/spotify/luigi/pull/1735
  • Cleanup: https://github.com/spotify/luigi/pull/1723
  • Cleanup: https://github.com/spotify/luigi/pull/1677
  • Cleanup: https://github.com/spotify/luigi/pull/1640
  • Cleanup: https://github.com/spotify/luigi/pull/1722
  • Speedup: https://github.com/spotify/luigi/pull/1737
  • Speedup: https://github.com/spotify/luigi/pull/1681
  • Docs: https://github.com/spotify/luigi/pull/1729
  • Docs: https://github.com/spotify/luigi/pull/1662
  • Docs: https://github.com/spotify/luigi/pull/1716
  • Bugfix: https://github.com/spotify/luigi/pull/1708
  • Bugfix: https://github.com/spotify/luigi/pull/1643
  • Improved error message: https://github.com/spotify/luigi/pull/1692

Contribution spirit

A few great examples that show how improvements are well-receieved no matter how small they are. Yet these "small" changes helps hundreads of people reading the docs of luigi. https://github.com/spotify/luigi/pull/1672 https://github.com/spotify/luigi/pull/1642

Other changes

There were even more changes which we didn't include in these release notes. Like every contribution, whether merged or not we are happy for getting them. So please keep contributing! :)

- Python
Published by Tarrasch over 9 years ago

luigi - Version 2.1.1 (Includes security fix)

Last release was only 2 business days ago (as opposed to the 5 months since the one before it). But this release got rushed as of a security fix!

In addition to doc fixes:

Additions

Security bugfix

  • The server now have an explicit whitelist of external commands.
    • Previous potential harm: Malicious hackers can run arbitrary code if they have file system (even external mounts!)+network access on the machine running luigid (executed by the user that you run luigid with).

We wait for a while with saying how to use this exploit, giving time to people to apply the bugfix.

- Python
Published by Tarrasch almost 10 years ago

luigi - Version 2.1.0

Finally, a new PyPI release in a rather long while. Thanks @Tarrasch and @erikbern, and all the contributors!

Added

luigi:
  • Notifications: more emails and proper coloring (#1471), improved SMTP handling
  • EnumParameter (#1479), DictParameter (#1574)
  • Support for Python 3.5 (#1494)
  • Process locking on Alpine Linux (#1530) and Windows (#1557)
  • Visualizer: resources tab (#1566), GUI functionality to disable a worker (#1564)
luigi.contrib:
  • ExternalBigqueryTask (#1434), BigqueryCreateViewTask (#1465)
  • Luigi tasks for Dataproc, Google's managed Hadoop MapReduce, Spark, Pig, and Hive service (#1601)
  • ExternalProgramTask, ExternalPythonProgramTask - commonalities for running any external application or script (#1520)
  • Support for SFTP (#1585)
  • Sped-up Hive client using Metastore (#1533)
  • OpenerTarget, a single Luigi target to open multiple file system types (#1555)
  • Query base task (giving rise also to luigi.contrib.redshift.RedshiftQuery and luigi.postgres.PostgresQuery) (#1493)
  • RedshiftUnloadTask (#1527)
  • UploadToSalesforceTask (#1404)
  • Support for S3 assumed role (#1596)

Changed

luigi:
  • Semi-opaque, hashed task_id (as opposed to TaskName(param1=value1, param2=foo bar)) (#1444)
  • More explicit way to handle timelike parameters (date vs datetime) (#1473)
  • Optimizations in scheduler algorithm

Removed

luigi:
  • Old deprecated (2014) stuff around scheduler and its state (#1592)
luigi.contrib:
  • Deprecated classes SparkJob, Spark1xBackwardCompat, Spark1xJob, PySpark1xJob (#1442)

Fixed

luigi.contrib:
  • Ensure that FTP RemoteTarget successfully creates temporary files (meaning, in a directory relative to output) (#1515)
  • Remove superfluous init_mapper()/init_reducer() calls in LocalJobRunner (#1475)
  • Humanly format HadoopJobError (#1528)
  • Broken Redshift table creation (#1453)
  • Improved Salesforce reliability (#1597, #1600)
  • Missing call to post_copy() (#1502)

...and a slew of other additions, fixes, improvements and documentation.

- Python
Published by ulzha almost 10 years ago

luigi - Version 2.0.1

This version contains mostly bugfixes, in particular making Luigi 2.0 working on Windows

Thanks @daveFNbuck @econchick @Tarrasch @dlstadther @DeaconDesperado @jpiper @nresare @dhurlburtusa @stephenpascoe @mbruggmann and some more people I probably forgot!

Pull requests - #1441 don't print nested stack traces - #1439 spark: use list2command - #1447 fix unix socket - #1446 fix daemon - #1448 support symlinks in hdfs - #1431 #1433 worker as a context manager - #1429 keyboardinterrupt fix - #1428 make it work without tornado - #1425 positional parameter qualification - #1422 fixed broken task trigger - #1411 mention misspelled task name - #1418 bigquery fix - #1417 visualizer fix - #1413 kwargs for s3 - #1345 retry getwork - #1315 limit graph size - #1391 amazon sns support - #1392 pyspark parameter fix - #1395 tornado fix - #1397 spark package fix - #1400 serialize TaskParameter - #1402 kwargs for boto s3 - #1388 disable email from cfg - #1341 getwork fix - #1340 stop subprocesses on exception - #1352 fix bug in scheduler - #1357 fix issues in requires during scheduling - #1367 fix visualizer w weird task names - #1372 s3target.path - #1314 redshift updates - #1370 gmail authentication for notifications - #1366 pig fix - #1331 path warning fix - #1338 ssh forwarding agent - #1337 ssh multiplexing - #1363 windows signal fix - #1298 tracking fix - #1346 fix dep graph on firefox

- Python
Published by erikbern about 10 years ago

luigi - 2.0.0

Highlights: - Improved documentation. - New visualiser page! https://github.com/spotify/luigi/pull/1086, thanks @stephenpascoe - Execution summaries at end of runs! https://github.com/spotify/luigi/pull/1091, thanks @nicolehedblom - Smarter layout algorithm for the SVG dependency graphs, thanks @daveFNbuck - Command line interface now has a useful --help, see https://github.com/spotify/luigi/pull/1276 - Luigi finally has semantic exit codes! https://github.com/spotify/luigi/pull/1264

Contrib: - Stablization of GCP tools - AWS EC2 support. https://github.com/spotify/luigi/pull/1317, thanks @jfeala - Sungrid support. https://github.com/spotify/luigi/pull/1124, thanks @jfeala (again :))

Scheduling: - UNIX sockets are now supported, hence also encryption. https://github.com/spotify/luigi/pull/1129, thanks @graingert - Smarter scheduling! https://github.com/spotify/luigi/pull/1272 - Speedup in scheduler. https://github.com/spotify/luigi/pull/1046 - A --take-lock option to smoother roll out code changes. https://github.com/spotify/luigi/pull/1137

Deprecations: - There have been a lot of deprecations of features regarding Parameters. Before upgrading, please consider taking a look at the new module documentation for parameters.

- Python
Published by Tarrasch over 10 years ago

luigi - 1.3.0

Notable changes: - is_global finally removed! (It still exists to not break old code, but it only sets positional=False) - Initial support for Google Cloud Storage and Google BigQuery. Thanks @mikekap! https://github.com/spotify/luigi/pull/999 https://github.com/spotify/luigi/pull/1002 - Massive luigid speedup for get_work() response times. https://github.com/spotify/luigi/pull/986 - Various bugfixes. - Standardized FileSystem/Target interface. Again thanks @mikekap! https://github.com/spotify/luigi/pull/997 - SQLAlchemy works for multiple engines. Thanks @dylanjbarth! https://github.com/spotify/luigi/pull/991

- Python
Published by Tarrasch over 10 years ago

luigi - 1.2.1

Same as v1.2.1. Only that we can't have a initial "v" as otherwise readthedocs won't build the documentation for us. See https://github.com/spotify/luigi/issues/970#issuecomment-105623162

- Python
Published by Tarrasch over 10 years ago

luigi - v1.2.1

Same as 1.2.0, only that I didn't forget to update the setup.py file this time.

- Python
Published by Tarrasch over 10 years ago

luigi - v1.2.0

Notable changes: - Dropped Python 2.6 support. https://github.com/spotify/luigi/pull/934 - New experimental "assistant" feature. - External tasks are now also registered. This might cause breakage in settings like described here - Many configuration settings have been hoisted to the new configuration system. Particularly for hdfs-related parameters. The --help command will show what new options are available. - While is_global is not removed yet. It'll get removed anytime soon. Please use the new positional=False setting as a replacement in your parameters. In fact, we suggest to always use positional=False. - The central scheduler now supports searching within groups. https://github.com/spotify/luigi/pull/903 - New dependency graph rendering using D3. https://github.com/spotify/luigi/pull/920 - The built in map reduce framework can now use ujson instead of doing eval. It's still not clear to us how much faster it is. https://github.com/spotify/luigi/pull/939 - Various bug fixes and other features.

- Python
Published by Tarrasch over 10 years ago

luigi - 1.1.1

Features better Spark support: #837 #812 #806 and various bug fixes

- Python
Published by erikbern almost 11 years ago

luigi - Version 1.1.0

This mainly features Python 3 support (thanks mostly to @gpoulin!)

1.1.0 corresponds to 0abe27e5c6be616e107745b48dff8939815da73c and has been run internally at Spotify for a while so should be fairly robust.

I would have released latest master as 1.1.0 had it not been for the bugs around Tornado (#824). Hoping to launch an 1.1.1 soon.

setup.py has recorded 1.0.24 for a while now, which is not correct for all commits after 0abe27e5c6be616e107745b48dff8939815da73c. I bumped the version of setup.py to 1.1.1

- Python
Published by erikbern almost 11 years ago

luigi - 1.0.24

.24, .23, and .22 were quick fixes of big issues that came up when trying to release .21

- Python
Published by erikbern about 11 years ago

luigi - 1.0.20

Latest release

- Python
Published by erikbern almost 12 years ago