## What is luigi? > Conceptually, Luigi is similar to GNU Make where you have certain > tasks and these tasks in turn may have dependencies on other tasks. > There are also some similarities to Oozie and Azkaban. One major > difference is that Luigi is not just built specifically for Hadoop, > and it’s easy to extend it with other kinds of tasks
## What is luigi? ```python class AggregateArtists(luigi.Task): date_interval = luigi.DateIntervalParameter() def output(self): return luigi.LocalTarget("%s.tsv" % self.date_interval) def requires(self): return [Streams(date) for date in self.date_interval] def run(self): for input in self.input(): with input.open('r') as in_file: pass with self.output().open('w') as out_file: pass ```
## Feature [#445](https://github.com/spotify/luigi/pull/445): Resource limitations ```python class MyHeavyTask(luigi.Task): def run(self): pass def output(self): pass def process_resources(self): # We put estimations here, relative to what we put for other jobs return { 'database_load': 5, 'cluster_load': 7 } ```
Then you can specify how much you have for each resource: ```bash $ cat /etc/luigi/server.cfg [sectionA] ... [sectionB] ... [resources] database_load: 100 cluster_load: 1000 ``` So far, the resources are always scoped globally. It's interesting to scope it to say per machine or per rack.
## Feature [#424](https://github.com/spotify/luigi/pull/424): Dynamic requirements So instead of doing ```python def requires(self): return OtherTask(...) def run(self): f = self.input() ``` you can do ```python def run(self): f = yield OtherTask(...) ``` Strategy is to re-schedule when depedency is missing. So `run()` should be idempotent-ish.
Use case is for when you have a dependency hierarchy *within* the task. ```python def run(self): # This could be done using regular requires method machine = yield MachineConfig() file = yield (CompressedFile() if machine.is_network_bound() else UncompressedFile()) with self.output().open('w') as f: f.write(file.sum_ints()) ``` Is this safe?
### Thanks for listening @Tarrasch on github http://tarrasch.github.io/luigi-presentation/