### Intended audience
The intended audience are analysts and other luigi *users*. After watching
these slides, you'll have a better idea of what luigi is actually doing each
time you start it. In particular the `complete()` checks and its interaction
with the central luigi scheduler.
## What is luigi?
> Conceptually, Luigi is similar to GNU Make where you have certain
> tasks and these tasks in turn may have dependencies on other tasks.
> There are also some similarities to Oozie and Azkaban. One major
> difference is that Luigi is not just built specifically for Hadoop,
> and it’s easy to extend it with other kinds of tasks
## Reminder of Task DSL
```python
class MyTask(luigi.Task):
some_parameter = luigi.Parameter(default="hello")
def complete(self):
return True or False
def requires(self):
return [TaskA(), TaskB(param='yay')]
def run(self):
print self.some_parameter, 'world'
```
## Reminder of Central Scheduler
Luigi's Task DSL doesn't dictate a scheduling paradigm. There could be
different scheduling paradigms, like polling vs event-driven. Luigi also comes
with a build in scheduler.
These slides are about the implementation of the built in scheduler. It's known
as luigid.
Summary of luigid
* about 1500 lines of python, 1000 html and 1500 js
* graphing capabilities
* A single point of failure in case of hardware catches fire.
* Distributes tasks to luigi clients, first come first serve.
Current client-server API:
* `add_task(task_id, worker_id, status)`
* `get_work(worker_id)`
* `ping(worker_id)`
When tasks fail
![]()
When tasks fail too often
The scheduler, when configured to, will DISABLE a task if it fails more than X
times in the last Y minutes. The gets reenabled again automatically after Z
minutes.
### Thanks for listening
[@Tarrasch](https://github.com/Tarrasch) on github
http://tarrasch.github.io/luigid-basics-jun-2015/
https://github.com/Tarrasch/luigid-basics-jun-2015
