0

Problem

We use Apache Airflow (v.1.10.3) for scheduling our external data provider jobs. So far the system ran smoothly with few exceptions, and those were usually caused by us (full disk, too few database connections). However, we now experienced that Airflow simply stopped scheduling jobs at some (random) point in time, without really telling what went wrong.

Logs

Looking through the syslogs yields the following pattern (I removed the overly verbose beginning of every logline and also added some comments regarding its structure):

[2019-08-29 03:50:55,196] {jobs.py:406} INFO - Processing /path/to/airflow/dags/all_purpose_dag.py took 53.097 seconds
[2019-08-29 03:50:55,200] {settings.py:206} DEBUG - Disposing DB connection pool (PID 19287)

# Start of repeated pattern until we restart the server.
[2019-08-29 03:50:55,375] {jobs.py:1573} DEBUG - Starting Loop...
[2019-08-29 03:50:55,376] {jobs.py:1584} DEBUG - Harvesting DAG parsing results
[2019-08-29 03:50:55,376] {jobs.py:1586} DEBUG - Harvested 0 SimpleDAGs
[2019-08-29 03:50:55,376] {jobs.py:1621} DEBUG - Heartbeating the executor
[2019-08-29 03:50:55,376] {base_executor.py:124} DEBUG - 0 running task instances
[2019-08-29 03:50:55,376] {base_executor.py:125} DEBUG - 0 in queue
[2019-08-29 03:50:55,376] {base_executor.py:126} DEBUG - 120 open slots
[2019-08-29 03:50:55,376] {base_executor.py:146} DEBUG - Calling the <class 'airflow.executors.local_executor.LocalExecutor'> sync method
[2019-08-29 03:50:55,377] {jobs.py:1642} DEBUG - Ran scheduling loop in 0.00 seconds
[2019-08-29 03:50:55,377] {jobs.py:1645} DEBUG - Sleeping for 1.00 seconds
[2019-08-29 03:50:56,378] {jobs.py:1663} DEBUG - Sleeping for 1.00 seconds to prevent excessive logging
# End of repeated pattern until we restart the server.

# Here is the (slightly different) pattern again.
[2019-08-29 03:50:57,379] {jobs.py:1573} DEBUG - Starting Loop...
[2019-08-29 03:50:57,379] {jobs.py:1584} DEBUG - Harvesting DAG parsing results
[2019-08-29 03:50:57,380] {jobs.py:1586} DEBUG - Harvested 0 SimpleDAGs
[2019-08-29 03:50:57,380] {jobs.py:1621} DEBUG - Heartbeating the executor
[2019-08-29 03:50:57,380] {base_executor.py:124} DEBUG - 0 running task instances
[2019-08-29 03:50:57,380] {base_executor.py:125} DEBUG - 0 in queue
[2019-08-29 03:50:57,380] {base_executor.py:126} DEBUG - 120 open slots
[2019-08-29 03:50:57,380] {base_executor.py:146} DEBUG - Calling the <class 'airflow.executors.local_executor.LocalExecutor'> sync method
[2019-08-29 03:50:57,380] {jobs.py:1633} DEBUG - Heartbeating the scheduler
[2019-08-29 03:50:57,391] {jobs.py:193} DEBUG - [heartbeat]

Question

What parameter(s) do we have to check that might cause the scheduler to work (i.e. not crash), but not to schedule new Tasks?

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.