0

I am running airflow as a managed service on Google Cloud Composer (GCP). I can see the below error message on the webserver URL:

Error: Server Error
The server encountered a temporary error and could not complete your request.
Please try again in 30 seconds.


This started to happen after the GCP outage during 2019-05-02 08:17 to 2019-05-02 09:41. Google Cloud Console Incident #19003.

I see the below error in Cloud Composer Environment Logs.

2019-05-02 17:11:24.254 EDT
Traceback (most recent call last): File "/usr/local/bin/airflow", line 7, in <module> exec(compile(f.read(), __file__, 'exec')) File "/usr/local/lib/airflow/airflow/bin/airflow", line 32, in <module> args.func(args) File "/usr/local/lib/airflow/airflow/utils/cli.py", line 74, in wrapper return f(*args, **kwargs) File "/usr/local/lib/airflow/airflow/bin/cli.py", line 923, in scheduler job.run() File "/usr/local/lib/airflow/airflow/jobs.py", line 196, in run session.commit() File "/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 927, in commit self.transaction.commit() File "/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 465, in commit self._assert_active(prepared_ok=True) File "/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 276, in _assert_active % self._rollback_exception sqlalchemy.exc.InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (MySQLdb._exceptions.OperationalError) (1045, "Access denied for user 'root'@'cloudsqlproxy~35.225.3.25' (using password: NO)")

{
 insertId:  "1a2jm26f86bsb6"  
 logName:  "projects/electrik-ai-dev/logs/airflow-scheduler"  
 receiveTimestamp:  "2019-05-02T21:12:08.225785192Z"  
 resource: {
  labels: {
   environment_name:  "orchestrator"    
   location:  "us-central1"    
   project_id:  "electrik-ai-dev"    
  }
  type:  "cloud_composer_environment"   
 }
 severity:  "ERROR"  
 textPayload:  "Traceback (most recent call last):
  File "/usr/local/bin/airflow", line 7, in <module>
    exec(compile(f.read(), __file__, 'exec'))
  File "/usr/local/lib/airflow/airflow/bin/airflow", line 32, in <module>
    args.func(args)
  File "/usr/local/lib/airflow/airflow/utils/cli.py", line 74, in wrapper
    return f(*args, **kwargs)
  File "/usr/local/lib/airflow/airflow/bin/cli.py", line 923, in scheduler
    job.run()
  File "/usr/local/lib/airflow/airflow/jobs.py", line 196, in run
    session.commit()
  File "/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 927, in commit
    self.transaction.commit()
  File "/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 465, in commit
    self._assert_active(prepared_ok=True)
  File "/opt/python3.6/lib/python3.6/site-packages/sqlalchemy/orm/session.py", line 276, in _assert_active
    % self._rollback_exception
sqlalchemy.exc.InvalidRequestError: This Session's transaction has been rolled back due to a previous exception during flush. To begin a new transaction with this Session, first issue Session.rollback(). Original exception was: (MySQLdb._exceptions.OperationalError) (1045, "Access denied for user 'root'@'cloudsqlproxy~35.225.3.25' (using password: NO)")
"
Show matching entries
Hide matching entries
Add field to summary line

 timestamp:  "2019-05-02T21:11:24.254905088Z"  
}

I tried restarting the environment, airflow-scheduler, airflow-worker, airflow-sqlproxy.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.