issues with CloudSqlInstanceExportOperator - google-cloud-storage

Question

I am trying do to what seems to be a simple task but I can't make it work for now. I want to use cloud composer to gather data from a SQL database and save it in GCS. I am having permissions issues
Here is my DAG:
from airflow.contrib.operators.gcp_sql_operator import CloudSqlInstanceExportOperator
from airflow import models
import datetime
export_body = {
"exportContext": {
"kind": "sql#exportContext",
"fileType": "csv",
"uri": "gs://mybucket/export_sql.csv",
"csvExportOptions": {
"selectQuery": "select count(*) as number from some_table"
}
}
}
yesterday = datetime.datetime.combine(
datetime.datetime.today(),
datetime.datetime.min.time())
start_date = yesterday
JOB_NAME = "job_name"
default_args = {
'start_date': start_date,
}
with models.DAG(JOB_NAME,
schedule_interval="#hourly",
default_args=default_args) as dag:
sql_export_task = CloudSqlInstanceExportOperator(body=export_body,
project_id="project_id",
instance='instance',
task_id='sql_export_task')
sql_export_task
I created a specific service account that have some roles:
Cloud SQL Admin
Composer Worker
Storage Object Creator
When I create the environment i specify this account and then I upload the above DAG in the appropriate bucket.
I get this error:
"error":
"code": 403
"message": "The service account does not have the required permissions for the bucket."
"errors":
"message": "The service account does not have the required permissions for the bucket."
"domain": "global"
"reason": "notAuthorized
Traceback (most recent call last)
File "/usr/local/lib/airflow/airflow/models/__init__.py", line 1491, in _run_raw_tas
result = task_copy.execute(context=context
File "/usr/local/lib/airflow/airflow/contrib/operators/gcp_sql_operator.py", line 643, in execut
body=self.body
File "/usr/local/lib/airflow/airflow/contrib/hooks/gcp_api_base_hook.py", line 247, in inner_wrappe
return func(self, *args, **kwargs
File "/usr/local/lib/airflow/airflow/contrib/hooks/gcp_sql_hook.py", line 310, in export_instanc
'Exporting instance {} failed: {}'.format(instance, ex.content
AirflowException: Exporting instance prod failed:
"error":
"code": 403
"message": "The service account does not have the required permissions for the bucket."
"errors":
"message": "The service account does not have the required permissions for the bucket."
"domain": "global"
"reason": "notAuthorized
I thought the Storage Object Creator role should give me permission.
Should I add an other role to the service account? which one?
Any advice or solution on how to proceed would be most appreciated. Thanks!
EDIT: I added the Storage Admin role and this removed this error.
However it seems despite that my DAG is not working.
The airflow interface send mixed signals: The task has no status:
But it is somehow a success ?
I checked my bucket the csv file I hoped would be created is missing.
Any advice or solution on how to proceed would be most appreciated. Thanks!

Answer 1

I was looking into your permission denied issue, glad that you sorted it out.
I was curious as to why the CSV is missing and I'm thinking that this might have to do with it.
If fileType is CSV, you can specify one database, either by using this property or by using the csvExportOptions.selectQuery property, which takes precedence over this property.
When using a CSV you have to specify the DB either by using exportContext.databases[] or by specifying it in the query you're doing.
Let me know.