I've followed kaxil answer in How to set up Airflow Send Email? to setup airflow, however gcp doesn't allow to set:
smtp_password = 16_DIGIT_APP_PASSWORD
explicity inside cloud composer. I checked on google documentation https://cloud.google.com/composer/docs/configure-email#smtp_password and found out that there is a way to set it by Using a command to retrieve an SMTP password.
smtp_password_cmd does not appear as a key in https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html?highlight=smtp_password_cmd#smtp-password but as an environment variable, therefore is it possibile to do as suggested by the google documentation and set smtp smtp_password_cmd Specify a command that returns the SMTP password. ?
i'm absolutely clueless on how to create a command and what to pass as an argument in Specify a command that returns the SMTP password, if for example my smtp password is : dummypassword1 could someone provide an example of such command and how to set it inside the config?
There is an better way to do this than smtp_passwd_cmd:
Use Google Secrets Manager.
Configure Google Secrets Manager as the Airflow secrets backend
Add the SMTP password to the secrets manager.
Override the Airflow configuration for smtp_password_secret to point to the secret you just created for this.
This is better explained here: https://cloud.google.com/composer/docs/composer-2/configure-email#using_a_secret_stored_in_to_retrieve_an_smtp_password
This way everything is just configuration.
Using Google Cloud Secret Manager for this purpose falls in the free tier for the service (see Secret Manager pricing)
Related
I am trying to perform a very basic command like:
gcloud compute machine-types list
And I get this error:
ERROR: (gcloud.compute.machine-types.list) There was a problem
refreshing your current auth tokens: invalid_grant: Bad Request Please
run:
It tells me to login using 'gcloud auth login' which opens up the browser.
Is it possible to use a ssh key to skip this authentication process or I have to do this always? ssh keys are for accessing compute instances only?
Just trying to understand what SSH keys are used for and how this web based authorization fits into the picture here.
Generally, you authenticate to gcloud (and GCP services) using credentials from a Google (often Gmail) account. Such accounts use 3-legged (O)Auth and this requires the browser prompt for the human to confirm the scopes etc.
If you haven't, you should confirm the prompt, copy the token provided and paste that back into gcloud so that auth will occur transparently.
This process is different than SSH'ing to Compute Engine instances.
When you run gcloud compute machine-types list, you're authenticating (and being authorized) by Google Cloud Platform to invoke (meta)services.
When you run gcloud compute ssh ..., the command uses ssh to connect you to the (Linux) instance.
NOTE gcloud auth login --no-launch-browser is available too (link). This requires you to separately launch a browser and complete the process but it doesn't launch the browser directly from the command.
If you are trying to automate some sort of service, that runs cloud commands on-demand, without operator/browser involved - your best bet would be to create a Service Account for that task, get the key for that account and activate it, using
gcloud auth activate-service-account --key-file=my-service-account-key-file.json
If this service runs on Google Cloud platform - you don't even need to deal with the key. Just associate the service account with an instance you are running.
https://cloud.google.com/compute/docs/access/create-enable-service-accounts-for-instances
I have a makefile in which I use the gcloud CLI to create a service account and set roles. I also would like to enable domain wide delegation for this service account using the gcloud CLI. I can only find instructions on how to do this using the developer console. Is it possible to enable domain wide delegation using the gcloud CLI?
According to the gCloud SDK documentions of gcloud iam, there are specific coomands to set roles and service-accounts.
The commands available for domains are list-user-verified and verify.
In this way, there is no way (yet) through the gCloudSDK to Perform G Suite Domain-Wide Delegation of Authority.
I'm confident (!) that this is not exposed through a public API.
I pinged your question to a buddy at the Goo and hopefully, he'll reply here with a definitive answer.
Currently I am using keycloak on postgres db. and the db creds are provided to environment files. Wanted to know how I can make keycloak obtain the db creds from keyvault something like Azure keyvault ? Is there any documentation / guideline around it?
As per the official documentation ,some part already done but look like still work in progress
To use a vault, a vault provider must be registered within Keycloak.
It is possible to either use a built-in provider described below or
implement your own provider. See the Server Developer Guide for more
information.
To obtain a secret from a vault instead of entering it directly, enter the following specially crafted string into the appropriate field: ${vault.entry-name} where you replace the entry-name with the name of the secret as recognized by the vault.
https://www.keycloak.org/docs/latest/server_admin/#_vault-administration
https://issues.redhat.com/browse/KEYCLOAK-3205
I'm trying to build an automatic sync solution that uses a Google Cloud storage bucket for storing data.
When I install the cloud SDK it asks for my authentication, but obviously I don't want to use my credentials on the client's server, it should be done with a service account with specific permissions, right?
The documentation just says to authenticate with your credentials. What is the security best practice here?
Found it, it's this simple command:
gcloud auth activate-service-account --key-file=credentials.json
And it works! I can upload stuff with PowerShell
The doc is here
I was sure it will be simple but couldn't find any documentation or resolution.
I'm trying to write a script using gcloud to perform some operations in my GCP instances.
Is there anyway to login/authenticate using gcloud via command line only?
Thanks
You have a couple of options here (depending on what exactly you're trying to do).
The first option is to log in using the --no-launch-browser option. This still requires interaction from a human user, but doesn't require a browser on the machine you're using:
> gcloud auth login --no-launch-browser
Go to the following link in your browser:
https://accounts.google.com/o/oauth2/auth?redirect_uri=urn%3Aietf%3Awg%3Aoauth%3A2.0%3Aoob&prompt=select_account&response_type=code&client_id=32555940559.apps.googleusercontent.com&scope=https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fuserinfo.email+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcloud-platform+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fappengine.admin+https%3A%2F%2Fwww.googleapis.com%2Fauth%2Fcompute&access_type=offline
Enter verification code: *********************************************
Saved Application Default Credentials.
You are now logged in as [user#example.com].
Your current project is [None]. You can change this setting by running:
$ gcloud config set project PROJECT_ID
The non-interactive option involves service accounts. The linked documentation explains them better than I can, but the short version of what you need to do is as follows:
Create a service account in the Google Developers Console. Make sure it has the appropriate "scopes" (these are permissions that determine what this service account can do. Download the corresponding JSON key file.
Run gcloud auth activate-service-account --key-file <path to key file>.
Note that Google Compute Engine VMs come with a slightly-different service account; the difference is described here.