How to use connection hooks with `KubernetesPodOperator` as environment variables on Apache Airflow on GCP Cloud Composer - kubernetes

I'd like to use connections saved in airflow in a task which uses the KubernetesPodOperator.
When developing the image I've used environment variables to pass database connection information down to the container, but the production environment has the databases saved as connection hooks.
What is the best way to extract the database connection information and pass it down to the container?
env_vars = {'database_usr': 'xxx', 'database_pas': 'xxx'}
KubernetesPodOperator(
dag=dag,
task_id="example-task",
name="example-task",
namespace="default",
image="eu.gcr.io/repo/image:tag",
image_pull_policy="Always",
arguments=["-v", "image-command", "image-arg"],
env_vars=env_vars,
)

My current solution is to grab the variables from the connection using BaseHook:
from airflow.hooks.base_hook import BaseHook
def connection_to_dict(connection_id):
"""Returns connection params from Airflow as a dictionary.
Parameters
----------
connection_id : str
Name of the connection in Airflow, e.g. `mysql_default`
Returns
-------
dict
Unencrypted values.
"""
conn_obj = BaseHook.get_connection(connection_id)
d = conn_obj.__dict__
if ('is_encrypted', True) in d.items():
d['password'] = conn_obj.get_password()
return d
and then passing those as environment variables to the Kubernetes pod operator.

Related

How to get internal ip of postgreSQL DB in GCP created by Terraform

I am learning terraform deployments coupled with GCP to streamline deployments.
I have successfully deployed a postgreSQL db.
Now I am trying to utilize terraform outputs to write a the private ip generated by the postgreSQL DB server to the output directory where terraform is initiated from.
What is not clear to me is:
(1) The output is defined within the same main.tf file?
(2) Where is the output parameters referenced from? I cannot find the documentation to properly aline. Such I keep getting the error upon applying: Error: Reference to undeclared resource
My main.tf looks like this
resource "google_sql_database_instance" "main" {
name = "db"
database_version = "POSTGRES_12"
region = "us-west1"
settings {
availability_type = "REGIONAL"
tier = "db-custom-2-8192"
disk_size = "10"
disk_type = "PD_SSD"
disk_autoresize = "true"
}
}
output "instance_ip_addr" {
value = google_sql_database_instance.private_network.id
description = "The private IP address of the main server instance."
}
As for the code style, usually there would be a separate file called outputs.tf where you would add all the values you want to have outputted after a successful apply. The second part of the question is two-fold:
You have to understand how references to resource attributes/arguments work [1][2]
You have to reference the correct logical ID of the resource, i.e., the name you assigned to it, followed by the argument/attribute [3]
So, in your case that would be:
output "instance_ip_addr" {
value = google_sql_database_instance.main.private_ip_address # <RESOURCE TYPE>.<NAME>.<ATTRIBUTE>
description = "The private IP address of the main server instance."
}
[1] https://www.terraform.io/language/expressions/references#references-to-resource-attributes
[2] https://www.terraform.io/language/resources/behavior#accessing-resource-attributes
[3] https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/sql_database_instance#attributes-reference
To reference an attribute of a resource, you should put something like:
[resource type].[resource name].[attribute]
In this case, the output should be:
output "instance_ip_addr" {
value = google_sql_database_instance.main.private_ip_address
description = "The private IP address of the main server instance."
}
The output attributes are listed in the documentation. It's fine to put that in main.tf.

How To Push Gatling Perf Results To EC2 Grafana/InfluxDB instance

I have spun an t2.micro Ubuntu 18.04 EC2 instance and in this EC2 instance i have installed manually Grafana and InfluxDB .
Both Grafana and InfluxDB have been installed successfully with no errors,but now what i expect is when i run Gatling tests at my
windows local ,results should get pushed live to InfluxDB and eventually to Grafana
Here is my extract of Gatling.conf settings
data {
writers = [console, file, graphite] # The list of DataWriters to which Gatling write simulation data (currently supported : console, file, graphite, jdbc)
console {
#light = false # When set to true, displays a light version without detailed request stats
#writePeriod = 5 # Write interval, in seconds
}
graphite {
light = false # only send the all* stats
host = "http://ec2-54-67-97-86.us-west-1.compute.amazonaws.com" # The host where the Carbon server is located
port = 2003 # The port to which the Carbon server listens to (2003 is default for plaintext, 2004 is default for pickle)
protocol = "tcp" # The protocol used to send data to Carbon (currently supported : "tcp", "udp")
rootPathPrefix = "gatling" # The common prefix of all metrics sent to Graphite
bufferSize = 8192 # GraphiteDataWriter's internal data buffer size, in bytes
writeInterval = 1 # GraphiteDataWriter's write interval, in seconds
}
Problem is I see no data in influx instance when i run my Gatling tests from local
ubuntu#ip-172-31-9-16:~$ influx -host ec2-54-67-97-86.us-west-1.compute.amazonaws.com Connected to http://ec2-54-67-97-86.us-west-1.compute.amazonaws.com:8086 version 1.7.7
InfluxDB shell version: 1.7.7
> show databases
name: databases
name
----
_internal
gatling
graphite
> use graphite
Using database graphite
> show series
key
---
X-Grafana-Org-Id:
Can someone help to debug this ,that why no data is being received at influx DB
I suggest you to check your graphite listener in influx.
To do it open your influxdb.conf and find [[graphite]] block.
For default settings it should look like that:
[[graphite]]
# Determines whether the graphite endpoint is enabled.
enabled = true
database = "gatlingdb"
retention-policy = ""
bind-address = ":2003"
protocol = "tcp"
consistency-level = "one"
templates = [
"gatling.*.*.*.* measurement.simulation.request.status.field",
"gatling.*.users.*.*measurement.simulation.measurement.request.field"
]
More info here: https://gatling.io/docs/current/realtime_monitoring/#influxdb

PostgresOperator in Airflow getting timeout

I made a function in Postgres that have the following statement:
FUNCTION
SET statement_timeout TO "3600s"
SELECT * FROM schema.table_name
END
FUNCTION
In Airflow I use the PostgresOperator to execute this function, but I receive the message [2018-06-01 00:00:01,066] {models.py:1595} ERROR - canceling statement due to statement timeout.
I saw that PostgresOperator uses the postgres_hook, and postgres_hook uses the psycopg2 as connector.
As I see, I can be a timeout by a cli application instead a timeout from the database.
I would like to know how to solve this thing? Do I need to configure the Psycopg in Airflow or can I use some environmental variables to set the timeout to avoid this problem?
You can pass in connection arguments into psycopg2 library through the Airflow extras property on connection. At the time of writing the postgres_hook supports the following arguments
['sslmode', 'sslcert', 'sslkey','sslrootcert', 'sslcrl', 'application_name', 'keepalives_idle']
In order to pass in the statement_timeout argument to the PostgresHook you will need to override the get_conn of the PostgresHook to accept your desired argument.
Ex. Class Method Override
class NewPostgresHook(PostgresHook):
def __init__(self, *args, **kwargs):
super(NewPostgresHook, self).__init__(*args, **kwargs)
def get_conn(self):
conn = self.get_connection(self.postgres_conn_id)
conn_args = dict(
host=conn.host,
user=conn.login,
password=conn.password,
dbname=self.schema or conn.schema,
port=conn.port)
# check for ssl parameters in conn.extra
for arg_name, arg_val in conn.extra_dejson.items():
if arg_name in ['sslmode', 'sslcert', 'sslkey',
'sslrootcert', 'sslcrl', 'application_name',
'keepalives_idle', 'statement_timeout']:
conn_args[arg_name] = arg_val
self.conn = psycopg2.connect(**conn_args)
return self.conn
You can then specify this argument on the connection extras field in the form of a JSON string.
Ex. JSON String in Connection Extras Field
{'statement_timeout': '3600s'}

kubernetes volumes and sockets

I have two containers inside the same pod. One is an haproxy container and I'm pushing the haproxy statistics to a socket inside the container. I want to access the socket inside the haproxy container from the other container. I tried to use volume type mkdir but an error occurred mentioning that there is no unix sockets under the directory which I'm trying to access.
I'm new to these technologies and please help me to solve this problem.
The yaml file is as follows.
yaml file
In reference to kubernetes documentation :
Every container in a Pod shares the network namespace, including the IP address and network ports.
You don't need to use a volume to access to haproxy statistics, just use 127.0.0.1 and the port where the process for haproxy statistics is bound.
Here is an example of a telegraph configuration container deployed in the same pod of an haproxy :
# Telegraf Configuration
[global_tags]
env = "$ENV"
tenant = "$TENANT"
[agent]
round_interval = true
metric_batch_size = 1000
metric_buffer_limit = 10000
collection_jitter = "0s"
flush_jitter = "5s"
precision = ""
debug = false
quiet = false
logfile = ""
hostname = ""
omit_hostname = false
[[outputs.influxdb]]
urls = ["http://influxdb.host:2001"]
database = "db_name"
retention_policy = ""
write_consistency = "any"
timeout = "5s"
[[inputs.haproxy]]
servers = [ "http://$STATS_USERNAME:$STATS_PASSWORD#127.0.0.1:$STATS_PORT/haproxy?stats" ]
Input use haproxy plugin, output use influxdb. $STATS_USERNAME $STATS_PASSWORDand $STATS_PORTare environment variable shared between 2 containers.

ex_modify_instance_attribute and create_node has AuthFailure error use apache-libcloud AWS EC2 driver

When I use AWS EC2 driver invoke create_node and ex_modify_instance_attribute API , I got this error:
raise InvalidCredsError(err_list[-1])
libcloud.common.types.InvalidCredsError: 'AuthFailure: AWS was not able to validate the provided access credentials'
But ex_create_subnet/ list_nodes API success , and I'm sure about I have the permission on AWS IAM to create EC2 instance.
By the way , I am using AWC cn-north-1 region.
I find create node with some parameters will got AuthFailure
The Code:
node = self.conn.create_node(name=instance_name,
image=image,
size=size,
ex_keyname=ex_keyname,
ex_iamprofile=ex_iamprofile,
ex_subnet=ex_subnet,
ex_security_group_ids=ex_security_group_ids,
ex_mincount=ex_mincount,
ex_maxcount=ex_mincount,
ex_blockdevicemappings=config['block_devices'],
ex_assign_public_ip=config['eth0']['need_eip']
)
I just delete some parameters and works:
node = self.conn.create_node(name=instance_name,
image=image,
size=size,
ex_keyname=ex_keyname,
# ex_iamprofile=ex_iamprofile,
ex_subnet=ex_subnet,
# ex_security_group_ids=ex_security_group_ids,
ex_mincount=ex_mincount,
ex_maxcount=ex_mincount,
# ex_blockdevicemappings=config['block_devices'],
# ex_assign_public_ip=config['eth0']['need_eip']
)