explanation of Service.get from pulumi - pulumi

I am using pulumi release to deploy a helm chart including many service and trying to get one of the deployed service. https://www.pulumi.com/blog/full-access-to-helm-features-through-new-helm-release-resource-for-kubernetes/#how-do-i-use-it shows we can use Service.get to achieve this goal but I failed to find any information of the parameters of the method. Could someone explain it a bit or point me to the correct documentation on Service.get?
Thanks

I think there's a bug in that post; it should be -master, not -redis-master:
...
srv = Service.get("redis-master-svc", Output.concat(status.namespace, "/", status.name, "-master"))
As for what's going on here, I'll try to explain, as you're right that this doesn't seem to be documented in a way that's easy to find, as it isn't part of the Kubernetes provider API, but rather part of the core Pulumi resource API.
To address the
If you change up the example to use -master instead, you should be able to run the Pulumi program as otherwise quoted in that blog post. Here's the complete, modified program I'm using for reference:
import pulumi
from pulumi import Output
from pulumi_random.random_password import RandomPassword
from pulumi_kubernetes.core.v1 import Namespace, Service
from pulumi_kubernetes.helm.v3 import Release, ReleaseArgs, RepositoryOptsArgs
namespace = Namespace("redis-ns")
redis_password = RandomPassword("pass", length=10)
release_args = ReleaseArgs(
chart="redis",
repository_opts=RepositoryOptsArgs(
repo="https://charts.bitnami.com/bitnami"
),
version="13.0.0",
namespace=namespace.metadata["name"],
# Values from Chart's parameters specified hierarchically,
# see https://artifacthub.io/packages/helm/bitnami/redis/13.0.0#parameters
# for reference.
values={
"cluster": {
"enabled": True,
"slaveCount": 3,
},
"metrics": {
"enabled": True,
"service": {
"annotations": {
"prometheus.io/port": "9127",
}
},
},
"global": {
"redis": {
"password": redis_password.result,
}
},
"rbac": {
"create": True,
},
},
# By default Release resource will wait till all created resources
# are available. Set this to true to skip waiting on resources being
# available.
skip_await=False)
release = Release("redis-helm", args=release_args)
# We can lookup resources once the release is installed. The release's
# status field is set once the installation completes, so this, combined
# with `skip_await=False` above, will wait to retrieve the Redis master
# ClusterIP till all resources in the Chart are available.
status = release.status
pulumi.export("namespace", status.namespace)
srv = Service.get("redis-master-svc", Output.concat(status.namespace, "/", status.name, "-master"))
pulumi.export("redisMasterClusterIP", srv.spec.cluster_ip)
When you deploy this program with pulumi up (e.g., locally with Minikube), you'll have a handful of running services:
$ pulumi up --yes
...
Updating (dev)
...
Type Name Status
+ pulumi:pulumi:Stack so-71802926-dev created
+ ├─ kubernetes:core/v1:Namespace redis-ns created
+ ├─ random:index:RandomPassword pass created
+ ├─ kubernetes:helm.sh/v3:Release redis-helm created
└─ kubernetes:core/v1:Service redis-master-svc
Outputs:
namespace : "redis-ns-0f9e4b1e"
redisMasterClusterIP: "10.103.98.199"
Resources:
+ 4 created
Duration: 1m13s
$ minikube service list
|-------------------|------------------------------|--------------|-----|
| NAMESPACE | NAME | TARGET PORT | URL |
|-------------------|------------------------------|--------------|-----|
| default | kubernetes | No node port |
| kube-system | kube-dns | No node port |
| redis-ns-0f9e4b1e | redis-helm-b5f3ea12-headless | No node port |
| redis-ns-0f9e4b1e | redis-helm-b5f3ea12-master | No node port |
| redis-ns-0f9e4b1e | redis-helm-b5f3ea12-metrics | No node port |
| redis-ns-0f9e4b1e | redis-helm-b5f3ea12-slave | No node port |
|-------------------|------------------------------|--------------|-----|
Getter functions like Service.get are explained here, in the Resources docs: https://www.pulumi.com/docs/intro/concepts/resources/get/
Service.get takes two arguments. The first is the logical name you want to use to refer to the fetched resource in your stack; it can generally be any string, as long as it's unique among other resources in the stack. The second is the "physical" (i.e., provider-native) ID by which to look it up. It looks like the Kubernetes provider wants that ID to be of the form {namespace}/{name}, which is why you need to use Output.concat to assemble a string composed of the eventual values of status.namespace and status.name (as these values aren't known until the update completes). You can learn more about Outputs and Output.concat in the Resources docs as well: https://www.pulumi.com/docs/intro/concepts/inputs-outputs/
Hope that helps! Let me know if you have any other questions. I've also submitted a PR to get that blog post fixed up.

Related

Diff values file using helm provider

I hope you can help me light some more light on my issue.
Currently I'm using:
Terraform v1.3.5
Helm provider v2.8.0
Kubernetes provider v2.16.1
Lately I've been adapting the helm provider on Terraform, to help me manage Helm releases with more resiliency. It helps a lot to be able to plan changes and see what has changed and what remains the same. It merges really well with the infrastructure details and I can manage everything with just one tool, it has been great.
There is one thing that bothers me a little. It's the terraform plan preview of the values file, it just shows you that some changes have been made, but not where or which. Let me add an example.
File main.tf:
# I'm using the "manifest" setting to calculate manifest diffs
provider "helm" {
kubernetes {
config_path = "~/.kube/config"
config_context = "clusterconfig"
}
experiments {
manifest = true
}
}
# The helm chart lives locally on my repo. I'm passing the values file and an
# override for the image tag.
resource "helm_release" "release" {
name = "example"
chart = "../../helm/chart"
namespace = "example"
wait = true
set {
name = "image.tag"
value = "latest"
}
values = [
file("helm/values-example.yaml")
]
}
This works great, the problem comes when I make a change on the values file. It shows the whole file instead of just the changes. For example in my values file I change the replicas from 1 to 2:
File values-example.yaml:
replicaCount: 1
image:
repository: test
pullPolicy: ifNotPresent
The execution:
$ terraform plan
(...)
Terraform will perform the following actions:
# helm_release.example will be updated in-place
~ resource "helm_release" "example" {
~ manifest = jsonencode(
~ {
~ "deployment.apps/apps/v1/deployname" = {
~ spec = {
- replicas = 1 -> 2
}
}
}
}
)
~ values = [
- <<-EOT
replicaCount: 1
image:
repository: test
pullPolicy: ifNotPresent
EOT,
+ <<-EOT
replicaCount: 2
image:
repository: test
pullPolicy: ifNotPresent
EOT,
]
This makes it very difficult to see which values settings have been changed, when the values file is bigger.
So then, my question, do you know if there is a way to diff the values? I would like to see only the changes instead of the whole file.
What I've seen online:
It's been asked for on GitHub, but closed https://github.com/hashicorp/terraform-provider-helm/issues/305
Maybe something like this can be implemented: Use diferent values in helm deploy through Terraform (for_each)
Thanks in advance for the help. Let me know if I can help with any more information.

Is it possible to fetch the image tag from a deployment in EKS using terraform kubernetes provider?

Context:
I'm reusing terraform modules and I deploy microservices using helm provider within terraform.
Problem:
I'm trying to translate this line into terraform code, to get the current image tag live from prod (in the interest of reusing it). I'm already using kubernetes provider's auth and doesn't make sense to pull kubectl in my CI just for this.
k get deploy my-deployment -n staging -o jsonpath='{$.spec.template.spec.containers[:1].image}'
Kubernetes terraform provider doesn't seem to support data blocks nor helm provider outputs blocks.
Does anyone know how could we get (read) the image tag of a deployment using terraform?
EDIT:
My deployment looks like this:
resource "helm_release" "example" {
name = "my-redis-release"
repository = "https://charts.bitnami.com/bitnami"
chart = "redis"
version = "6.0.1"
values = [
"${file("values.yaml")}"
]
set {
name = "image.tag"
value = "latest"
}
}
The tag will be a hash that will change often and passed on from another repo.
latest in this case should be replaced by the current running tag in the cluster. I can get it using kubectl, using the line above, but not sure how using terraform.
It turns out there are multiple ways of doing it, where the easiest one for me is to reference the set argument of the helm_release resource:
output "helm_image_tag" {
value = [ for setting in helm_release.example.set : setting.value if setting.name == "image.tag" ]
}
The output will then be a list where you can reference it in a shell script (or another scripting language):
+ helm_image_tag = [
+ "latest",
]
If the list format does not suit you, you can create a map output:
output "helm_image_tag" {
value = { for setting in helm_release.example.set : setting.name => setting.value if setting.name == "image.tag" }
}
This produces the following output:
+ helm_image_tag = {
+ "image.tag" = "latest"
}
By using terraform output helm_image_tag you can access this output value and decide what to do with it in the CI.

Airflow duplicated plugin entries in virtualenv

Was setting up Airflow (2.1.4) in a virtual environment followed by an install of a third-party plugin "pip install simple-dag-editor"
Plugin installed successfully, however upon checking the plugin list, there were duplicated entries.
(venv) root#test-server:/opt/airflow$ airflow plugins
name | source | flask_blueprints | appbuilder_views
==================+============================================================+=======================================================+=============================================================
simple_dag_editor | simple-dag-editor==0.1.1: | <flask.blueprints.Blueprint object at 0x7f69e5e427b8> | {'category': 'Admin', 'name': 'Simple DAG editor', 'view':
| EntryPoint(name='simple_dag_editor', value='simple_dag_edi | | <simple_dag_editor.app_builder_view.AppBuilderDagEditorView
| tor.simple_dag_editor:SimpleDagEditor', | | object at 0x7f69e5dd1470>}
| group='airflow.plugins') | |
simple_dag_editor | simple-dag-editor==0.1.1: | <flask.blueprints.Blueprint object at 0x7f69e5e427b8> | {'category': 'Admin', 'name': 'Simple DAG editor', 'view':
| EntryPoint(name='simple_dag_editor', value='simple_dag_edi | | <simple_dag_editor.app_builder_view.AppBuilderDagEditorView
| tor.simple_dag_editor:SimpleDagEditor', | | object at 0x7f69e5dd1470>}
| group='airflow.plugins') | |
Airflow portal also resulted in 2 entries in the "Admin" section
Any idea what is happening? I tested the setup again both on a docker container and standalone on the server. Both instances did not result in the duplicated entries therefore I am suspecting it is related to running Airflow in a Python virtual environment. The server is running on CentOS 7.
I believe you might have plugin installed twice in two different places:
In "plugins" folder as simply a python package
Installed as python package
Aiflow Allows for both types of installations, and I think if you have both - it will install both.
If you change the airflow log level to verbose, you should be able to see two entries:
"Loading plugins from entrypoints"
"Loading plugins from directory: <DIRECTORY>"
They should be followed by attempts to import the plugins.
The solution would be to remove the plugings from the "plugins" directory.
It's also possible that you have two packages that have the same entrypoint - for example if you installed it with different package name before, it could also be discovered twice. Airflow checks all pacakges available and if they have appropriate entrypoint declared, it will load it as plugin. But if you enable DEBUG logging level, you should see details. You can easily set airflow logging level by config option (or environment variable):
https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#logging-level

List of AWS instances by TAG value using powershell

We are taging our AWS instances, I will like to retrieve a list of ALL our instances (ELB, S3, EC2, Security Groups) by TAG reference. for instance we consistently TAG our resources with something like this:
{ "Key": "Project",
"Value": "bananas"
},
How can we obtain trough power-shell a list of ALL our resources that contain the TAG Project value "bananas"?
I was able to get all my EC2s using the below script:
$instance = Get-EC2Instance
-Filter #( #{name='tag:Project'; values="bananas"};
#{name='instance-state-code'; values = 16} )
| Select-Object -ExpandProperty instances #Get instance ID ignoring any terminated instances
$instance | Export-CSV "C:\ec2.csv"
But I'm not sure how to obtain all my tagged resources using one script.
Check out the AWS Resource Groups Tagging API cmdlets -- these are relatively new, so you may have to update your AWS Tools for PowerShell to the latest version to be able to use them.
Example
The example below calls Get-RGTResource for the tag Key=Project, Value=Bananas, and filters the response to all ResourceARNs that were retrieved. The ResourceARN is a unique identifier for each AWS resource, and you can use these as a starting point to call out to other AWS services to get more details about each associated resource.
(Get-RGTResource -TagFilter #{Key="Project"; Values = #("bananas")}).ResourceARN
Example Output
arn:aws:ec2:us-east-1:<accountid>:instance/i-abcd1234
arn:aws:ec2:us-west-2:<accountid>:vpc/vpc-abcd1234
arn:aws:ec2:us-east-2:<accountid>:security-group/sg-abcd1234
arn:aws:elasticloadbalancing:us-east-1:<accountid>:loadbalancer/abcd1234
arn:aws:elasticmapreduce:us-east-1:<accountid>:cluster/abcd1234
Further Reading
AWS Documentation - Get-RGTResource
AWS Documentation - Amazon Resource Names (ARNs)

Start OrientDB without user input

I'm attempting to start OrientDB in distributed mode on AWS.
I have an auto scaling group that creates new nodes as needed. When the nodes are created, they start with a default config without a node name. The idea is that the node name is generated randomly.
My problem is that the server starts up and ask for user input.
+---------------------------------------------------------------+
| WARNING: FIRST DISTRIBUTED RUN CONFIGURATION |
+---------------------------------------------------------------+
| This is the first time that the server is running as |
| distributed. Please type the name you want to assign to the |
| current server node. |
| |
| To avoid this message set the environment variable or JVM |
| setting ORIENTDB_NODE_NAME to the server node name to use. |
+---------------------------------------------------------------+
Node name [BLANK=auto generate it]:
I don't want to set the node name because I need a random name and the server never starts because it's waiting for user input.
Is there a parameter I can pass to dserver.sh that will pass this check and generate a random node name?
You could create a random string to pass to OrientDB as node name with the ORIENTDB_NODE_NAME variable. Example:
ORIENTDB_NODE_NAME=$(cat /dev/urandom | tr -dc 'a-zA-Z0-9' | fold -w 32 | head -n 1)
For more information about this, look at: https://gist.github.com/earthgecko/3089509