I have found lots of information on how to use environment variables in Google App Engine projects.
However I am yet to find some best practice on what to do with environment variables on compute engine.
Is it possible to use Google Cloud Deployment Manager to achieve this? My main goal is to simplify deployment between prod/stag/dev.
Right now I am moving towards using dotenv files.
Stack is webpack 4, express, node 10, vuejs 2.
For Compute Engine instances I'd suggest to use custom metadata. You can find detailed documentation about this here. From within your instance, you can access your custom metadata by performing an empty request to the instance().get method, for example:
GET https://www.googleapis.com/compute/v1/projects/myproject/zones/us-central1-a/instances/example-instance
Now, to set your custom metadata, you can indeed use the Google Cloud Deployment Manager. As per the doc here, you just need to add the metadata property and the relevant metadata keys and values for your VM resource, for example:
resources:
- name: my-first-vm-template
type: compute.v1.instance
properties:
zone: us-central1-a
machineType:
...[snip]...
metadata:
items:
- key: custom-key
value: "custom-value"
Related
I am trying to do MongoDB on AWS following the AWS deployment guide. It is defaulted to boot up m5.large EC2s. However, I am only experimenting so I want to use a free tier EC2. When I add t2.micro to the allowed values and set it as default I get an error as pictured below.
Is there anyway I can get MongoDB running on AWS with 3 replications using the cloudformation method with free tier t2.micro instances.? If not, any better methods?
The MongoDB on AWS - Quick Start has multiple templates that are deployed.
I notice that the NodeInstanceType is used and defined in multiple templates, presumably with the values passed from the master template to the node templates. Therefore, your changes will probably need to be made on any template that defines the NodeInstanceType parameter. I recommend you check all of the templates for such references.
Ok.. so, we have Google Secret Manager on GCP, AWS Secret Manager in AWS, Key Vault in Azure... and so on.
Those services give you libs so you can code the way your software will access the secrets there. They all look straightforward and sort of easy to implement. Right?
For instance, using Google SM you can like:
from google.cloud import secretmanager
client = secretmanager.SecretManagerServiceClient()
request = {"name": f"projects/<proj-id>/secrets/mysecret/versions/1"}
response = client.access_secret_version(request)
payload = response.payload.data.decode("UTF-8")
and you're done.
I mean, if we talk about K8S, you can improve the code above by reading the vars from a configmap where you may have all the resources of your secrets, like:
apiVersion: v1
kind: ConfigMap
metadata:
name: myms
namespace: myns
data:
DBPASS: projects/<proj-id>/secrets/mysecretdb/versions/1
APIKEY: projects/<proj-id>/secrets/myapikey/versions/1
DIRTYSECRET: projects/<proj-id>/secrets/mydirtysecret/versions/1
And then use part of the code above to load the vars and get the secrets from the SM.
So, when I was looking the interwebs for best practices and examples, I found projects like the below:
https://github.com/GoogleCloudPlatform/secrets-store-csi-driver-provider-gcp
https://github.com/doitintl/secrets-init
https://github.com/doitintl/kube-secrets-init
https://github.com/aws-samples/aws-secret-sidecar-injector
https://github.com/aws/secrets-store-csi-driver-provider-aws
But those projects don't clearly explain what's the point of mounting my secrets as files or env_vars..
I got really confused, maybe I'm too newbie on the K8S and cloud world... and that's why I'm here asking, maybe, a really really dumb questions. Sorry :/
My questions are:
Are the projects, mentioned above, recommended for old code that I do not want to touch? I mean, let's say that my code already use a env var called DBPASS=mypass and I would like to workaround it so the value from the DBPASS env would be hackinjected by a value from a SM.
The implementation to handle a secret manager is very hard. So it is recommended to use one of the solutions above?
What are the advantages of such injection approach?
Thx a lot!
There are many possible motivations why you may want to use an abstraction (such as the CSI driver or sidecar injector) over a native integration:
Portability - If you're multi-cloud or multi-target, you may have multiple secret management solutions. Or you might have a different secret manager target for local development versus production. Projecting secrets onto a virtual filesystem or into environment variables provides a "least common denominator" approach that decouples the application from its secrets management provider.
Local development - Similar to the previous point on portability, it's common to have "fake" or fakeish data for local development. For local dev, secrets might all be fake and not need to connect to a real secret manager. Moving to an abstraction avoids error-prone spaghetti code like:
let secret;
if (process.env.RAILS_ENV === 'production') {
secret = secretmanager.access('...')
} else {
secret = 'abcd1234'
}
De-risking - By avoiding a tight coupling, you can de-risk upstream API changes in an abstraction layer. This is conceptual similar to the benefits of microservices. As a platform team, you make a guarantee to your developers that their secret will live at process.env.FOO, and it doesn't matter how it gets there, so long as you continue to fulfill that API contract.
Separate of duties - In some organizations, the platform team (k8s team) is separate from the security team, is separate from development teams. It might not be realistic for a developer to ever have direct access to a secret manager.
Preserving identities - Depending on the implementation, it's possible that the actor which accesses the secret varies. Sometimes it's the k8s cluster, sometimes it's the individual pod. They both had trade-offs.
Why might you not want this abstraction? Well, it adds additional security concerns. Exposing secrets via environment variables or via the filesystem makes you subject to a generic series of supply chain attacks. Using a secret manager client library or API directly doesn't entirely prevent this, but it forces a more targeted attack (e.g. core dump) instead of a more generic path traversal or env-dump-to-pastebin attack.
Whenever an S3 artifact is used, the following declaration is needed:
s3:
endpoint: s3.amazonaws.com
bucket: "{{workflow.parameters.input-s3-bucket}}"
key: "{{workflow.parameters.input-s3-path}}/scripts/{{inputs.parameters.type}}.xml"
accessKeySecret:
name: s3-access-user-creds
key: accessKeySecret
secretKeySecret:
name: s3-access-user-creds
key: secretKeySecret
It would be helpful if this could be abstracted to something like:
custom-s3:
bucket: "{{workflow.parameters.input-s3-bucket}}"
key: "{{workflow.parameters.input-s3-path}}/scripts/{{inputs.parameters.type}}.xml"
Is there a way to make this kind of custom definition in Argo to reduce boilerplate?
For a given Argo installation, you can set a default artifact repository in the workflow controller's configmap. This will allow you to only specify the key (assuming you set everything else in the default config - if not everything is defined for the default, you'll need to specify more things).
Unfortunately, that will only work if you're only using one S3 config. If you need multiple configurations, cutting down on boilerplate will be more difficult.
In response to your specific question: not exactly. You can't create a custom some-keyname (like custom-s3) as a member of the artifacts array. The exact format of the YAML is defined in Argo's Workflow Custom Resource Definition. If your Workflow YAML doesn't match that specification, it will be rejected.
However, you can use external templating tools to populate boilerplate before the YAML is installed in your cluster. I've used Helm before to do exactly that with a collection of S3 configs. At the simplest, you could use something like sed.
tl;dr - for one S3 config, use default artifact config; for multiple S3 configs, use a templating tool.
I am trying to setup a module to deploy resources in the cloud (it could be any cloud provider). I don't see the advantages of using templates (ie. the deploy manager) over direct API calls :
Creation of VM using a template :
# deployment.yaml
resources:
- type: compute.v1.instance
name: quickstart-deployment-vm
properties:
zone: us-central1-f
machineType: f1-micro
...
# bash command to deploy yaml file
gcloud deployment-manager deployments create vm-deploy --config deployment.yaml
Creation of VM using a API call :
def addInstance(http, listOfHeaders):
url = "https://www.googleapis.com/compute/v1/projects/[PROJECT_ID]/zones/[ZONE]/instances"
body = {
"name": "quickstart-deployment-vm",
"zone": " us-central1-f",
"machineType": "f1-micro",
...
}]
bodyContentURLEncoded = urllib.urlencode(bodyContent)
http.request(uri=url, method="POST", body=body)
Can someone explain to me what benefits I get using templates?
readability\easy of use\authentication handled for you\no need to be a coder\etc. There can be many advantages, it really depends on how you look at it. It depends on your background\tools you use.
It might be more beneficial to use python all the way for you specifically.
It's easier to use templates and you get a lot of builtin functionality such as running a validation on your template to scan for possible security vulnerabilities and similar. You can also easily delete your infra using the same template as you create it. FWIW, I've gone all the way with templates and do as much as I can with templates and in smaller units. It makes it easy to move out a part of the infra or duplicate it to another project, using a pipeline in GitLab to deploy it for example.
The reason to use templates over API calls is that templates can be used in use cases where a deterministic outcome is required.
Both Template and API call has its own benefits. There is always a tradeoff between the two options. If you want more flexibility in the deployment, then the API call suits you better. On the other hand, if the security and complete revision is your priority, then Template should be your choice. Details can be found in this online documentation.
When using a template, orchestration of the deployment is handled by the platform. When using API calls (or other imperative approaches) you need to handle orchestration.
I want to convert the existing terraform templates(hcl) to aws cloudformation templates(json/yaml).
I basically want to find security issues with these templates through CFN_NAG.
An approach that I have already tried was converting HCL to JSON and then passing the template to CFN_NAG but I received a failure since both the templates have different structure.
Can anyone please provide any suggestions here?
A rather convoluted way of achieving this is to use Terraform to stand-up actual AWS environments, and then to use AWS’s CloudFormer to extract CloudFormation templates (JSON or YAML) from what Terraform has built. At which point you can use cfn-nag.
CloudFormer has some limitations, in that not all AWS resources are currently supported (RDS Security Groups for example) , but it will get you all the basic AWS resources.
Don't forget to remove all the environments, including CloudFormer's, to minimise the cost.
You want to use static code analysis to find security issues in your Terraform setup.
Trying to converting Terraform to CloudFormation to later use cfn-nag is one way. However, there exist tools now that directly operate on the Terraform setup.
I would recommend to take a look at terrascan. It is built on terraform_validate.
https://github.com/bridgecrewio/checkov/ runs security scanning for both terraform and cloudformation