I have a key in KMS that I want to use for decrypting in a shell script. I have installed aws-encryption-cli (https://docs.aws.amazon.com/encryption-sdk/latest/developer-guide/crypto-cli.html).
The hello world example works fine in my development environment, where I have created a kms-key (its arn is stored in the variable $dev_key_arn).
> echo 'Hello World' | aws-encryption-cli --encrypt --master-keys key=$dev_key_arn --input - --output - --encode -S | aws-encryption-cli --decrypt --input - --output - --decode -S
Hello World
I also have a production environment, so I want to use the kms-key $prod_key_arn. I have valid AWS-credentials in a profile called prod, so I would expect
> echo 'Hello World' | aws-encryption-cli --encrypt --master-keys key=$prod_key_arn profile=prod region=eu-west-1 --input - --output - --encode -S | aws-encryption-cli --decrypt --input - --output - --decode -S
Encountered unexpected error: increase verbosity to see details.
DecryptKeyError("Unable to decrypt any data key")
But it fails with above message. If examine the encrypted message, I see that it successfully encrypts the message
> aws-encryption-cli --encrypt --master-keys key=arn:aws:kms:eu-west-1:953495156568:key/2197020f-5b3c-4d05-bffc-04cf6114e405 profile=prod region=eu-west-1 provider=aws-kms --input /tmp/prod_key --output - --encode -S
AYADePCKfZUuL<....>iza1AU=
The arn of the key is stored in the encrypted material (base64 decode shows it). But where goes the information about profile? It can not be passed in as parameter when decoding and it does not seem to be part of the encoded materail.
aws-encryption-cli author here. :)
The problem you are encountering is that in your encrypt command you set the profile to "prod", but in your decrypt command you did not set a profile, so it is using your default profile.
aws-encryption-cli --decrypt --input - --output - --decode -S
If you add in a master key definition identifying the profile you want to use, it should work.
aws-encryption-cli --decrypt --master-keys provider=aws-kms profile=prod --input - --output - --decode -S
Because we default to the aws-kms provider if you don't specify a name, just specifying the profile should also work, but I prefer to identify the provider since that makes the intention clearer.
aws-encryption-cli --decrypt --master-keys profile=prod --input - --output - --decode -S
If you run into any other issues with aws-encryption-cli, please feel free to open an issue in our GitHub repo[1].
[1] https://github.com/aws/aws-encryption-sdk-cli
Related
I'm using a github workflow to automate some actions for AWS. I haven't changed anything for a while as the script has been working nicely for me. Recently I've been getting this error: Unable to process file command 'env' successfully whenever the workflow runs. I've got no idea why this is happening. Any help or pointers would greatly appreciated. Thanks. Here's the workflow which is outputting the error:
- name: "Get AWS Resource values"
id: get_aws_resource_values
env:
SHARED_RESOURCES_ENV: ${{ github.event.inputs.shared_resources_workspace }}
run: |
BASTION_INSTANCE_ID=$(aws ec2 describe-instances \
--filters "Name=tag:env,Values=$SHARED_RESOURCES_ENV" \
--query "Reservations[*].Instances[*].InstanceId" \
--output text)
RDS_ENDPOINT=$(aws rds describe-db-instances \
--db-instance-identifier $SHARED_RESOURCES_ENV-rds \
--query "DBInstances[0].Endpoint.Address" \
--output text)
echo "rds_endpoint=$RDS_ENDPOINT" >> $GITHUB_ENV
echo "bastion_instance_id=$BASTION_INSTANCE_ID" >> $GITHUB_ENV
From the RDS endpoint query expression (Reservations[*].Instances[*].InstanceId) in your aws cli command, it seems you expect a multiline string. It could also be that before you started to receive this error the command was producing a single line string, and that changed at some point.
In GitHub actions, multiline strings for environment variables and outputs need to be created with a different syntax.
For the RDS endpoint you should set the environment variable like this:
echo "rds_endpoint<<EOF" >> $GITHUB_ENV
echo "$RDS_ENDPOINT" >> $GITHUB_ENV
echo "EOF" >> $GITHUB_ENV
I guess that the bastion instance id will not be a problem since it's a single line string.
Suppose i've got a zip file available under some URL. I need to get its hash, which should be identical to the one output by nix-prefetch-url --unpack <URL>, but without a working Nix installation. How can one do it?
Seems there is no easy way, as nix-prefetch-url adds the file to the store. More details here: https://discourse.nixos.org/t/generate-a-file-hash-similar-to-the-one-output-by-nix-prefetch-url/19907 (many thanks to prompt and thorough community member's response)
Use Docker.
Demo:
$ nix-prefetch-url --unpack https://github.com/hraban/git-hly/archive/06ff628d5f2b02d1a883c94b01d58187d117f4f3.tar.gz
path is '/nix/store/gxx1pfp19s3a39j6gl0xw197b4409cmp-06ff628d5f2b02d1a883c94b01d58187d117f4f3.tar.gz'
164gyvpdm6l6rdvn2rwjz95j1jz0w2igcbk9shy862sdx2rdw9hn
$ # Or .zip: it's the same, because of --unpack:
$ nix-prefetch-url --unpack https://github.com/hraban/git-hly/archive/06ff628d5f2b02d1a883c94b01d58187d117f4f3.zip
path is '/nix/store/1bpjlzknnmq1x3hq213r44jwag1xkaqs-06ff628d5f2b02d1a883c94b01d58187d117f4f3.zip'
164gyvpdm6l6rdvn2rwjz95j1jz0w2igcbk9shy862sdx2rdw9hn
Download to a local directory
$ cd "$(mktemp -d)"
$ curl -sSL --fail https://github.com/hraban/git-hly/archive/06ff628d5f2b02d1a883c94b01d58187d117f4f3.tar.gz | tar xz
$ cd *
And test it:
$ # Using the modern nix command:
$ nix hash path --base32 .
164gyvpdm6l6rdvn2rwjz95j1jz0w2igcbk9shy862sdx2rdw9hn
$ # Or the same, using nix-hash:
$ nix-hash --type sha256 --base32 .
164gyvpdm6l6rdvn2rwjz95j1jz0w2igcbk9shy862sdx2rdw9hn
Same in Docker:
$ docker run --rm -v "$PWD":/data nixos/nix nix --extra-experimental-features nix-command hash path --base32 /data
164gyvpdm6l6rdvn2rwjz95j1jz0w2igcbk9shy862sdx2rdw9hn
$ docker run --rm -v "$PWD":/data nixos/nix nix-hash --type sha256 --base32 /data
164gyvpdm6l6rdvn2rwjz95j1jz0w2igcbk9shy862sdx2rdw9hn
P.S.: I'm not a huge fan of nix-prefetch-url's default output (base32). The default output of nix hash path is better, if you can use it:
$ nix hash path .
sha256-FibesuhNC4M81Gku9qLg4MsgS/qSZ2F3y4aa2u72j5g=
$ # Sanity check:
$ nix-hash --type sha256 --to-base32 $(<<<"FibesuhNC4M81Gku9qLg4MsgS/qSZ2F3y4aa2u72j5g=" base64 -d | hexdump -v -e '/1 "%02x"' )
164gyvpdm6l6rdvn2rwjz95j1jz0w2igcbk9shy862sdx2rdw9hn
Context:
I have gitlab runners which are executing terraform init command which is pulling all necessary terraform modules. Recently, I started hitting github throttling issues (60 calls to github api per hour). So I am trying to reconfigure my pipeline so it uses Github user's private key.
Currently, I have the following in my pipeline but it still doesn't seem to work and private key isn't used to pull the terraform modules.
- GITHUB_SECRET=$(aws --region ${REGION} ssm get-parameters-by-path --path /github/umotifdev --with-decryption --query 'Parameters[*].{Name:Name,Value:Value}' --output json);
- PRIVATE_KEY=$(echo "${GITHUB_SECRET}" | jq -r '.[] | select(.Name == "/github/umotifdev/private_key").Value' | base64 -d);
- PUBLIC_KEY=$(echo "${GITHUB_SECRET}" | jq -r '.[] | select(.Name == "/github/umotifdev/public_key").Value' | base64 -d);
- mkdir -p ~/.ssh;
- echo "${PRIVATE_KEY}" | tr -d '\r' > ~/.ssh/id_rsa;
- chmod 700 ~/.ssh/id_rsa;
- eval $(ssh-agent -s);
- ssh-add ~/.ssh/id_rsa;
- ssh-keyscan -H 'github.com' >> ~/.ssh/known_hosts;
- ssh-keyscan github.com | sort -u - ~/.ssh/known_hosts -o ~/.ssh/known_host;
- echo -e "Host github.com\n\tStrictHostKeyChecking no\n\n" > ~/.ssh/config;
- echo ${PUBLIC_KEY} >> ~/.ssh/authorized_keys
The error I am seeing in my pipeline is something like (which is basically throttling from github):
Error: Failed to download module
Could not download module "vpc" (vpc.tf:17) source code from
"https://api.github.com/repos/terraform-aws-modules/terraform-aws-vpc/tarball/v2.21.0//*?archive=tar.gz":
bad response code: 403.
Anyone can advise how to resolve an issue where private key isn't used to pull terraform modules?
I want to access my github repositories via ssh. When I access the repository for the first time, I am asked If I want to add the github ssh server to my known_hosts file, which works fine. That request also shows me the RSA key fingerprint of that server and I can manually verify that it is the same that is provided by github here.
These are the SHA256 hashes shown in OpenSSH 6.8 and newer (in base64 format):
SHA256:nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8 (RSA)
SHA256:br9IjFspm1vxR3iA35FWE+4VTyz1hYVLIE2t1/CeyWQ (DSA)
The problem is that I want to prevent that request by adding a public key
to my known_hosts file before my first access to my git repository. This can be done by using the ssh-keyscan -t rsa www.github.com command which will give me a public key in the format required by the known_hosts file. But people mention repeatedly, that this is not safe and is vulnerable to man-in-the-middle attacks. What they do not mention is how to do it right.
So how can I use the RSA fingerprint provided on the github page to safely get the public host key of the ssh server? I am more or less looking for an option to the ssh-keyscan command that lets me add the expected rsa fingerprint and causes the command to fail if the hosts fingerprint does not match the given one.
Thank you for your time!
I would not use ssh-keyscan in that case.
Rather, I would use it and double-check the result by comparing its fingerprint with the one provided by GitHub.
And then proceed with an SSH GitHub test, to check I do get:
Hi username! You've successfully authenticated, but GitHub does not
provide shell access.
So, as recommended here, for the manual process:
ssh-keyscan github.com >> githubKey
Generate the fingerprint:
ssh-keygen -lf githubKey
Compare it with the ones provided by GitHub
Finally, copy githubKey content to your ~/.ssh/known_hosts file.
You can automate that process (still including the fingerprint step check) with wercker/step-add-to-known_hosts: it is a wercker step, but can be extrapolated as its own independent script.
- add-to-known_hosts:
hostname: github.com
fingerprint: 16:27:ac:a5:76:28:2d:36:63:1b:56:4d:eb:df:a6:48
type: rsa
But that would lack the check against help.github.com/articles/github-s-ssh-key-fingerprints: see below.
Using nmap does not help much, as explained here:
using nmap to get the SSH host key fingerprint and then comparing it to what ssh-keyscan says the fingerprint: In both cases, the fingerprint comes from the same place.
It's just as vulnerable to MITM as any other of these automated solutions.
The only secure and valid way to verify an SSH public key is over some trusted out-of-band channel. (Or set up some kind of key-signing infrastructure.)
Here, help.github.com/articles/github-s-ssh-key-fingerprints remains the "trusted out-of-band channel".
Based on VonC's answer, the script below can verify and add the key automatically. Use it like this:
$ ./add-key.sh github.com nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8
It tells you whether it successfully verified and saved the fingerprint.
For usage info, use ./add-key.sh --help
The script:
#!/usr/bin/env bash
# Settings
knownhosts="$HOME/.ssh/known_hosts"
if [ "x$1" == "x-h" ] || [ "x$1" == "x--help" ] || [ ${#1} == 0 ]; then
echo "Usage: $0 <host> <fingerprint> [<port>]"
echo "Example: $0 github.com nThbg6kXUpJWGl7E1IGOCspRomTxdCARLviKw6E5SY8"
echo "The default port is 22."
echo "The script will download the ssh keys from <host>, check if any match"
echo "the <fingerprint>, and add that one to $knownhosts."
exit 1
fi
# Argument handling
host=$1
fingerprint=$2
port=$(if [ -n "$3" ]; then echo "$3"; else echo 22; fi)
# Download the actual key (you cannot convert a fingerprint to the original key)
keys="$(ssh-keyscan -p $port $host |& grep -v ^\#)";
echo "$keys" | grep -v "^$host" # Show any errors
keys="$(echo "$keys" | grep "^$host")"; # Remove errors from the variable
if [ ${#keys} -lt 20 ]; then echo Error downloading keys; exit 2; fi
# Find which line contains the key matching this fingerprint
line=$(ssh-keygen -lf <(echo "$keys") | grep -n "$fingerprint" | cut -b 1-1)
if [ ${#line} -gt 0 ]; then # If there was a matching fingerprint (todo: shouldn't this be -ge or so?)
# Take that line
key=$(head -$line <(echo "$keys") | tail -1)
# Check if the key part (column 3) of that line is already in $knownhosts
if [ -n "$(grep "$(echo "$key" | awk '{print $3}')" $knownhosts)" ]; then
echo "Key already in $knownhosts."
exit 3
else
# Add it to known hosts
echo "$key" >> $knownhosts
# And tell the user what kind of key they just added
keytype=$(echo "$key" | awk '{print $2}')
echo Fingerprint verified and $keytype key added to $knownhosts
fi
else # If there was no matching fingerprint
echo MITM? These are the received fingerprints:
ssh-keygen -lf <(echo "$keys")
echo Generated from these received keys:
echo "$keys"
exit 1
fi
My one-liner allows for error reporting on failure:
touch ~/.ssh/known_hosts && if [ $(grep -c 'github.com ssh-rsa' ~/.ssh/known_hosts) -lt 1 ]; then KEYS=$(KEYS=$(ssh-keyscan github.com 2>&1 | grep -v '#'); ssh-keygen -lf <(echo $KEYS) || echo $KEYS); if [[ $KEYS =~ '(RSA)' ]]; then if [ $(curl -s https://help.github.com/en/github/authenticating-to-github/githubs-ssh-key-fingerprints | grep -c $(echo $KEYS | awk '{print $2}')) -gt 0 ]; then echo '[GitHub key successfully verified]' && ssh-keyscan github.com 1>~/.ssh/known_hosts; fi; else echo \"ssh-keygen -lf failed:\\n$KEYS\"; exit 1; fi; unset KEYS; fi
GitHub now offers this information in its Meta API, see About GitHub's IP addresses. The JSON output includes the public SSH keys, so assuming your HTTPS client correctly verifies the certificate chain, you can fetch the keys from there.
Below is an Ansible task that accomplishes this:
# Copyright 2022 Google LLC.
# SPDX-License-Identifier: Apache-2.0
- name: Add github.com public keys to known_hosts
ansible.builtin.known_hosts:
path: /etc/ssh/ssh_known_hosts
name: github.com
# Download the keys from the GitHub API and prepend 'github.com' to them to
# match the known_hosts format.
key: |
{% for key in (lookup('ansible.builtin.url',
'https://api.github.com/meta',
split_lines=False, validate_certs=True)
|from_json)['ssh_keys'] %}
github.com {{ key }}
{% endfor %}
Did I missed anything during this?
I am running a this command from command prompt in Windows 7 to create a certificate but it is showing a syntax error.
Command I ran:
makecert.exe "c:\1\Test.cer" -a sha1 -n "CN=AppGuid" -sr LocalMachine -ss My -sky signature -pe -len 2048
Output I got:
Usage: MakeCert [ basic|extended options] [outputCertificateFile]
Basic Options
-sk
-ss
-sr
.
.
.
Did I missed anything during this?
Well, as you can see from the format it says MakeCert [basic] [output file].
The last item in your command is not the output file, so theres one thing wrong. As for the rest, I would assume that "C:\1\test.cer" is not an option, so that is probably wrong too.