Alternative to long lines - sh

Context: I have a script in that long lines are being striped.
In bash, I could do this
CURL=/usr/bin/curl
declare -a ARGS=(
--silent
--location
--output /dev/null
--write-out "%{http_code}"
--request GET
--max-time 30
--retry 3
"https://httpbin.org/status/200"
)
http_code=$("$CURL" "${ARGS[#]}")
However, ash does not have arrays. Is there another alternative to avoid long lines like I can do on bash in ash or in sh?


Long lines can be split with backslashes:
CURL=/usr/bin/curl
curlWithArgs(){
"$CURL" \
--silent \
--location \
--output /dev/null \
--write-out "%{http_code}" \
--request GET \
--max-time 30 \
--retry 3 \
"https://httpbin.org/status/200" \
;
}
http_code=$(curlWithArgs)

Related

Datadog keeps creating monitors when dynamic source id changes

I have a Datadog monitor generated by terraform.
The main query is as follows:
sum(last_1m):avg:app.application.health{application.health:healthy,cluster_name:${local.eks_cluster_name},!source:api-service-full} by {source}.as_count() < 60"
The issue is that after a system restart the {source} container changes it's name.
For example from app-tier-1-1abc-agent
to app-tier-1-def2-agent
The Datadog instead of updating, or removing the old monitors just creates new ones and leaves the old monitors in Alarm and N/A.
Is there anyway to improve this? All ideas appreciated, thanks!

Solved this by sending API calls to edit each monitor query on shutdown and startup.
I made a very clunky bash script, because I couldn't find a way to store -data for the curl in a variable in bash, but if using other scripting languages this could have been done in much less code, example is for 2 monitors
monitor_id=$(curl -L -X GET "https://api.datadoghq.com/api/v1/monitor" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DATADOG_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DATADOG_APP_KEY}" \
--data-raw '{"tags":["application_id:'$APP_ID_LOWERCASE'"]}'| jq -r ' .[] | select((.name |endswith("XXXXXXX Heartbeats") or endswith("XXXXX HeartBeat monitoring")) and (.tags[]=="application_id:'$APP_ID_LOWERCASE'")) | .id')
# curl gets cluster name used in queries
CLUSTER_NAME=$(curl -L -X GET "https://api.datadoghq.com/api/v1/monitor" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DATADOG_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DATADOG_APP_KEY}" \
--data-raw '{"tags":["application_id:'$APP_ID_LOWERCASE'"]}'| jq -r ' .[] | select((.name |endswith("XXXX HeartBeat monitoring")) and (.tags[]=="application_id:'$APP_ID_LOWERCASE'")) | .query' | awk -F',cluster_name:|,' '{print $2}')
# For each monitor id edit monitor query
while IFS= read -r monitors
do
# curl gets monitor name
monitor_name=$(curl -L -X GET "https://api.datadoghq.com/api/v1/monitor/"$monitors"" \
-H "Content-Type: application/json" \
-H "Accept: application/json" \
-H "DD-API-KEY: ${DATADOG_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DATADOG_APP_KEY}" | jq -r .name)
# checking which monitor query to send based on name, the queries are hardcoded because I couldn't find a way to set the query as a variable in bash
if [[ $monitor_name == *"XXXXXX HeartBeat monitoring"* ]]; then
shutdown_query=$(curl -L -X PUT "https://api.datadoghq.com/api/v1/monitor/"$monitors"" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DATADOG_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DATADOG_APP_KEY}" \
--data-raw '{"query":"sum(last_1m):avg:health{application.health:healthy,cluster_name:'${CLUSTER_NAME}',!source:XXXXXX-full-1,source:DUMMY-VALUE-TO-RESEt-QUERY} by {source}.as_count() < 60"}')
elif
[[ $monitor_name == *"XXXXXX Instance Heartbeats"* ]]; then
shutdown_query=$(curl -L -X PUT "https://api.datadoghq.com/api/v1/monitor/"$monitors"" \
-H "Accept: application/json" \
-H "Content-Type: application/json" \
-H "DD-API-KEY: ${DATADOG_API_KEY}" \
-H "DD-APPLICATION-KEY: ${DATADOG_APP_KEY}" \
--data-raw '{"query":"sum(last_1m):avg:heartbeat{cluster_name:'${CLUSTER_NAME}',source:DUMMY-VALUE-TO-RESEt-QUERY} by {source}.as_count() < 3"}')
fi
done <<< "$monitor_id"
Then just remove the dummy query value on startup and it will pick up your new monitors while forgetting the non existing ones

reject-regex in wget

I'm trying to mirror a section of intranet TWiki for offline usage as follows:
wget \
--user=twiki \
--password=******** \
--recursive \
-l 2 \
--adjust-extension \
--page-requisites \
--convert-links \
--reject-regex '\?rev=' \
--reject-regex '/twiki/rdiff/' \
--reject-regex '/twiki/attach/' \
--reject-regex '/twiki/edit/' \
--reject-regex '/twiki/oops/' \
--reject-regex '\?raw=on' \
--reject-regex '\?cover=print' \
http://twiki/cgi-bin/twiki/view/SectionToMirror/
For some reason all --reject-regex are ignored. The content I want to reject appears in the copy. Running the above command without any --reject-regex rules renders the same results.
What am I doing wrong?

Create protocol mapper in Keycloak using kcadm.sh

From Add protocol-mapper to keycloak using kcadm.sh
Has anyone figured this out yet? I tried it the way Oscar suggested and it still does not work.
The lines that are not commented work perfectly.
The lines that are commented do not work. I get an error that says "./clientmapper.sh: 59 (or whatever line number that I have uncommented): -s: not found"
sudo docker exec $keycontainer /opt/jboss/keycloak/bin/kcadm.sh create \
clients/$cid/protocol-mappers/models \
-r myrealm \
-s name=roles \
-s protocol=openid-connect \
-s protocolMapper=oidc-usermodel-attribute-mapper
#-s 'config."id.token.claim"=true' \
#-s claim.name=roles \
#-s jsonType.label=String \
#-s multivalued=true \
#-s userinfo.token.claim=true \
#-s access.token.claim=true

I made this work by formatting as Oscar suggested and using -i after the docker exec command. It works perfectly now.
sudo docker exec -i $keycontainer /opt/jboss/keycloak/bin/kcadm.sh create \
clients/$cid/protocol-mappers/models \
-r testrealm \
-s name=testmap \
-s protocol=openid-connect \
-s protocolMapper=oidc-usermodel-realm-role-mapper \
-s 'config."id.token.claim"=true' \
-s 'config."claim.name"=testmap' \
-s 'config."jsonType.label"=String' \
-s 'config."multivalued"=true' \
-s 'config."userinfo.token.claim"=true' \
-s 'config."access.token.claim"=true'

Cloudflare DDNS repeat sh script using API v4, with multiple A record in a single sh script, but fail

I am trying to use Cloudflare API v4 to setup DDNS on my server. But I am new in scripting .sh file. I hope to update multiple DNS records in a single .sh file.
I got a script from the internet (script1.sh):
NEW_IP=`curl -s http://ipv4.icanhazip.com`
CURRENT_IP=`cat /Users/foo/Desktop/cloudflare/current_ip.txt`
if [ "$NEW_IP" = "$CURRENT_IP" ]
then
echo "No Change in IP Adddress"
else
curl -X PUT "https://api.cloudflare.com/client/v4/zones/{zone_id}/dns_records/{dns_record_id}" \
-H "X-Auth-Email: {my_email}" \
-H "X-Auth-Key: {global_api_key}" \
-H "Content-Type: application/json" \
--data '{"type":"A","name":"{domain_name}","content":"'$NEW_IP'","ttl":1,"proxied":true}'
echo $NEW_IP > /Users/foo/Desktop/cloudflare/current_ip.txt
fi
The above script is work fine for single DNS record update instead of multiple record update like below (script2.sh):
NEW_IP=`curl -s http://ipv4.icanhazip.com`
CURRENT_IP=`cat /Users/foo/Desktop/cloudflare/current_ip.txt`
if [ "$NEW_IP" = "$CURRENT_IP" ]
then
echo "No Change in IP Adddress"
else
#domain-one
curl -X PUT "https://api.cloudflare.com/client/v4/zones/{zone_id_for_domain_one}/dns_records/{dns_record_id_for_domain_one_record_one}" \
-H "X-Auth-Email: {my_email}" \
-H "X-Auth-Key: {global_api_key}" \
-H "Content-Type: application/json" \
--data '{"type":"A","name":"domain-one.com","content":"'$NEW_IP'","ttl":1,"proxied":true}'
curl -X PUT "https://api.cloudflare.com/client/v4/zones/{zone_id_for_domain_one}/dns_records/{dns_record_id_for_domain_one_record_two}" \
-H "X-Auth-Email: {my_email}" \
-H "X-Auth-Key: {global_api_key}" \
-H "Content-Type: application/json" \
--data '{"type":"A","name":"subdomain.domain-one.com","content":"'$NEW_IP'","ttl":1,"proxied":true}'
#domain-two
curl -X PUT "https://api.cloudflare.com/client/v4/zones/{zone_id_for_domain_two}/dns_records/{dns_record_id_for_domain_two_record}" \
-H "X-Auth-Email: {my_email}" \
-H "X-Auth-Key: {global_api_key}" \
-H "Content-Type: application/json" \
--data '{"type":"A","name":"domain-two.com","content":"'$NEW_IP'","ttl":1,"proxied":true}'
echo $NEW_IP > /Users/foo/Desktop/cloudflare/current_ip.txt
fi
Can you please help to explain and solve the problem? Please tell me what's wrong in the script. Thanks!
[edit] I run it once by sh /Users/foo/Desktop/script-name.sh, for the first example (script1.sh), it is ok; for second example (script2.sh), return -bash: fork: Resource temporarily unavailable. As I use automatic run script like cron, it is same result.

Every curl request done, needed to add & or && to continue the following function.
More explanation here.
NEW_IP=`curl -s http://ipv4.icanhazip.com`
CURRENT_IP=`cat /Users/foo/Desktop/cloudflare/current_ip.txt`
if [ "$NEW_IP" = "$CURRENT_IP" ]
then
echo "No Change in IP Adddress"
else
#domain-one
curl -X PUT "https://api.cloudflare.com/client/v4/zones/{zone_id_for_domain_one}/dns_records/{dns_record_id_for_domain_one_record_one}" \
-H "X-Auth-Email: {my_email}" \
-H "X-Auth-Key: {global_api_key}" \
-H "Content-Type: application/json" \
--data '{"type":"A","name":"domain-one.com","content":"'$NEW_IP'","ttl":1,"proxied":true}' &
curl -X PUT "https://api.cloudflare.com/client/v4/zones/{zone_id_for_domain_one}/dns_records/{dns_record_id_for_domain_one_record_two}" \
-H "X-Auth-Email: {my_email}" \
-H "X-Auth-Key: {global_api_key}" \
-H "Content-Type: application/json" \
--data '{"type":"A","name":"subdomain.domain-one.com","content":"'$NEW_IP'","ttl":1,"proxied":true}' &
#domain-two
curl -X PUT "https://api.cloudflare.com/client/v4/zones/{zone_id_for_domain_two}/dns_records/{dns_record_id_for_domain_two_record}" \
-H "X-Auth-Email: {my_email}" \
-H "X-Auth-Key: {global_api_key}" \
-H "Content-Type: application/json" \
--data '{"type":"A","name":"domain-two.com","content":"'$NEW_IP'","ttl":1,"proxied":true}'
echo $NEW_IP > /Users/foo/Desktop/cloudflare/current_ip.txt
fi

After going through many posts on this topic I've ended up taking parts from a number posts and settled for the below.
The below will loop through the entries in the array "dnsrecords" and update each with the machines current external address.
Have had a little more time to look into this, I wanted it to run it on my NGINX server as a cronjob and update multiple records, some to be proxied and others not.
Still want to make some of the below into functions, but for now it is working as I need it to.
And it also logs onto a "log.log" file on every run.
#!/usr/bin/bash
## Cloudflare authentication details
## Keep these private
cloudflare_auth_email=Your_Email
cloudflare_auth_key="Your_API_Key"
zoneid="Your_Zone_ID"
## Cloudflare Proxied DNS Records as Array
dnsrecords_proxied=(
"domain.com"
"www.domain.com"
"sub1.domain.com"
"sub2.domain.com"
"sub3.domain.com"
"sub3.domain.com"
)
## Cloudflare Non-Proxied DNS Records as Array
dnsrecords_not_proxied=(
"vpn01.domain.com"
"vpn02.domain.com"
)
## Files to log to (replace "path/to/" with script path)
log=path/to/log.log
log_ip=path/to/previous_ip
## Getting Date/Time
dt=$(date '+%d/%m/%Y %H:%M:%S')
## Get old IP from file
old_ip=$(cat $log_ip)
## Get the current external IP address
ip=$(curl -s -X GET https://api.ipify.org)
#echo "Current IP is $ip"
## Checking if IP changed since last update
if [ $ip = $old_ip ]; then
echo -en "$dt - Previous IP:$old_ip\n$dt - Current IP:$ip\n$dt - No Changes Required....\n" >> $log
echo "$(tail -n 1000 $log)" > $log
exit
## Exit if IP has not changed
else
## If the IP changed, not match the one on file "previous_ip"
echo -en "$dt - Previous IP:$old_ip\n$dt - Current IP:$ip\n$dt - Starting Updates....\n" >> $log
## Processing Proxied DNS Records
for dnsrecord in "${dnsrecords_proxied[#]}"
do
## For each DNS Record in Array "dnsrecords"
# Getting the DNS Record ID
dnsrecordid=$(curl -s -X GET "https://api.cloudflare.com/client/v4/zones/$zoneid/dns_records?type=A&name=$dnsrecord" \
-H "X-Auth-Email: $cloudflare_auth_email" \
-H "Authorization: Bearer $cloudflare_auth_key" \
-H "Content-Type: application/json" | jq -r '{"result"}[] | .[0] | .id') &&
# Updating the DNS Record
curl -s -X PUT "https://api.cloudflare.com/client/v4/zones/$zoneid/dns_records/$dnsrecordid" \
-H "X-Auth-Email: $cloudflare_auth_email" \
-H "Authorization: Bearer $cloudflare_auth_key" \
-H "Content-Type: application/json" \
--data "{\"type\":\"A\",\"name\":\"$dnsrecord\",\"content\":\"$ip\",\"ttl\":1,\"proxied\":true}" | jq
echo -en "$dt - Updated - $dnsrecord \n" >> $log
done
## Processing Non Proxied DNS Records
for dnsrecord in "${dnsrecords_not_proxied[#]}"
do
## For each DNS Record in Array "dnsrecords"
# Getting the DNS Record ID
dnsrecordid=$(curl -s -X GET "https://api.cloudflare.com/client/v4/zones/$zoneid/dns_records?type=A&name=$dnsrecord" \
-H "X-Auth-Email: $cloudflare_auth_email" \
-H "Authorization: Bearer $cloudflare_auth_key" \
-H "Content-Type: application/json" | jq -r '{"result"}[] | .[0] | .id') &&
# Updating the DNS Record
curl -s -X PUT "https://api.cloudflare.com/client/v4/zones/$zoneid/dns_records/$dnsrecordid" \
-H "X-Auth-Email: $cloudflare_auth_email" \
-H "Authorization: Bearer $cloudflare_auth_key" \
-H "Content-Type: application/json" \
--data "{\"type\":\"A\",\"name\":\"$dnsrecord\",\"content\":\"$ip\",\"ttl\":1,\"proxied\":false}" | jq
echo -en "$dt - Updated - $dnsrecord \n" >> $log
done
echo $ip > $log_ip
echo -en "$dt - Updates Completed.... \n" >> $log
echo "$(tail -n 1000 $log)" > $log
fi

Add protocol-mapper to keycloak using kcadm.sh

I have been trying to setup my full test system in keycloak using the kcadmin cli, but I have some problems creating protocol mappers:
HTTP error - 400 Bad Request
I have been trying to implement a request using:
http://www.keycloak.org/docs-api/3.3/rest-api/index.html
http://blog.keycloak.org/2017/01/administer-keycloak-server-from-shell.html
Am I missing something in the request:
/opt/jboss/keycloak/bin/kcadm.sh create \
clients/7e8ef93b-0d0f-487d-84a5-5cfaee7ddf13/protocol-mappers/models \
-r $test_realm \
-s config.user.attribute=tenants \
-s config.claim.name=tenants \
-s config.jsonType.label=String \
-s config.id.token.claim=true \
-s config.access.token.claim=true \
-s config.userinfo.token.claim=true \
-s config.multivalued=true \
-s name=tenants \
-s protocolMapper=oidc-usermodel-attribute-mapper
This works:
/opt/jboss/keycloak/bin/kcadm.sh create \
clients/7e8ef93b-0d0f-487d-84a5-5cfaee7ddf13/protocol-mappers/models \
-r $test_realm \
-s name=tenants1 \
-s protocol=openid-connect \
-s protocolMapper=oidc-usermodel-attribute-mapper

You need to specify nested config values like this in Linux:
-s 'config."id.token.claim"=true'
-s 'config."included.client.audience"=theclient'

In the failing example the following value is missing:
-s protocol=openid-connect