I have a Prometheus monitoring running on Kubernetes cluster. I want to receive SMS notification when my alerts firing.
How should i set my number for receive SMS in Alertmanager?
Couple options coming from official docs: https://prometheus.io/docs/alerting/configuration/
Option 1. If you have PagerDuty / VictorOps subscription use https://prometheus.io/docs/alerting/configuration/#pagerduty_config receiver, and setup SMS rule inside the service.
Option 2. Use a webhook receiver https://prometheus.io/docs/alerting/configuration/#webhook_config
Set it to send notification to AWS SNS https://docs.aws.amazon.com/sns/latest/dg/sns-http-https-endpoint-as-subscriber.html, then use AWS SNS to send an SMS. Or use any other webhook based SMS sender.
The best and most simple option is just to use the sns_configs in alert-manager, This is my configuration on the prometheus-operator helm-chart in the alert-manager section, Just create an SNS topic give the pod/node permissions to that topic and use the configuration below in the values.yaml file:
alertmanager:
enabled: true
config:
inhibit_rules:
- source_matchers:
- 'severity = critical'
target_matchers:
- 'severity =~ warning|info'
equal:
- 'namespace'
- 'alertname'
- source_matchers:
- 'severity = warning'
target_matchers:
- 'severity = info'
equal:
- 'namespace'
- 'alertname'
- source_matchers:
- 'alertname = InfoInhibitor'
target_matchers:
- 'severity = info'
equal:
- 'namespace'
global:
resolve_timeout: 5m
smtp_require_tls: true
smtp_smarthost: 'email-smtp.eu-central-1.amazonaws.com:587'
smtp_from: 'alertmanager#bubu.energy'
smtp_auth_username: 'ZZZZ'
smtp_auth_password: 'ZZZZ'
slack_api_url: 'https://hooks.slack.com/services/T02GV75J1HP/B03QN9JL0J0/lXXCX'
route:
receiver: 'weev-sns'
group_by: ['alertname', 'service']
routes:
- receiver: 'weev-sns'
match:
alertname: 'Watchdog'
group_wait: '120s'
group_interval: '1m'
repeat_interval: '1m'
receivers:
- name: 'weev-sns'
sns_configs:
- sigv4:
region: 'eu-central-1'
attributes:
key: severity
value: "critical"
topic_arn: 'arn:aws:sns:eu-central-1:104553556165:alertmanagertopic'
send_resolved: false
Related
How can we pass the captured Key-Value Data (Log filter) into the mail Template,
For example, my current template looks like this
<html>
<head>
<title>create Heap dump</title>
</head>
<body>
<p>
Hi,<br><br>
${option.Description} <br>
${logoutput.data}<br><br>
Regards,<br>
Game World</p>
</body>
</html>
Currently i am not able to pass any captured value like ${data.value}. Is there anything i am missing ?
The easiest way is to export that data value variable to a global one and then use it in your notifications.
The first step print some text, with a filter, capture the data value and it's s stored in a ${data.MYDATA}.
The second step takes that data variable and creates a global one using the "Global Variable" Step.
You can use that global variable in any notification as ${export.myvariable}.
Take a look at this job definition example:
- defaultTab: nodes
description: ''
executionEnabled: true
id: ea07f41a-71b4-4ed9-91fb-6113de996e48
loglevel: INFO
name: TestJob
nodeFilterEditable: false
notification:
onsuccess:
plugin:
configuration:
authentication: None
body: ${export.myglobal}
contentType: application/json
method: POST
remoteUrl: https://any/webhook/url
timeout: '30000'
type: HttpNotification
notifyAvgDurationThreshold: null
plugins:
ExecutionLifecycle: {}
scheduleEnabled: true
schedules: []
sequence:
commands:
- exec: env
plugins:
LogFilter:
- config:
invalidKeyPattern: \s|\$|\{|\}|\\
logData: 'true'
regex: ^(USER)\s*=\s*(.+)$
type: key-value-data
- configuration:
export: myglobal
group: export
value: ${data.USER*}
nodeStep: false
type: export-var
- exec: echo ${export.myglobal}
keepgoing: false
strategy: node-first
uuid: ea07f41a-71b4-4ed9-91fb-6113de996e48
Using the HTTP notification (in the body section) you can see the value (same for your case using email notification).
I'm trying to follow AWS CloudFormation best practices, and set up lots of little stacks that nest under a parent stack. Mostly, this is working fine; however, in an intermediate stack I've now exceeded the max 60 parameter count. This will only get worse when I go the next level up and aggregate these intermediate stacks into a master parent stack.
The CloudFormation documentation says, "To specify more parameters, you can use mappings or lists in order to assign multiple values to a single parameter." In fact, that exact quotation has been given at least twice here in response to the 60 parameter limit question. However, how does one go about doing that? I can see how the mapping capability could work when you're selecting from a bunch of pre-defined sets of values (e.g., select from these AMI IDs, or these subnets), but don't see how it could work when the parameters are strings or numbers or other user-entered data; and I can't see an example at all of how lists do it.
Does anyone have an example or two of how they've used mappings and or lists to consolidate multiple user-entered parameters into a single parameter?
Edit, per a request in comments, I'm adding more details on what my templates look like. Here is the ParameterGroups section of the parent stack in question. It rolls up nested stacks for "mail storage" (which creates EBS volumes), "mail" which launches a primary and backup MX server and configures Postfix, Dovecot, etc..., and "Roundcube" which launches a autoscaling group of web servers running roundcube. All of this sits on top of another nested stack, "infrastructure", which deploys VPC, Subnets, ELB, NAT, RDS, security groups, and phpmyadmin.
AWS::CloudFormation::Interface:
ParameterGroups:
-
Label:
default: Mandatory Parameters
Parameters:
- Vpc
- NumberOfAZs
- AvailabilityZones
- AMIID
- WantRoundCube
-
Label:
default: Recommended Parameters
Parameters:
- AdminUser
- AdminPubKey
-
Label:
default: Mail Storage Parameters
Parameters:
- SpoolSize
- LogSize
- CreateSwap
- SwwapSize
- EncryptedBoolean
- Cmk
- SpoolDelPolicy
- LogDelPolicy
- MasterSpoolSnapshot
- MasterLogSnapshot
- BackupSpoolSnapshot
- BackupLogSnapshot
-
Label:
default: Primary MX EC2 Instance Parameters
Parameters:
- PrimaryServerName
- VirtualUID
- EC2KeyName
- MailInstanceType
- MailSecurityGroup
- StorageStackName
- PrimarySubnet
- InsertTestData
-
Label:
default: Backup MX EC2 Instance Parameters
Parameters:
- BackupServerName
- BackupMailInstanceType
- BackupSubnet
-
Label:
default: Database Parameters
Parameters:
- DatabaseStackName
- MailDBName
- MailDBUser
- MailDBPassword
- MailDBBackup
-
Label:
default: Postfix Primary Parameters
Parameters:
- Origin
- Banner
- RelayHost
- RootMailRecipient
- InetProtocols
- EnableSpfChecks
- EnableOpenDkim
- OpenDkimDomains
-
Label:
default: SSL Parameters
Parameters:
- CertSource
- AccountEmail
- DNSZone
- DNSSleep
- CertFileDir
-
Label:
default: Dovecot Parameters
Parameters:
- MaxUserConnections
-
Label:
default: Amavisd, SpamAssassin, ClamAV
Parameters:
- SAFinalDest
- SAKillLevelDeflt
- SetMyHost
-
Label:
default: Roundcube
Parameters:
- AllowPasswdChange
- AlternativeIAMRole
- AlternativeInitializationScript
- DnsName
- EnableDebug
- EnvironmentVariables
- Hash
- PublicAlbListenerArn
- RoundcubeDBName
- RoundcubeDBUser
- RoundcubeDBPass
- RoundcubeInstanceType
- RoundcubeSecurityGroup
- Skin
-
Label:
default: AWS S3 Configuration
Parameters:
- MirovoyCFAssetsBucket
- MailS3KeyPrefix
- RoundcubeS3KeyPrefix
- SkelFileDir
- Label:
default: Testing Configuration
Parameters:
- UnknownLocalErrorCode
- XandrosPass
- VivitaPass
- ExternalTestEmail
- SSLDebug
- ADLogLevel
- DisableVirusChecks
- DisableSpamChecks
- DisableServices
-
Label:
default: Postfix Optional Parameters
Parameters:
- DelayWarningTime
- MaxQueueLifetime
- MinBackoffTime
- MaxBackoffTime
- HeloTimeout
- RecipientLimit
- SoftErrLimit
- HardErrLimit
- SpfTimeLimit
I'm happy to post more of the template as necessary, but it's obviously already getting quite long :)
Thanks!
I am having issues deploying a lambda with a handler in a nested directory using sam.
I perform the following steps:
package:
sam package --template template.yaml --output-template-file packaged.yaml --s3-bucket
Creates a packaged.yaml that I use in the next step.
deploy:
aws cloudformation deploy --template-file /Users/localuser/Do/learn-sam/dynamo-stream-lambda/packaged.yaml --stack-name barkingstack
ERROR
Failed to create the changeset: Waiter ChangeSetCreateComplete failed: Waiter encountered a terminal failure state Status: FAILED. Reason: Transform AWS::Serverless-2016-10-31 failed with: Invalid Serverless Application Specification document. Number of errors found: 1. Resource with id [PublishNewBark] is invalid. Missing required property 'Handler'.
Cloudformation/SAM Template
AWSTemplateFormatVersion: '2010-09-09'
Transform: 'AWS::Serverless-2016-10-31'
Globals:
Function:
Runtime: nodejs8.10
Timeout: 300
Resources:
PublishNewBark:
Type: AWS::Serverless::Function
FunctionName: publishNewBark
CodeUri: .
Handler: src/index.handler
Role: "<ROLE_ARN>"
Description: Reads from the DynamoDB Stream and publishes to an SNS topic
Events:
- ReceiveBark:
Type: DynamoDB
Stream: !GetAtt BarkTable.StreamArn
StartingPosition: TRIM_HORIZON
BatchSize: 1
BarkTable:
Type: AWS::DynamoDB::Table
Properties:
TableName: BarkTable
KeySchema:
- KeyType: HASH
AttributeName: id
AttributeDefinitions:
- AttributeName: id
AttributeType: S
StreamSpecification:
StreamViewType: NEW_AND_OLD_IMAGES
ProvisionedThroughput:
WriteCapacityUnits: 5
ReadCapacityUnits: 5
WooferTopic:
Type: AWS::SNS::Topic
Properties:
DisplayName: wooferTopic
TopicName: wooferTopic
Subscription:
- Endpoint: <my_email>
Protocol: email
DIRECTORY STRUCTURE
root_directory/
events/ (for sample events)
policies/ (for IAM Role to be created for the lambda using CLI)
src/index.js
package.json
node_modules
template.yaml
HANDLER CODE
async function handler (event, context) {
console.log(JSON.stringify(event, null, 2))
return {}
}
module.exports = {handler}
I believe you have to put everything except the resource type under "Properties".
Your function declaration should be:
PublishNewBark:
Type: AWS::Serverless::Function
Properties:
FunctionName: publishNewBark
CodeUri: .
Handler: src/index.handler
Role: "<ROLE_ARN>"
Description: Reads from the DynamoDB Stream and publishes to an SNS topic
Events:
- ReceiveBark:
Type: DynamoDB
Stream: !GetAtt BarkTable.StreamArn
StartingPosition: TRIM_HORIZON
BatchSize: 1
I have a little doubt about "Mappings section" of the aws cloudformation syntax:
Example:
...
Mappings:
accounts:
56565d644801:true
986958470041:true
090960219037:true
05166767667:false
functions:
MyFunction:
handler: src/MyFunction/func.lambda_handler
role: MyRole
events:
- schedule:
rate: rate(12 hours)
enabled: Fn::FindInMap
- accounts
- Ref "AWS::AccountId"
...
Could the Mappings section be included in a serverless.yml file ?
I meant, eventhough it is a valid cloudformation syntax, would it possible include it in the serverless.yml, so that later we can implement it (serverless | sls deploy ...)?
thanks,
You might be able to use:
functions:
# ...
resources:
Mappings:
accounts:
56565d644801:true
986958470041:true
090960219037:true
05166767667:false
Just another way to work with mapping is through stage params.
https://www.serverless.com/framework/docs/guides/parameters
params:
stage1:
schedule:true
stage2:
schedule:false
functions:
MyFunction:
handler: src/MyFunction/func.lambda_handler
role: MyRole
events:
- schedule:
rate: rate(12 hours)
enabled: ${param:schedule}
Then call adding the stage arg (default is dev)
serverless deploy --stage stage1
I have nodejs code running inside a pod. From inside the pod I want to find the zone of the node where this pod is running. What is the best way do do that? Do I need extra permissions?
I have not been able to find a library but I post the code that does it below. The getContent function was slightly adapted from this post This code should work inside a GKE pod or and GCE host.
Use it as following:
const gcp = require('./gcp.js')
gcp.zone().then(z => console.log('Zone is: ' + z))
Module: gcp.js
const getContent = function(lib, options) {
// return new pending promise
return new Promise((resolve, reject) => {
// select http or https module, depending on reqested url
const request = lib.get(options, (response) => {
// handle http errors
if (response.statusCode < 200 || response.statusCode > 299) {
reject(new Error('Failed to load page, status code: ' + response.statusCode));
}
// temporary data holder
const body = [];
// on every content chunk, push it to the data array
response.on('data', (chunk) => body.push(chunk));
// we are done, resolve promise with those joined chunks
response.on('end', () => resolve(body.join('')));
});
// handle connection errors of the request
request.on('error', (err) => reject(err))
})
};
exports.zone = () => {
return getContent(
require('http'),
{
hostname: 'metadata.google.internal',
path: '/computeMetadata/v1/instance/zone',
headers: {
'Metadata-Flavor': 'Google'
},
method: 'GET'
})
}
You can use failure-domain.beta.kubernetes.io/region and failure-domain.beta.kubernetes.io/zone labels of the pod to getting its region and AZ.
But, please keep in mind, that:
Only GCE and AWS are currently supported automatically (though it is easy to add similar support for other clouds or even bare metal, by simply arranging for the appropriate labels to be added to nodes and volumes).
To get access to labels, you can use DownwardAPI for attaching a Volume with your current labels and annotations of the pod. You don't need any extra permissions for use it, just mount them as a volume.
Here is an example from a documentation:
apiVersion: v1
kind: Pod
metadata:
name: kubernetes-downwardapi-volume-example
labels:
zone: us-est-coast
cluster: test-cluster1
rack: rack-22
annotations:
build: two
builder: john-doe
spec:
containers:
- name: client-container
image: k8s.gcr.io/busybox
command: ["sh", "-c"]
args:
- while true; do
if [[ -e /etc/podinfo/labels ]]; then
echo -en '\n\n'; cat /etc/podinfo/labels; fi;
if [[ -e /etc/podinfo/annotations ]]; then
echo -en '\n\n'; cat /etc/podinfo/annotations; fi;
sleep 5;
done;
volumeMounts:
- name: podinfo
mountPath: /etc/podinfo
readOnly: false
volumes:
- name: podinfo
downwardAPI:
items:
- path: "labels"
fieldRef:
fieldPath: metadata.labels
- path: "annotations"
fieldRef:
fieldPath: metadata.annotations
When you have a mounted volume with labels, you can read a file /etc/labels which will contain information about AZ and Region as a Key-Pairs, like this:
failure-domain.beta.kubernetes.io/region=us-east-1
failure-domain.beta.kubernetes.io/zone=us-east-1c