After installing datadog chart (version=3.1.3) on EKS, I need to limit some of namespaces from metric collection. I'm using containerExclude to limit the namespace scope. The values I'm using is as follows:
datadog:
site: datadoghq.eu
clusterName: eks-test
kubeStateMetricsEnabled: false
kubeStateMetricsCore:
enabled: false
containerExclude: "kube_namespace:astronomer kube_namespace:astronomer-.* kube_namespace:kube-system kube_namespace:kube-public kube_namespace:kube-node-lease"
prometheusScrape:
enabled: true
serviceEndpoints: true
additionalConfigs:
- autodiscovery:
kubernetes_annotations:
include:
prometheus.io/scrape: "true"
exclude:
prometheus.io/scrape: "false"
clusterAgent:
enabled: true
metricsProvider:
enabled: false
agents:
enabled: true
Looking at the pod environment variables, I see this is being passed to agent correctly:
DD_CONTAINER_EXCLUDE: kube_namespace:astronomer kube_namespace:astronomer-.* kube_namespace:kube-system kube_namespace:kube-public kube_namespace:kube-node-lease
However I see see scrape logs from those namespaces:
2022-09-26 13:48:38 UTC | CORE | INFO | (pkg/collector/python/datadog_agent.go:127 in LogMessage) | openmetrics:f744b75c375b067b | (base.py:60) | Scraping OpenMetrics endpoint: http://172.20.79.111:9102/metrics
2022-09-26 13:48:38 UTC | CORE | INFO | (pkg/collector/python/datadog_agent.go:127 in LogMessage) | openmetrics:c52e8d14335bb33d | (base.py:60) | Scraping OpenMetrics endpoint: http://172.20.22.119:9102/metrics
2022-09-26 13:48:39 UTC | CORE | ERROR | (pkg/collector/python/datadog_agent.go:123 in LogMessage) | openmetrics:ba505488f569ffa0 | (base.py:66) | There was an error scraping endpoint http://172.20.0.10:9153/metrics: HTTPConnectionPool(host='172.20.0.10', port=9153): Max retries exceeded with url: /metrics (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fd271df8190>, 'Connection to 172.20.0.10 timed out. (connect timeout=10.0)'))
2022-09-26 13:48:39 UTC | CORE | ERROR | (pkg/collector/worker/check_logger.go:69 in Error) | check:openmetrics | Error running check: [{"message": "There was an error scraping endpoint http://172.20.0.10:9153/metrics: HTTPConnectionPool(host='172.20.0.10', port=9153): Max retries exceeded with url: /metrics (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fd271df8190>, 'Connection to 172.20.0.10 timed out. (connect timeout=10.0)'))", "traceback": "Traceback (most recent call last):\n File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/base.py\", line 1116, in run\n self.check(instance)\n File \"/opt/datadog-agent/embedded/lib/python3.8/site-packages/datadog_checks/base/checks/openmetrics/v2/base.py\", line 67, in check\n raise_from(type(e)(\"There was an error scraping endpoint {}: {}\".format(endpoint, e)), None)\n File \"<string>\", line 3, in raise_from\nrequests.exceptions.ConnectTimeout: There was an error scraping endpoint http://172.20.0.10:9153/metrics: HTTPConnectionPool(host='172.20.0.10', port=9153): Max retries exceeded with url: /metrics (Caused by ConnectTimeoutError(<urllib3.connection.HTTPConnection object at 0x7fd271df8190>, 'Connection to 172.20.0.10 timed out. (connect timeout=10.0)'))\n"}]
2022-09-26 13:48:39 UTC | CORE | INFO | (pkg/collector/python/datadog_agent.go:127 in LogMessage) | openmetrics:8e74b74102a3722 | (base.py:60) | Scraping OpenMetrics endpoint: http://172.20.129.200:9102/metrics
2022-09-26 13:48:39 UTC | CORE | INFO | (pkg/collector/python/datadog_agent.go:127 in LogMessage) | openmetrics:e6012834c5d9bc2e | (base.py:60) | Scraping OpenMetrics endpoint: http://172.20.19.241:9127/metrics
2022-09-26 13:48:39 UTC | CORE | INFO | (pkg/collector/python/datadog_agent.go:127 in LogMessage) | openmetrics:ba98e825c73ee1b4 | (base.py:60) | Scraping OpenMetrics endpoint: http://172.20.64.136:9102/metrics
Those services belong to kube-system which is excluded. But I see that these metrics belong to Prometheus [Openmetrics]. Do I have to apply similar setting in prometheusScrape.additionalConfigs?
Related
I have a namespace already available in a k8s cluster. However, it is empty. Next, I am trying to create k8s configmap, deployment, service, and secrets in it using k8s provider terraform.
However, it is giving error as below for every object.
stdout | 11/23/2022, 5:31:06 PM | Checking state lock
stdout | 11/23/2022, 5:31:06 PM | $ terraform apply -input=false -no-color tfplan
stdout | 11/23/2022, 5:31:06 PM | [OUTPUT]
stdout | 11/23/2022, 5:31:07 PM | Acquiring state lock. This may take a few moments...
stdout | 11/23/2022, 5:31:09 PM | module.service.kubernetes_service.test_nginx: Creating...
stdout | 11/23/2022, 5:31:09 PM | module.configmap.kubernetes_config_map.nginx_translations_resourcebundle: Creating...
stdout | 11/23/2022, 5:31:09 PM | module.configmap.kubernetes_config_map.nginx_config: Creating...
stdout | 11/23/2022, 5:31:10 PM | module.deployment.kubernetes_deployment.test_nginx: Creating...
stderr | 11/23/2022, 5:31:10 PM |
stderr | 11/23/2022, 5:31:10 PM | Error: an empty namespace may not be set during creation
on configmap/configmap.tf line 7, in resource "kubernetes_config_map" "config":
7: resource "kubernetes_config_map" "nginx_ext_config" {
This is the tf file for configmap.
locals {
directory_external_config = "./configmap/config"
}
resource "kubernetes_config_map" "config" {
metadata {
name = "test-nginx"
namespace = var.TEST_NAMESPACE
}
data = {
for f in fileset(local.directory_external_config, "*") :
f => file(join("/", [local.directory_external_config, f]))
}
}
Here, I am passing the namespace from a terraform variable (namespace = var.TEST_NAMESPACE).
Please guide me on what I am missing.
My sample log looks like :
2022-09-01 23:13:05Z | error | 2022-09-02 02:13:05 - [Task] Id:120 Name:OPT_VIM_1HEAD Exception with index:18 | 18.9251137 | Exception:
ERROR connection to partner '10.19.101.17:3300' broken
2022-09-01 23:13:25Z | error | 2022-09-02 02:13:25 - [Task] Id:121 Name:OPT_VIM_1ITEM
ERROR connection to partner '10.19.101.22:3300' broken
2022-09-01 23:13:25Z | error | 2022-09-02 02:13:25 - [Task] Id:121 Name:OPT_VIM_1ITEM RunId:7 Task execution failed with error: One or more errors occurred., detail:
ERROR connection to partner '10.19.101.22:3300' broken
I want to extract the job name OPT_VIM_1HEAD or OPT_VIM_1ITEM (its dynamic) and also the timestamp after the "error" pattern : 2022-09-02 02:13:25 or 2022-09-02 02:13:05 in different variables.
I have also written the script as :
$dir = 'C:\ProgramData\AecorsoftDataIntegrator\logs\'
$StartTime = get-date
$fileList = (Get-ChildItem -Path $dir -Filter '2022-09-02.log' | Sort-Object LastWriteTime -Descending | Select-Object -First 1).fullname
$message = Get-Content $fileList | Where-Object {$_ -like ‘*error*’}
$message
$details = Select-String -LiteralPath $fileList -Pattern 'error' -Context 0,14 | Select-Object -First 1 | Select-Object Path, FileName, Pattern, Linenumber
$details[0]
But not able to retrieve the tokens mentioned above
Use regex processing via the -match operator to extract the tokens of interest from each line:
# Sample lines from the log file.
$logLines = #'
2022-09-01 23:13:05Z | error | 2022-09-02 02:13:05 - [Task] Id:120 Name:OPT_VIM_1HEAD Exception with index:18 | 18.9251137 | Exception:
ERROR connection to partner '10.19.101.17:3300' broken
2022-09-01 23:13:25Z | error | 2022-09-02 02:13:25 - [Task] Id:121 Name:OPT_VIM_1ITEM
ERROR connection to partner '10.19.101.22:3300' broken
2022-09-01 23:13:25Z | error | 2022-09-02 02:13:25 - [Task] Id:121 Name:OPT_VIM_1ITEM RunId:7 Task execution failed with error: One or more errors occurred., detail:
ERROR connection to partner '10.19.101.22:3300' broken
'# -split '\r?\n'
# Process each line...
$logLines | ForEach-Object {
# ... by matching it ($_) against a regex with capture groups - (...) -
# using the -match operator.
if ($_ -match '\| (\d{4}-.+?) - \[.+? Name:(\w+)') {
# The line matched.
# Capture groups 1 and 2 in the automatic $Matches variable contain
# the tokens of interest; assign them to variables.
$timestamp = $Matches.1
$jobName = $Matches.2
# Sample output, as an object
[PSCustomObject] #{
JobName = $jobName
Timestamp = $timestamp
}
}
}
Output:
JobName Timestamp
------- ---------
OPT_VIM_1HEAD 2022-09-02 02:13:05
OPT_VIM_1ITEM 2022-09-02 02:13:25
OPT_VIM_1ITEM 2022-09-02 02:13:25
For an explanation of the regex and the ability to experiment with it, see this regex101.com page.
I have installed apache drill and it is working fine. I am trying to fetch data from MYSQL table in postman REST APi
localhost:8047/query.json - launching properly
mysqlplugin configured successfully
From drill command line, MYSQL command working properly
apache drill> SELECT * FROM mysqlplugin.test.`table` LIMIT 10;
+----+-------+
| id | name |
+----+-------+
| 1 | anil |
| 2 | amit |
| 3 | ankur |
But when I connect REST API to localhost:8047/query.json
using body:
{
"queryType" : "SQL",
"query" : "select * from mysqlplugin.test.`table`;"
}
It gives response:
{
"errorMessage": "HTTP 405 Method Not Allowed"
}
What is the problem?
I am trying to write a code to parse a logfile within a specific date range, logfile content is below:
For eg: extracting the date (11/28 07:08:46) and parsing it.
[C79C] ComputerName:BETHGARWICK UserID:A0006 Beth Garwick Station 9 LanId: | (11/28 07:08:46) | Client is disconnected from agent.
[C79C] ComputerName:BETHGARWICK UserID: Logged out Station 0 LanId: | (11/28 07:08:51) | Client is connected to agent.
[EB7C] ComputerName:APT UserID:A0005 Kelley Zajac Station 4 LanId: | (11/28 07:12:08) | Client is disconnected from agent.
[EB7C] ComputerName:APT UserID:A0005 Kelley Zajac Station 4 LanId: | (11/28 07:12:13) | Client is connected to agent.
[EC44] ComputerName:KCUTSHALL-PC UserID:GO kcutshall Station 9900 LanId: | (11/28 07:55:08 - 11/28 07:55:18) | Average limit (300) exceeded while pinging www.google.com [74.125.224.82] 3 times
[EC44] ComputerName:KCUTSHALL-PC UserID:GO kcutshall Station 9900 LanId: | (11/28 07:55:23) | Average limit (300) exceeded while pinging www.google.com [www.google.com]
[EC44] ComputerName:KCUTSHALL-PC UserID:GO kcutshall Station 9900 LanId: | (11/28 07:55:29 - 11/28 07:55:49) | Average limit (300) exceeded while pinging www.google.com [74.125.224.50] 5x
[EC44] ComputerName:KCUTSHALL-PC UserID:GO kcutshall Station 9900 LanId: | (11/28 07:55:54 - 11/28 07:56:45) | Average limit (300) exceeded while pinging www.google.com [74.125.224.50] 11 times
[EC44] ComputerName:KCUTSHALL-PC UserID:GO kcutshall Station 9900 LanId: | (11/28 07:56:50) | Average limit (300) exceeded while pinging www.google.com [www.google.com]
Tried .net functions parseexact(), parse() but did not work.
$patter = 'mm/dd'
$culture = [Globalization.CultureInfo]::InvariantCulture
$logfiles = Get-Content -Path "C:\Users\ABC\Desktop\Temp\HTTPS\QoS_logs\test.logs"
$logfiles | foreach {
$dateasText = $_.ToString().Split("|")[1].Replace("(","").Replace(")","").Trim()
$date = [DateTime]::ParseExact($dateasText,$pattern,$null)
Exeption calling "ParseExact" with "3" argument(s): "String was not recognized as a valid DateTime."
At "C:\Users\ABC\Desktop\Temp\HTTPS\QoS_logs\test.logs:23 char:1
+ $date = [DateTime]::ParseExact($dateasText,$pattern,$null)
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [], MethodInvocationException
+ FullyQualifiedErrorId : FormatException
Use $pattern as a RegEx to grep the date in a capture group dt
## Q:\Test\2019\09\06\SO_57819852.ps1
$pattern = ' LanId: \| \((?<dt>[01][0-9]/[0-3][0-9] [0-2][0-9]:[0-5][0-9]:[0-5][0-9])'
$logfile = "C:\Users\ABC\Desktop\Temp\HTTPS\QoS_logs\test.logs" # ".\test.logs" #
foreach($Line in Get-Content $logfile) {
if ($Line -match $pattern){
$date = [DateTime]::ParseExact($Matches.dt,'MM/dd HH:mm:ss',$null)
$date
}
}
Sample output in my German locale:
Donnerstag, 28. November 2019 07:08:46
Donnerstag, 28. November 2019 07:08:51
Donnerstag, 28. November 2019 07:12:08
Donnerstag, 28. November 2019 07:12:13
Donnerstag, 28. November 2019 07:55:08
Donnerstag, 28. November 2019 07:55:23
Donnerstag, 28. November 2019 07:55:29
Donnerstag, 28. November 2019 07:55:54
Donnerstag, 28. November 2019 07:56:50
I was facing the below errors in OpenStack installation. Problem in connecting the dashboard URL.
Changed the url in openrc file by removing version like below http://100.1.201.102:5000/identity.
But still facing the same below issue. please let me know where exactly the error is being thrown .
INFO keystone.cmd.cli [req-412f0ceb-6b28-0bb1-b2scc-aae69e76ff7b - - - - -] Created domain default
INFO keystone.cmd.cli [req-412f0ceb-6b28-0bb1-b2scc-aae69e76ff7b - - - - -] Created project admin
DEBUG passlib.registry [req-412f0ceb-6b28-0bb1-b2scc-aae69e76ff7b - - - - -] registered 'sha512_crypt' handler: <class 'passlib.handlers.sha2_crypt.sha512_crypt'> register_crypt_handler /usr/lib/python2.7/site-packages/passlib/registry.py:284
INFO keystone.cmd.cli [req-412f0ceb-6b28-0bb1-b2scc-aae69e76ff7b - - - - -] Created user admin
INFO keystone.cmd.cli [req-412f0ceb-6b28-4bb1-b2cc-aae69e76ff7b - - - - -] Created role admin
INFO keystone.cmd.cli [req-412f0ceb-6b28-4bb1-b2cc-aae69e76ff7b - - - - -] Granted admin on admin to user admin.
INFO keystone.cmd.cli [req-412f0ceb-6b28-4bb1-b2cc-aae69e76ff7b - - - - -] Created region RegionOne
INFO keystone.cmd.cli [req-412f0ceb-6b28-4bb1-b2cc-aae69e76ff7b - - - - -] Created admin endpoint http://100.1.201.102/identity_v2_admin
INFO keystone.cmd.cli [req-412f0ceb-6b28-4bb1-b2cc-aae69e76ff7b - - - - -] Created internal endpoint http://100.1.201.102/identity
INFO keystone.cmd.cli [req-412f0ceb-6b28-4bb1-b2cc-aae69e76ff7b - - - - -] Created public endpoint http://100.1.201.102/identity
2016-09-16 10:39:01.969 | +./stack.sh:main:1038 cat
2016-09-16 10:39:01.998 | +./stack.sh:main:1053 is_service_enabled tls-proxy
2016-09-16 10:39:02.055 | +functions-common:is_service_enabled:2078 return 1
2016-09-16 10:39:02.064 | +./stack.sh:main:1057 source /devstack/userrc_early
2016-09-16 10:39:02.075 | ++userrc_early:source:4 export OS_IDENTITY_API_VERSION=3
2016-09-16 10:39:02.087 | ++userrc_early:source:4 OS_IDENTITY_API_VERSION=3
2016-09-16 10:39:02.098 | ++userrc_early:source:5 export OS_AUTH_URL=http://100.1.201.102/identity_v2_admin
2016-09-16 10:39:02.109 | ++userrc_early:source:5 OS_AUTH_URL=http://100.1.201.102/identity_v2_admin
2016-09-16 10:39:02.121 | ++userrc_early:source:6 export OS_USERNAME=admin
2016-09-16 10:39:02.133 | ++userrc_early:source:6 OS_USERNAME=admin
2016-09-16 10:39:02.144 | ++userrc_early:source:8 export OS_PASSWORD=secret
2016-09-16 10:39:02.156 | ++userrc_early:source:8 OS_PASSWORD=secret
2016-09-16 10:39:02.168 | ++userrc_early:source:9 export OS_PROJECT_NAME=admin
2016-09-16 10:39:02.181 | ++userrc_early:source:9 OS_PROJECT_NAME=admin
2016-09-16 10:39:02.192 | ++userrc_early:source:11 export OS_REGION_NAME=RegionOne
2016-09-16 10:39:02.204 | ++userrc_early:source:11 OS_REGION_NAME=RegionOne
2016-09-16 10:39:02.214 | +./stack.sh:main:1059 create_keystone_accounts
2016-09-16 10:39:02.223 | +lib/keystone:create_keystone_accounts:384 local admin_project
2016-09-16 10:39:02.235 | ++lib/keystone:create_keystone_accounts:385 openstack project show admin -f value -c id
2016-09-16 10:40:19.898 | Discovering versions from the identity service failed when creating the password plugin. Attempting to determine version from URL.
2016-09-16 10:40:19.898 | Could not determine a suitable URL for the plugin
2016-09-16 10:40:19.981 | +lib/keystone:create_keystone_accounts:385 admin_project=
2016-09-16 10:40:19.992 | +lib/keystone:create_keystone_accounts:1 exit_trap
2016-09-16 10:40:20.003 | +./stack.sh:exit_trap:487 local r=1
2016-09-16 10:40:20.014 | ++./stack.sh:exit_trap:488 jobs -p
2016-09-16 10:40:20.025 | +./stack.sh:exit_trap:488 jobs=
2016-09-16 10:40:20.036 | +./stack.sh:exit_trap:491 [[ -n '' ]]
2016-09-16 10:40:20.047 | +./stack.sh:exit_trap:497 kill_spinner
2016-09-16 10:40:20.059 | +./stack.sh:kill_spinner:383 '[' '!' -z '' ']'
2016-09-16 10:40:20.069 | +./stack.sh:exit_trap:499 [[ 1 -ne 0 ]]
2016-09-16 10:40:20.080 | +./stack.sh:exit_trap:500 echo 'Error on exit'
2016-09-16 10:40:20.080 | Error on exit
2016-09-16 10:40:20.093 | +./stack.sh:exit_trap:501 generate-subunit 1474021993 427 fail
2016-09-16 10:40:20.697 | +./stack.sh:exit_trap:502 [[ -z /opt/stack/logs ]]
2016-09-16 10:40:20.703 | +./stack.sh:exit_trap:505 devstack/tools/worlddump.py -d /opt/stack/logs
2016-09-16 10:40:21.394 | +./stack.sh:exit_trap:511 exit 1