I'm trying set ElastAlert for Opensearch 2.8.
I Write config
# This is the folder that contains the rule yaml files
# Any .yaml file will be loaded as a rule
rules_folder: /etc/elastalert/rules
# How often ElastAlert will query Elasticsearch
# The unit can be anything from weeks to seconds
run_every:
minutes: 1
# ElastAlert will buffer results from the most recent
# period of time, in case some log sources are not in real time
buffer_time:
minutes: 15
# The Elasticsearch hostname for metadata writeback
# Note that every rule can have its own Elasticsearch host
es_host: localhost
# The Elasticsearch port
es_port: 9200
# The AWS region to use. Set this when using AWS-managed elasticsearch
#aws_region: us-east-1
# The AWS profile to use. Use this if you are using an aws-cli profile.
# See http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-started.html
# for details
#profile: test
# Optional URL prefix for Elasticsearch
#es_url_prefix: elasticsearch
# Connect with TLS to Elasticsearch
use_ssl: True
# GET request with body is the default option for Elasticsearch.
# If it fails for some reason, you can pass 'GET', 'POST' or 'source'.
# See http://elasticsearch-py.readthedocs.io/en/master/connection.html?highlight=send_get_body_as#transport
# for details
# es_send_get_body_as: GET
# Option basic-auth username and password for Elasticsearch
es_username: admin
es_password: password
# Use SSL authentication with client certificates client_cert must be
# a pem file containing both cert and key for client
verify_certs: False
#ca_certs: /path/to/cacert.pem
#client_cert: /path/to/client_cert.pem
#client_key: /path/to/client_key.key
# The index on es_host which is used for metadata storage
# This can be a unmapped index, but it is recommended that you run
# elastalert-create-index to set a mapping
writeback_index: elastalert_status
writeback_alias: elastalert_alerts
# If an alert fails for some reason, ElastAlert will retry
# sending the alert until this time period has elapsed
alert_time_limit:
days: 2
... And rule file
# Alert when the rate of events exceeds a threshold
.
# (Optional)
# Elasticsearch host
es_host: localhost
.
# (Optional)
# Elasticsearch port
es_port: 9200
.
# (OptionaL) Connect with SSL to Elasticsearch
use_ssl: True
ssl_show_warn: False
verify_certs: False
.
# (Optional) basic-auth username and password for Elasticsearch
# es_username: admin
# es_password: ytnhfvgkby
.
# (Required)
# Rule name, must be unique
name: Loopdetect
.
# (Required)
# Type of alert.
# the frequency rule type alerts when num_events events occur with timeframe time
type: any
.
# (Required)
# Index to search, wildcard supported
index: syslog-20221104
.
# (Required, frequency specific)
# Alert when this many documents matching the query occur within a timeframe
num_events: 1
.
# (Required, frequency specific)
# num_events must occur within this amount of time to trigger an alert
timeframe:
hours: 24
.
# (Required)
# A list of Elasticsearch filters used for find events
# These filters are joined with AND and nested in a filtered query
# For more info: http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/query-dsl.html
# filter:
# - term:
# process.name: "JUSTME"
filter:
- query:
query_string:
query: "message: *loop*"
# (Required)
# The alert is use when a match is found
alert:
- "email"
.
# (required, email specific)
# a list of email addresses to send alerts to
email:
- "myemail"
But when I try check this rule.
I get error:
elastalert-test-rule rules/loopdetect_alert.yaml
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
To send them but remain verbose, use --verbose instead.
WARNING:elasticsearch:POST https://localhost:9200/syslog-20221104/_search?ignore_unavailable=true&size=1 [status:400 request:0.048s]
Error running your filter:
RequestError(400, 'search_phase_execution_exception', {'error': {'root_cause': [{'type': 'query_shard_exception', 'reason': 'No mapping found for [#timestamp] in order to sort on', 'index': 'syslog-20221104', 'index_uuid': 'BG6MQmmYRUyLBY3tEFykEQ'}], 'type': 'search_phase_execution_exception', 'reason': 'all shards failed', 'phase': 'query', 'grouped': True, 'failed_shards': [{'shard': 0, 'index': 'syslog-20221104', 'node': '5spTsU7-QienT8Jn064MMA', 'reason': {'type': 'query_shard_exception', 'reason': 'No mapping found for [#timestamp] in order to sort on', 'index': 'syslog-20221104', 'index_uuid': 'BG6MQmmYRUyLBY3tEFykEQ'}}]}, 'status': 400})
INFO:elastalert:Note: In debug mode, alerts will be logged to console but NOT actually sent.
To send them but remain verbose, use --verbose instead.
INFO:elastalert:1 rules loaded
INFO:apscheduler.scheduler:Adding job tentatively -- it will be properly scheduled when the scheduler starts
WARNING:elasticsearch:POST https://localhost:9200/syslog-20221104/_search?_source_includes=%40timestamp%2C%2A&ignore_unavailable=true&scroll=30s&size=10000 [status:400 request:0.039s]
ERROR:elastalert:Error running query: RequestError(400, 'search_phase_execution_exception', 'No mapping found for [#timestamp] in order to sort on')
{"writeback": {"elastalert_error": {"message": "Error running query: RequestError(400, 'search_phase_execution_exception', 'No mapping found for [#timestamp] in order to sort on')", "traceback": ["Traceback (most recent call last):", " File \"/usr/local/lib/python3.11/dist-packages/elastalert2-2.8.0-py3.11.egg/elastalert/elastalert.py\", line 370, in get_hits", " res = self.thread_data.current_es.search(", " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", " File \"/usr/local/lib/python3.11/dist-packages/elasticsearch/client/utils.py\", line 152, in _wrapped", " return func(*args, params=params, headers=headers, **kwargs)", " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", " File \"/usr/local/lib/python3.11/dist-packages/elasticsearch/client/__init__.py\", line 1658, in search", " return self.transport.perform_request(", " ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^", " File \"/usr/local/lib/python3.11/dist-packages/elasticsearch/transport.py\", line 392, in perform_request", " raise e", " File \"/usr/local/lib/python3.11/dist-packages/elasticsearch/transport.py\", line 358, in perform_request", " status, headers_response, data = connection.perform_request(", " ^^^^^^^^^^^^^^^^^^^^^^^^^^^", " File \"/usr/local/lib/python3.11/dist-packages/elasticsearch/connection/http_requests.py\", line 199, in perform_request", " self._raise_error(response.status_code, raw_data)", " File \"/usr/local/lib/python3.11/dist-packages/elasticsearch/connection/base.py\", line 315, in _raise_error", " raise HTTP_EXCEPTIONS.get(status_code, TransportError)(", "elasticsearch.exceptions.RequestError: RequestError(400, 'search_phase_execution_exception', 'No mapping found for [#timestamp] in order to sort on')"], "data": {"rule": "Loopdetect", "query": {"query": {"bool": {"filter": {"bool": {"must": [{"range": {"#timestamp": {"gt": "2022-11-03T12:12:39.618168Z", "lte": "2022-11-03T12:27:39.618168Z"}}}, {"query_string": {"query": "message: *loop*"}}]}}}}, "sort": [{"#timestamp": {"order": "asc"}}]}}}}}
But if I try get data by CURL, it's ok
curl -X GET 'https://localhost:9200/syslog-20221104/_search?ignore_unavailable=true&size=1' -u 'admin:password' --insecure
{"took":4,"timed_out":false,"_shards":{"total":1,"successful":1,"skipped":0,"failed":0},"hits":{"total":{"value":10000,"relation":"gte"},"max_score":1.0,"hits":[{"_index":"syslog-20221104","_id":"_bSKQYQB_cpiH2g_hgvj","_score":1.0,"_source":{"host":"10.53.0.35","hostname":"10.53.0.35","message":"Port 2 link up, 100Mbps FULL duplex","source_ip":"91.195.230.4","source_type":"syslog","timestamp":"2022-11-04T07:28:27Z"}}]}}
Help me please understand, what I do wrong.
Thanks.
I add timestamp_field: timestamp.
And all work fine!
I am new to K6 and is trying to use the tool to perform a Get request by verifying an API.
When the script is executed I get a warning that terminates the scrip. As far as my understanding is that this error is somewhat related to Go (if I have understood it correctly).
The result that I want to achieve is to be able to execute the Get request to the endpoint URL, but would appreciate any kind of feedback if I have done any incorrectly or should try an other approach.
Script:
import http from "k6/http";
import { check } from "k6";
export default function () {
var url =
"https://endpoint.example.to.cloud/api/reports/v1/SMOKETESTC6KP6NWX";
var headerParam = {
headers: {
"Content-Type": "application/json",
},
};
const response = http.get(url, headerParam);
check(response, {
"Response status reciving a 200 response ": (r) => r.status === 200,
});
let body = JSON.parse(response.body);
}
Output:
WARN[0000] Request Failed error="Get \"https://endpoint.example.to.cloud/api/reports/v1/SMOKETESTC6KP6NWX\": x509: certificate relies on legacy Common Name field, use SANs or temporarily enable Common Name matching with GODEBUG=x509ignoreCN=0"
Changing URL endpoint:
If i change the URL endpoint (mockup url) like below, there will be no errors:
...
var url = "https://run.mocky.io/v3/16fa8113-57e0-4e47-99b9-b5c55da93d71";
...
Updated solution to run this locally:
In order to run this locally i had to add the certification and key:
Example:
export let options = {
...
tlsAuth: [
{
cert: open(`${__ENV.Certificate}`),
key: open(`${__ENV.Key}`),
},
],
};
In addition populate the execute command with --insecure-skip-tls-verify
Example:
k6 run -e Certificate=/home/cert/example_certification.crt -e Key=/home/cert/certification/example_key.key -e example.js --insecure-skip-tls-verify
k6 is written in Go, and the latest versions of Go have a breaking change in how they handle X.509 certificates: https://golang.org/doc/go1.15#commonname
As it says in the error message, you can temporarily allow the old behavior by setting a GODEBUG=x509ignoreCN=0 environment variable, but that will likely stop working in a few months with Go 1.17. Using the insecureSkipTLSVerify k6 option might also work, I haven't checked, but as the name implies, that stops any TLS verification and is insecure.
So the real solution is to re-generate your server-side certificate properly.
I am doing some infrastructure testing in Pester and there is repeating scenario that I don't know how to approach.
Let's say, I want to check whether all required web roles are enabled on IIS. I have a collection of required web roles and for each of them I want to assert it is enabled.
My current code looks like this:
$requiredRoles = #(
"Web-Default-Doc",
"Web-Dir-Browsing",
"Web-Http-Errors",
"Web-Static-Content",
"Web-Http-Redirect"
)
Context "WebRoles" {
It "Has installed proper web roles" {
$requiredRoles | % {
$feature = Get-WindowsOptionalFeature -FeatureName $_ -online
$feature.State | Should Be "Enabled"
}
}
}
It works in the sense that the test will fail if any of the roles are not enabled/installed. But that is hardly useful if the output of such Pester test looks like this:
Context WebRoles
[-] Has installed proper web roles 2.69s
Expected: {Enabled}
But was: {Disabled}
283: $feature.State | Should Be "Enabled"
This result doesn't give any clue about which feature is the Disabled one.
Is there any recommended practice in these scenarios? I was thinking about some string manipulation...
Context "WebRoles" {
It "Has installed proper web roles" {
$requiredRoles | % {
$feature = Get-WindowsOptionalFeature -FeatureName $_ -online
$toCompare = "{0}_{1}" -f $feature.FeatureName,$feature.State
$toCompare | Should Be ("{0}_{1}" -f $_,"Enabled")
}
}
}
which would output:
Context WebRoles
[-] Has installed proper web roles 2.39s
Expected string length 27 but was 28. Strings differ at index 20.
Expected: {IIS-DefaultDocument_Enabled}
But was: {IIS-DefaultDocument_Disabled}
-------------------------------^
284: $toCompare | Should Be ("{0}_{1}" -f $_,"Enabled")
...which is better, but it doesn't feel very good...
Also, there is second problem with the fact that the test will stop on first fail and I would need to re-run the test after I fix each feature...
Any ideas?
Put your It inside the loop like so:
Context "WebRoles" {
$requiredRole | ForEach-Object {
It "Has installed web role $_" {
(Get-WindowsOptionalFeature -FeatureName $_ -online).State | Should Be "Enabled"
}
}
}
I am trying to checkout code from SVN repo for which I am accepting the URL as argument. I have quoted the URL as shown below because it contains spaces. I also checked the parameter by redirecting the $svn_url in file (shown below). If I pick the URL from the file and pass it as is on the command line to the given script, it works fine but somehow when invoked from Puppet, it's not working.
Puppet manifests:
repo_checkout.pp:
define infra::svn::repo_checkout ($svn_url_params) {
$svn_url = $svn_url_params[svn_url]
include infra::params
$repo_checkout_ps = $infra::params::repo_checkout_ps
file { $repo_checkout_ps:
ensure => file,
source => 'puppet:///modules/infra/repo_checkout.ps1',
}
util::executeps { 'Checking out repo':
pspath => $repo_checkout_ps,
argument => "\'\"$svn_url\"\'",
}
}
params.pp:
$repo_checkout_ps = 'c:/scripts/infra/repo_checkout.ps1',
site.pp:
$svn_url_ad = {
svn_url => 'https:\\\\some_repo.abc.com\svn\dir with space\util',
}
infra::svn::repo_checkout { "Checking out code in C:\build":
svn_url_params => $svn_url_ad
}
executeps.pp:
define util::executeps ($pspath, $argument) {
$powershell = 'C:/Windows/System32/WindowsPowerShell/v1.0/powershell.exe -NoProfile -NoLogo -NonInteractive'
exec { "Executing PS file \"$pspath\" with argument \"$argument\"":
command => "$powershell -file $pspath $argument",
timeout => 900,
}
}
PowerShell code:
$svn_url = $args[0]
Set-Location C:\build
echo "svn co --username user --password xxx --non-interactive '$svn_url'" | Out-File c:\svn_url
svn co --username user --password xxx --non-interactive '$svn_url'
Puppet output on agent node:
Util::Executeps[Checking out repo]/Exec[Executing PS file "c:/scripts/infra/repo_checkout.ps1" with argument "'"https:\\some_repo.abc.com\svn\dir with space\util"'"]/returns: executed successfully
Notice: Applied catalog in 1.83 seconds
Content of c:\svn_url:
'https:\\\\some_repo.abc.com\svn\dir with space\util'
UPDATE: Sorry for the confusion but i was trying out several permutations and combinations and in doing that, i forgot to mention that when the $svn_url contains backslash (\), it does NOT work on the command line too if i copy the SVN URL from the text file where i am redirecting the echo output.
Based on #Ansgar's suggestion, i changed '$svn_url' to "$svn_url" in powershell code but the output in text file then contained ' quote twice around the URL. So i changed the argument parameter from "\'\"$svn_url\"\'" to "\"$svn_url\"". Now the output file had only single quote present around the URL. I copied only the URL (along with single quotes around it) from the output file and tried passing it to the powershell script. I now get the following error:
svn: E020024: Error resolving case of 'https:\\some_repo.abc.com\svn\dir with space\util'
Another thing to note is that if i change the back slashes in URL to forward slashes, it works fine on the command line. Invoking from Puppet still doesn't work.
Posting the final configuration that worked out for me based on #AnsgarWiechers' suggestion.
[tom#pe-server] cat repo_checkout.pp
define infra::svn::repo_checkout ($svn_url_params) {
$svn_url = $svn_url_params[svn_url]
...
...
util::executeps { 'Checking out repo':
pspath => $repo_checkout_ps,
argument => "\"$svn_url\"",
}
}
[tom#pe-server] cat repo_checkout.ps1
$svn_url = $args[0]
Set-Location C:\build
svn co --username user --password xxx --non-interactive "$svn_url"
[tom#pe-server] cat params.pp
$repo_checkout_ps = 'c:/scripts/infra/repo_checkout.ps1',
[tom#pe-server] cat site.pp
$svn_url_ad = {
svn_url => 'https://some_repo.abc.com/svn/dir with space/util',
}
infra::svn::repo_checkout { "Checking out code in C:\build":
svn_url_params => $svn_url_ad
}
Thanks a lot #AnsgarWiechers! :)
Note:
In site.pp: Used forwardslashes (/) when specifying svn_url
In repo_checkout.ps1: Changed '$svn_url' to "$svn_url"
In repo_checkout.pp: Changed double-nested (' and ") quoting in argument to single (") nested i.e., from "\'\"$svn_url\"\'" to "\"$svn_url\""
Am trying to import / read Windows server event logs to a text file, using a wevtutil command. I use the following command to write my logs to file.txt:
$ wevtutil qe Application \rd:true \f:text (reads application logs)
and the sample output of my command, is:
Event[1]:
Log Name: Application
Source: Microsoft-Windows-Security-SPP
Date: 2016-03-29T13:02:27.000
Event ID: 8196
Task: N/A
Level: Information
Opcode: N/A
Keyword: Classic
User: N/A
User Name: N/A
Computer: WIN-IONOGQTF9O5
Description: License Activation Scheduler (sppuinotify.dll)
Event[2]:
Log Name: Application
Source: Microsoft-Windows
Date: 2016-06-29T13:02:57.000
Event ID: 3444
Task: N/A
Level: Critical
Opcode: N/A
Keyword: Classic
User: N/A
User Name: N/A
Computer: WIN-IONOGDFFF9O5
Description: AIRO.Activation code(sppuinotify.dll)
(Actually,two sample logs).
but, i want to write my log as a single line to .txt file, rather than the above multi-line output for a single log. is there a wevtutil command utility to write a log to a single line, like below:
Event[1]:Log Name: Application Source: Microsoft-Windows-Security-SPP Date: 2016-03-29T13:02:27.000 Event ID: 8196 Task: N/A Level: Information Opcode: N/A Keyword: Classic User: N/A User Name: N/A Computer: WIN-IONOGQTF9O5 Description: License Activation Scheduler (sppuinotify.dll)
Event[2]:Log Name: Application Source: Microsoft-Windows Date: 2016-03-29T13:02:27.000 Event ID: 8196 Task: N/A Level: Information Opcode: N/A Keyword: Classic User: N/A User Name: N/A Computer: WIN-IONOGQTF9O5 Description: License Activation Scheduler (sppuinotify.dll)
Thanks!
$logname = "Application"
$events = Get-EventLog -LogName $logname
$arr = #()
$counter = 1
foreach($event in $events){
$arr += "Event[$counter]:Log Name: $logname Source: $($event.Source) Date: $($event.TimeWritten) Event ID: $($event.EventID) Task: $($event.Category) Level: $($event.EntryType) ..."
$counter++
}
$arr | out-file events.txt
If you need to have Opcode, Keyword etc. use Get-Winevent instead of Get-Eventlog