How to parse date pattern using grok - elastic-stack

How to parse the below log line using grok
Also how to match the pattern of the date.
I tried %{TIMESTAMP_ISO8601:logtime} but no match
Log Line:
13-Nov-2019 00:00:20.230 DEBUG [[ACTIVE] ExecuteThread: '272' for queue: 'weblogic.kernel.Default (self-tuning)'] [196.157.7.12] 965929132 [wire] >> "[\n]"

The question is a bit unclear as to exactly what fields you want them mapped to.
So, here's what matches for me:
%{MONTHDAY:day}[-]%{MONTH:month}[-]%{YEAR:year} %{TIME:time} %{WORD:logtype} \[\[%{WORD:status}\] ExecuteThread: '%{NUMBER:threadNumber}' for queue: '%{GREEDYDATA:queueData}'\] \[%{IP:ip}\] %{NUMBER:numbers} \[%{WORD:text}\] >> "\[\\n\]"
The first 4 fields, answer your date/time pattern query and the rest is what I have used to fit the rest of the fields. Since, no exact mappings were provided , I have mapped them as per my understanding using
This is the output:
{
"day": [
[
"13"
]
],
"month": [
[
"Nov"
]
],
"year": [
[
"2019"
]
],
"time": [
[
"00:00:20.230"
]
],
"logtype": [
[
"DEBUG"
]
],
"status": [
[
"ACTIVE"
]
],
"threadNumber": [
[
"272"
]
],
"queueData": [
[
"weblogic.kernel.Default (self-tuning)"
]
],
"ip": [
[
"196.157.7.12"
]
],
"numbers": [
[
"965929132"
]
],
"text": [
[
"wire"
]
]
}
You can break 'time' further if you want. For any other combinations of patterns, refer Grok Patterns.

Related

Enable SchemaStore support if filename does not match

VS Code has support for schemastore which gives you autocomplete in YAML files.
But VS Code does not detect the schema if the filename is different.
For example, if I edit .golangci.yaml the corresponding schema gets used. If I edit .golangci-foo.yaml the schema is not detected.
How can I enable the schema for files where the filename is different?
You can do this using the json.schemas setting. Like so:
"json.schemas": [
{
"fileMatch": [ "*tsconfig*.json" ],
"url": "http://json.schemastore.org/tsconfig",
},{
"fileMatch": [ "*cSpell.json" ],
"url": "https://raw.githubusercontent.com/streetsidesoftware/cspell/cspell4/cspell.schema.json",
},{
"fileMatch": [ "*.webmanifest" ],
"url": "http://json.schemastore.org/web-manifest",
},{
"fileMatch": [ "*package*.json" ],
"url": "https://json.schemastore.org/package",
}
],
As indicated in the comments, this worked for the asker, and they used the following:
"json.schemas": [
{
"fileMatch": [ "*.golangci*yaml" ],
"url": "https://json.schemastore.org/golangci-lint.json",
}
],

Creating an OpenLayers 5 MultiPolygon feature from a turf.js MultiPolygon

In OpenLayers 5.3.0, I've created a MultiPolygon using the 'difference' tool in turf.js. The turf.js MultiPolygon looks fine when I examine the JSON, but when I try to use that to create a feature in OpenLayers, I get "Uncaught TypeError: t.addEventListener is not a function".
I've tried many combinations of JSON.stringify, JSON.parse, GeoJSON.readFeatures, .getCoordinates()... I tried adding the turf.js MultiPolygon as a feature directly via source.addFeature(multiPolygonGeometry), but then I get 'Uncaught TypeError: e.getId is not a function'. I also tried source.addFeatures(multiPolygonGeometry) (note the plural 'addFeatures'), and that didn't give me any errors, but also didn't appear to add anything to the source.
Relevant lines in my code are as follows:
multiPolygonGeometry = turf.difference(largeArea,maskAreas);
multiPolygonFeature = new ol.Feature({
geometry: multiPolygonGeometry,
id: 'multiPolygonFeature1'
});
multiPolygonGeometry looks like this in the console:
{
"type": "Feature",
"properties": {},
"geometry": {
"type": "MultiPolygon",
"coordinates": [
[
[
[
140.9716711384525,
-36.97645228850101
],
[
140.97418321565786,
-36.97679331852701
],
[
140.9741163253784,
-36.97713531664132
],
[
140.9740304946899,
-36.97903805606076
],
[
140.97437381744382,
-36.98025509866784
],
[
140.97594864874696,
-36.98127512642501
],
[
140.9714880598484,
-36.9804459718428
],
[
140.9714500775476,
-36.97642733756345
],
[
140.9716711384525,
-36.97645228850101
]
]
],
[
[
[
140.97455248763328,
-36.97684309230892
],
[
140.97751071844857,
-36.97723786980259
],
[
140.97749308140382,
-36.977304276099005
],
[
140.97715289421623,
-36.97770848336402
],
[
140.97661807025068,
-36.97969050789806
],
[
140.97628355026242,
-36.97958658471583
],
[
140.97634792327878,
-36.97900377288852
],
[
140.9764981269836,
-36.97866094031662
],
[
140.97510337829587,
-36.97727245260485
],
[
140.97455248763328,
-36.97684309230892
]
]
],
[
[
[
140.97628420893903,
-36.98092777726751
],
[
140.97617893060388,
-36.98131793226549
],
[
140.97596635572492,
-36.98127841787872
],
[
140.97628420893903,
-36.98092777726751
]
]
]
]
},
"ol_lm": {
"change": []
}
}
And then I get this message:
events.js:174 Uncaught TypeError: t.addEventListener is not a function
at v (events.js:174)
at e.handleGeometryChanged_ (Feature.js:210)
at e (events.js:41)
at e.dispatchEvent (Target.js:101)
at e.notify (Object.js:151)
at e.set (Object.js:170)
at e.setProperties (Object.js:186)
at new e (Feature.js:108)
at getPolygon (maskedPolygon.js:319) <-- this is the second line in my code sample above
at <anonymous>:1:1
What am I doing wrong here? I'm sure it's something simple but I just can't seem to crack this.
Turf works with GeoJSON features so your "multiPolygonGeometry" is a GeoJSON feature which can be parsed by OpenLayers then given an Id:
multiPolygonFeature = new ol.format.GeoJSON().readFeature(multiPolygonGeometry);
multiPolygonFeature.setId('multiPolygonFeature1');

Watson Speech To Text, Word timestamps are not in sync with Audio

I am using speech to text with following params -
timestamps=true&max_alternatives=1&model=en-US_NarrowbandModel&smart_formatting=true',
Headers - 'Content-Type' => 'audio/flac', 'Transfer-Encoding' => 'chunked'
And providing an audio/flac file to process, but returned words time boundary is not in sync with audio.
For eg. Response is -:
take a morning I have 2 questions please %HESITATION first how how much of the ability
Timestamps are like these -
[
[
"take",
1409.48,
1409.62
],
[
"a",
1409.62,
1409.67
],
[
"morning",
1409.67,
1410.03
],
[
"I",
1410.06,
1410.17
],
[
"have",
1410.17,
1410.38
],
[
"two",
1410.41,
1410.58
],
[
"questions",
1410.58,
1411.05
],
[
"please",
1411.05,
1411.42
],
[
"%HESITATION",
1411.42,
1411.65
],
[
"first",
1411.65,
1412.17
],
[
"how",
1412.33,
1412.62
],
[
"how",
1412.65,
1412.77
],
[
"much",
1412.77,
1413
],
[
"of",
1413,
1413.1
],
[
"the",
1413.1,
1413.37
],
[
"ability",
1413.37,
1413.82
]
]
But in actual audio these words are at different times. (few seconds difference)
Any Suggestion ??

Is there a better way to write this Postgres JSONB query?

Below is a sample JSONB array. I'm trying to figure out how to write a query that doesn't require a cross product like this.
select b.id from brand b,jsonb_array_elements (b.tree) a where a#>>'{Name}' = 'Skiing';
Bonus points for helping me translate this to SQL Alchemy
[
{
"Name": "Snowboarding",
"Order": 1,
"Categories": {
"Jackets": [
22002,
23224
],
"Helmets": [
24920
],
"Freestyle Boards": [
20164
],
"Goggles": [
23169,
23280
],
"Hats": [
22966,
21727
],
"Bindings": [
19265
],
"Gloves": [
20461
],
"Boots": [
26374,
19079,
21765,
22669
],
"Freeride Boards": [
18395,
25505
],
"Pants": [
24143,
20957
]
}
},
{
"Name": "Skiing",
"Order": 2,
"Categories": {
"Jackets": [
22518,
25791,
19972
],
"Pants": [
17516,
23113
],
"Goggles": [
25066,
20996
],
"Helmets": [
24378
],
"Hats": [
20009,
21245
],
"Cross-country Skiing": [
17464
],
"Gloves": [
25822
],
"Boots": [
16616
],
"Poles": [
19280
]
}
},....]
SQL solution first:
SELECT brand.id
FROM brand
WHERE brand.tree #> '[{"Name": "Skiing"}]'::jsonb;
As for sqlalchemy version, you can simply use contains in order to generate SQL statement above:
q = (session.query(Brand.id)
.filter(Brand.tree.contains([{"Name": "Skiing"}]))
)

Exterior Shell of polygon is invalid - 2dSphere Polygon

Question, trying to index and getting the following error:
"Exterior shell of polygon is invalid"
However, I've tested the JSON on http://geojsonlint.com/ and it works
Here's my JSON
{
"type": "Polygon",
"coordinates": [
[
[
116.306655,
39.984977
],
[
116.30673,
39.984977
],
[
116.306734,
39.98483
],
[
116.30667,
39.98483
],
[
116.306678,
39.984714
],
[
116.306384,
39.984705
],
[
116.30638,
39.984858
],
[
116.306193,
39.984852
],
[
116.306198,
39.984601
],
[
116.306031,
39.984597
],
[
116.306031,
39.984596
],
[
116.306031,
39.984596
],
[
116.306023,
39.984961
],
[
116.306082,
39.984964
],
[
116.306082,
39.985019
],
[
116.306655,
39.985032
],
[
116.306655,
39.984977
]
]
]
}
What version of mongodb are you running. If you are running 2.4, try upgrading to 2.6. I had the same error message with census tiger line data that I had converted to GeoJSON, and which I had also confirmed via jsonlint. I had been running the index on version 2.4. I upgraded to mongodb version 2.6, and that solved my problem. I was able to create a 2dsphere index and run geonear queries just fine after that.