I know there are 30 of the same questions out there, but im not able to fix my issue with one of the answers. I got my index working for 7 days then I decided to remove the data with:
DELETE csv
Because I did upload the same date over and over to test. After this I tryed to upload the data again so I only got one copy of it. I did not change anything in my .csv files that im uploading.
But I got the error message:
[2017-11-29T14:23:44,345][WARN ][logstash.outputs.elasticsearch] Could not index event to Elasticsearch. {:status=>400, :action=>["index", {:_id=>nil, :_index=>"csv", :_type=>"csv", :_routing=>nil}, 2017-10-01T04:13:19.000Z DESKTOP-*** **.0.0.201,Fred Orr ,Fred_Orr#**.s**m,2017-10-01;06:13:19,** Story: This Tiny Pill Changes Everythi], :response=>{"index"=>{"_index"=>"csv", "_type"=>"csv", "_id"=>"*****", "status"=>400, "error"=>{"type"=>"mapper_parsing_exception", "reason"=>"failed to parse [timestamp]", "caused_by"=>{"type"=>"illegal_argument_exception", "reason"=>"Invalid format: \"2017-10-01;06:13:19\" is malformed at \";06:13:19\""}}}}}
In logstash I got the following configuration:
date {
match => ["timestamp", "YYYY-MM-dd;HH:mm:ss"]
target => "#timestamp"
}
My date in the csv file is: 2017-10-01;06:13:19
And I try to match it against 2017-10-01;06:13:19? But it fails on the ;06:13:19 part. What is going wrong?
I tryed to replace the ; with - or a space but nothing did work.
so:
2017-10-01 06:13:19
2017-10-01-06:13:19
But I keep getting the error with the last part of the time.
Mapping:
"properties": {
"#timestamp": {
"type": "date"
},
I dont understand what is wrong? It worked before I deleted the inde
For non-formatting syntax, you’ll need to put single-quote characters around the value. Try this instead:
date {
match => ["timestamp", "YYYY-MM-dd';'HH:mm:ss"]
target => "#timestamp"
}
Related
I have the following json data I am trying to import into mongodb.
{
"subjectId": "63cd96779e66d518f3af574c",
"name":"Earth and Space Science",
"subtopics": [
{
"name": "Rocks, Soil and Minerals",
"questions": [
{
"question": "What type of rock is formed when magma cools and hardens?",
"multipleChoice":["Sedimentary", "Metamorphic", "Igneous", "All of the above"],
"answer": "Igneous"
}
]
}
]
}
I am getting the error: Operation passed in cannot be an array.
Do you know why?
Maybe, the Problem is the formatted json structure with you tabs and spaces, it has to be unformatted in a Oneliner.
Try this:
{"subjectId":"63cd96779e66d518f3af574c","name":"Earth and Space Science","subtopics":[{"name":"Rocks, Soil and Minerals","questions":[{"question":"What type of rock is formed when magma cools and hardens?","multipleChoice":["Sedimentary","Metamorphic","Igneous","All of the above"],"answer":"Igneous"}]}]}
Refer to this.
MongoDB Compass has an unusual JSON file import format requirement in that each document must be delimited by a newline.
When importing data from a JSON file, you can format your data as:
Newline-delimited documents, or
Comma-separated documents in an array
After formatting your JSON document file as required, MongoDB Compass imports your file normally. Hopefully Compass's file import parser will become "smarter".
In my Below code , Getting syntax error in line no : 2
I am using only one integer variable today
async def get_users(self):
today = 5
return self.col.find({'prexdate': {$gte: int(today)}}) #in this line getting a syntax error
your field is 'prexdate':
try this instead prexdate:
Here is also a link https://www.mongodb.com/docs/manual/reference/method/db.collection.find/
I'm trying to find a pattern for this line of log (extracted from catalina.log) of an apache tomcat 8 installation.
30-Apr-2019 15:40:40.044 INFOS [main] org.apache.catalina.startup.VersionLoggerListener.log message
No one of the date pattern include in logstash matches with this date format.
Do you have idea how can I parse this date 30-Apr-2019 15:40:40.044 to a timestamp in my logstash filter ?
Thanks
As stated by #baudsp, you may add the date pattern for catalina using a custom pattern file, or use it embedded in the grok, as shown here
(?<date>%{MONTHDAY}-%{MONTH}-%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND}))
If you use the pattern often, put it in a file would probably be better, and provide more readability
Finally, there is a solution :
I put a new pattern in a file custom.txt
MY_DATE_PATTERN %{MONTHDAY}-%{MONTH}-%{YEAR} %{HOUR}:?%{MINUTE}(?::?%{SECOND})
Then in my logstash.conf I put this filter :
grok {
patterns_dir => ["./patterns"]
match => {
"message" => "%{MY_DATE_PATTERN:timestamp}%{SPACE}%{GREEDYDATA:loglevel}%{SPACE}\[%{GREEDYDATA:thread}\]%{SPACE}%{JAVACLASS:classname}%{SPACE}%{GREEDYDATA:logmessage}"
}
}
date {
match => [ "timestamp" , "dd-MMM-yyyy HH:mm:ss.SSS" ]
}
Thanks for your help
I'm trying to calculate the current age of file in Elasticsearch.
My plan was to do something like current time - timestamp.
I am now using scripts to do this and managed to subtract one date field from another
{ "query" :
{
"match_all" : {}
},
"fields" : ["_source"],
"script_fields" :
{ "date_diff" :
{ "script" : "doc[\"two\"].date-doc[\"one\"].date"}
}
}'
although I think it was done wrong as the answer was definitely not correct. (It worked out to thousands of days difference)
I have also tried using the Elasticsearch date-math suggestions such as "now", "time", "+1h" etc and all of these result in error
"JsonParseException[Unrecognized token 'time': was expecting 'null', 'true', 'false' or NaN\n at [Source: [B#220c4a0b; line: 1, column: 111]]; }]"
I'm unsure now if scripts is even the right thing to use. It seems like it should be simple to do, but I can't seem to find any examples.
Is this even possible to do? Is there a better way to do it?
Any help would be appreciated.
Can you show usecase for that? If you need to show age at UI, for example, its still better to response with creation date. If you need some analysis, filtering by creation date should do the trick.
If you are sure that you need to return "age" of document with ES query, try:
{
"query": {
"match_all": {}
},
"fields": ["_source"],
"script_fields": {
"age": {
"script": "(DateTime.now().getMillis() - doc['created'].value)/(24*60*60*1000)"
}
}
}
ES handles dates as unix epoch timestamps, you should cast current time to integer to reflect ms (with DateTime.now().getMillis()) and then subtract docs value from that. This will give you age in milliseconds. Divide by any coefficient, if needed (i.e., tj get age in days)
I am trying to query a row based on a date field in elasticsearch (not the range), using the lucene syntax but not getting the expected response,
I checked the mapping using _mapping it gives me the following result for the index, for the field ordered at:
ordered_at: {
type: "date",
format: "dateOptionalTime"
}
http://www.elasticsearch.org/guide/en/elasticsearch/reference/current/mapping-date-format.html#date-math
and
http://joda-time.sourceforge.net/api-release/org/joda/time/format/ISODateTimeFormat.html#dateOptionalTimeParser()
suggests that the following format should be acceptable yyyy-mm-dd (2015-01-03).
Following are some of the formats i have tried to fetch the result:
_search?pretty=true&q=ordered_at:"2015\-01\-03"
_search?pretty=true&q=ordered_at:2015\-01\-03*
_search?pretty=true&q=ordered_at:2015-01-03
_search?pretty=true&q=ordered_at:[2015\-01\-03 TO 15\-01\-20]
_search?pretty=true&q=ordered_at:[2015-01-03 TO 15-01-20]
_search?pretty=true&q=ordered_at:[20150103 TO 150120]
_search?pretty=true&q=ordered_at:01/03/2015
_search?pretty=true&q=ordered_at:"20150103"
_search?pretty=true&q=ordered_at:20150103
_search?pretty=true&q=ordered_at:"2015-01-03T18:53:37.000Z" (this does work but i need to just submit date and not time)
_search?pretty=true&q=ordered_at:"2015-01-03"
_search?pretty=true&q=ordered_at:"2015-01-03*"
_search?pretty=true&q=ordered_at:2015-01-03*
_search?pretty=true&q=ordered_at:2015/01/03
_search?pretty=true&q=ordered_at:2015\01\03
_search?pretty=true&q=ordered_at:2015-01-03
_search?pretty=true&q=ordered_at:2015\-01\-03
_search?pretty=true&q=ordered_at:2015-01-03*
_search?pretty=true&q=ordered_at:2015/-01/-03
_search?pretty=true&q=ordered_at:"2015-03-01"
_search?pretty=true&q=ordered_at:2015/03/01
_search?pretty=true&q=ordered_at:"2015-01-03"*
Following is a sample of how the data is displayed when we query by other fields:
ordered_at: [
"2015-01-03T18:53:37.000Z"
]
No idea what am i missing, any clue? Thanks..
Try this: q=date:[2015-01-03 TO 2015-01-20].