I'm searching Cloudwatch log events for errors with the following criteria:
?"error" ?"ERROR" ?"Error:"
How can I exclude specific terms from the result? For example, if I don't care about specific_error, how can I specify not to match on it?
I'm expecting to be able to do something like:
(?"error" AND -"specific_error") ?"ERROR" ?"Error:"
In the CloudWatch console, this can be accomplished with the - operand before the term you wish to exclude:
"error" -"something minor happened"
This is from the AWS docs for "Matching terms in log events".
Similarly, using aws logs tail, you can pass this to the --filter-pattern argument:
$ aws logs tail --format short /aws/lambda/my_lambda --filter-pattern '"error" -"something minor happened"' --since 3h
2021-07-09T19:28:47 error: something bad happened
2021-07-09T19:28:51 error: something bad happened
2021-07-09T19:29:52 error: something REALLY bad happened
2021-07-09T19:30:15 error: something CATASTROPHIC happened! Aiee!
2021-07-09T19:30:36 error: something bad happened
Related
My question is on a Sentry integration in an electronjs app.
We have some errors in Sentry that show a warning: 'Source code was not found'
Which usually means that source code mapping is not properly set. So I read the doc and made sure that our source code maps are properly set. Which is the case in my opinion.
Still according to the doc, I installed sentry-cli and run this command:
sentry-cli sourcemaps explain ec9d14e80e464fe49fb917a5d383cc61 --org outmind
Which return:
✔ Fetched data for event: bec1792b5c8048ce9663f35d07936369
✔ Event has release name: desktop#3.29.29
✔ Event has a valid exception present
✔ Event has a valid stacktrace present
✖ Event exception stacktrace has no in_app frames
And I do have a warning on the sentry page:
X By in-app with a hover: Non contributing variant: exception of system take precedence
Most of this scenario happen on axios call that return 401 errors - which is expected.
And I have absolutely no idea what's the issue here or what I'm supposed to do... Send a mail to the support, didn't get any answers.
Any suggestion on what I'm supposed to fix is welcome,
Thanks all
Do binary operators for Loki queries exist?
Is there a workaround to achieve something as this?
{job="myService"}
|~"AI Unable to update the session state in"
OR
|~"Database Write Error"
No, you can't do that but you can do, for example, the following:
{job="myService"} |~ "AI Unable to update the session state in|Database Write Error"
Or the following:
count_over_time({job="myService"} |~ "AI Unable to update the session state in" [1h])
or
count_over_time({job="myService"} |~ "Database Write Error" [1h])
I'm seeing the following warning in the /var/log/amazon/amazon-cloudwatch-agent/amazon-cloudwatch-agent.log:
2021-10-06T06:39:23Z W! [outputs.cloudwatchlogs] Invalid SequenceToken used, will use new token and retry: The given sequenceToken is invalid. The next expected sequenceToken is: 49619410836690261519535138406911035003981074860446093650
But there is no mention about which file is really the one that it's failing. Not even when I add "debug": true to the /opt/aws/amazon-cloudwatch-agent/bin/config.json.
cat /opt/aws/amazon-cloudwatch-agent/bin/config.json|jq .agent
{
"metrics_collection_interval": 60,
"debug": true,
"run_as_user": "root"
}
I have many (28) files in my .logs.logs_collected.files.collect_list section of the config.json file, so how can I find which file is exactly causing trouble?
As of 2021-11-29 a PR to improve the log messages has been merged to the cloudwatch-agent but a new version of the cloudwatch-agent has not been released yet, the next version after v1.247349.0 will likely include a fix for this.
The fix will change the log statements to say
INFO: First time sending logs to %v/%v since startup so sequenceToken is nil, learned new token: xxxx: yyyy: This is an INFO message, as this behaviour is expected at startup for example.
WARN: Invalid SequenceToken used (%v) while sending logs to %v/%v, will use new token and retry: xxxxxv: This on the other hand is not expected and may mean that someone else is writing to the loggroup/logstream concurrently.
If those warnings come right after a restart of the cloudwatch agent (cwagent) then you can safely ignore them, it's expected behaviour . The cloudwatch agent does not save the next sequence token in its persistent state so on restart it will "learn" the correct sequence number by issuing a PutLogEvent with no sequence token at all, that returns an InvalidSequenceTokenException with the next sequence token to use. So it's expected to see those at startup, anyway I proposed a PR to amazon-cloudwatch-agent to improve those log messages.
If the "Invalid SequenceToken used" is seen long after the restart then you may have other issues.
The "Invalid SequenceToken used" error usually means that two entities/sources are trying to write to the same log group/log stream as mentioned in 2 (which is really for the old awslogs agent but still useful):
Caught exception: An error occurred (InvalidSequenceTokenException)
when calling the PutLogEvents operation: The given sequenceToken is
invalid[…] -or- Multiple agents might be sending log events to log
stream[…] – You can't push logs from multiple log files to a single
log stream. Update your configuration to push each log to a log
stream-log group combination.
I could be that the amazon cloudwatch agent itself it's trying to upload the same file twice because you have duplicates in your config.json.
So first print all your log group / log stream pairs in your config.json with:
cat /opt/aws/amazon-cloudwatch-agent/bin/config.json|jq -r '.logs.logs_collected.files.collect_list[]|"\(.log_group_name) \(.log_stream_name)"'|sort
which should give an output similar to:
/tableauserver/apigateway apigateway_node5-0.log
/tableauserver/apigateway control_apigateway_node5-0.log
/tableauserver/appzookeeper appzookeeper-discovery_node5-1.log
...
/tableauserver/vizqlserver vizqlserver_node5-3.log
Then you can use uniq -d to find the duplicates in that list with:
cat /opt/aws/amazon-cloudwatch-agent/bin/config.json|jq -r '.logs.logs_collected.files.collect_list[]|"\(.log_group_name) \(.log_stream_name)"'|sort|uniq -d
# The list should be empty otherwise you have duplicates
If that command produces any output it means that you have duplicates in your config.json collect_list.
I personally think that cwagent itself should print the "offending" loggroup/logstream in the logs so I opened in issue in amazon-cloudwatch-agent GitHub page.
I am running a gremlin-server, and using restful api to query it. But I'm confused with this error below :
{"message":"Error encountered evaluating script: g.V().next()"}
but it is such a simple script.
for other script like "100-1" and "g", the query result is just OK. I have checked lots times for spelling and character. And I also have checked the gremlin-server logs, but there was no related records. So I ask for your help, Thanks!
You would run into that error if you don't have any data in the graph because it is an unchecked traversal. You should try a query like:
if (g.V().hasNext()) { g.V().next() }
If you're using Apache TinkerPop 3.1.2 or later, you would have seen a more informative stack trace in the Gremlin Server logs:
[WARN] HttpGremlinEndpointHandler - Invalid request - responding with 500 Internal Server Error and Error encountered evaluating script: g.V().next()
org.apache.tinkerpop.gremlin.process.traversal.util.FastNoSuchElementException
In the dashlet "My tasks" there are two items: "Current tasks" and "Completed tasks".
When I click on the "Completed tasks" I see the following error on a red background:
Error while loading items
When this error occurs in the logs I see the following.
catalina.out:
...
Caused by: org.activiti.engine.ActivitiObjectNotFoundException: no deployed process definition found with id 'publishWhitepaper:1:1115'
at org.activiti.engine.impl.persistence.deploy.DeploymentManager.findDeployedProcessDefinitionById(DeploymentManager.java:75)
at org.activiti.engine.impl.cmd.GetDeploymentProcessDefinitionCmd.execute(GetDeploymentProcessDefinitionCmd.java:39)
at org.activiti.engine.impl.cmd.GetDeploymentProcessDefinitionCmd.execute(GetDeploymentProcessDefinitionCmd.java:26)
at org.activiti.engine.impl.interceptor.CommandInvoker.execute(CommandInvoker.java:24)
at org.activiti.engine.impl.interceptor.CommandContextInterceptor.execute(CommandContextInterceptor.java:57)
at org.activiti.spring.SpringTransactionInterceptor$1.doInTransaction(SpringTransactionInterceptor.java:47)
at org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:131)
at org.activiti.spring.SpringTransactionInterceptor.execute(SpringTransactionInterceptor.java:45)
at org.activiti.engine.impl.interceptor.LogInterceptor.execute(LogInterceptor.java:31)
at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:40)
at org.activiti.engine.impl.cfg.CommandExecutorImpl.execute(CommandExecutorImpl.java:35)
at org.activiti.engine.impl.RepositoryServiceImpl.getDeployedProcessDefinition(RepositoryServiceImpl.java:138)
at org.alfresco.repo.workflow.activiti.ActivitiUtil.getDeployedProcessDefinition(ActivitiUtil.java:133)
at org.alfresco.repo.workflow.activiti.ActivitiTypeConverter.getTaskDefinition(ActivitiTypeConverter.java:223)
at org.alfresco.service.cmr.workflow.LazyActivitiWorkflowTask.<init>(LazyActivitiWorkflowTask.java:93)
at org.alfresco.repo.workflow.activiti.ActivitiWorkflowEngine.getAssignedTasks(ActivitiWorkflowEngine.java:1543)
... 92 more
Before that I installed and watched some examples of business processes, but then deleted them (and via workflow console). - most likely, I didn't do it correctly...
I can't understand why this error appear?..
no deployed process definition found with id
'publishWhitepaper:1:1115'
Maybe somewhere something is cached?
Axel Faust gave an exhaustive answer:
Is there enough functionality of the workflow admin console?
Now I understand the cause of the error: as Axel Faust said, "..the tables for historic information do require referential integrity in their relation to the process definition and are not automatically cascade-deleted when you undeploy a process."
Thanks to all for the assistance!
put this configuration in application.yml of springboot. basically it couldn't find your .bpmn file. just pointing to the correct location would solve this issue
spring:
activiti:
database-schema-update: true
db-history-used: true
check-process-definitions: true
process-definition-location-prefix: file:/opt/try-uploads/
# process-definition-location-prefix: classpath:/processes/
process-definition-location-suffixes: '*.bpmn, *.bpmn20.xml'
history-level: audit