according to
https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_cloudwatch_logs.html
I can install awslogs agent to send logs to CloudWatch and monitor common files, but it seems to be missing the instructions to send logs from the stdout & err of my app. I don't want to write to disk and risk filling up its capacity. I'm using KubeOne + Terraform and I don't want to touch on EKS.
Also, how I can make sure system logs wa
[/var/log/dmesg]
file = /var/log/dmesg
log_group_name = /var/log/dmesg
log_stream_name = {cluster}/{container_instance_id}
Related
I have created an init script that helps me in getting custom logs in databricks , By default log get created at local (Driver/ worker machine ) path log/log4j-active.log but how can I enable to ship it to DBFS or storage. ???`
%sh
ls logs
getting below output
lineage.json
log4j-active.log
log4j-mylog4j-active.log
metrics.json
product.json
stderr
stdout
ttyd_logs
usage.json
i want to copy my log file log4j-mylog4j-active.log to dbfs or blob storage anything would work ..
dbutils.fs.cp("logs/log4j-mylog4j-active.log", "dbfs:/cluster-logs/")
I am also trying filesystem copy but can't do
FileNotFoundException: /logs/log4j-active.log
I have also tried to create a folder and specify the path in the logging ( in cluster advance option)
but that also didn't work , i don't know why my fs logs are not getting ship to that location of dbfs.
can i get help that how can I transfer my fs log to dbfs or storage ??
thanks in advance !!
You just need to enable logging in your cluster configuration (unfold "Advanced options") & specify where logs should go - by default it's a dbfs:/cluster-logs/ (and cluster ID will be appended to it), but you can specify another path.
I have an app using Rails 2.5.1 which is deployed on Google Cloud Kubernetes but the logs are not working good.The FATAL errors are displayed as INFO logs instead of ERROR or CRITICAL.
Extra context:
Rails 5.2.1
This view is from Logs Viewer
I'm using RAILS_LOG_TO_STDOUT=true. If I remove this, logs are not displayed at all
Thanks.
[UPDATE]
I updated my cluster to use the newest Stackdriver API but it stills not working and Rails.logger.error is displayed as INFO.
I got this working changing the log format to use JSON.
config.log_formatter = proc do |severity, datetime, progname, msg|
message = msg
message << " from #{progname}" if progname.present?
content = JSON.dump(timestamp: datetime.to_s, severity: severity, message: message)
content << "\n"
content
end
Have a look at the documentation Cloud Logging for Legacy Logging and Monitoring section Best practices:
Severities: By default, logs written to the standard output are on the
INFO level and logs written to the standard error are on the ERROR
level. Structured logs can include a severity field, which defines the
log's severity.
and because you're using RAILS_LOG_TO_STDOUT=true log events generated by Ruby you can see with severity INFO.
Keep in mind that you should migrate to Kubernetes Engine Monitoring:
Warning: Legacy Logging and Monitoring support for Google Kubernetes
Engine is deprecated. If you are using Legacy Logging and Monitoring,
then you must migrate to Kubernetes Engine Monitoring before support
for Legacy Logging and Monitoring is removed.
it's better to return to this "issue" after migration.
EDIT Have a look at the documentation Writing Logs section Writing log entries where you can find an example for Ruby:
Here is some sample code to write a single log entry to mylog. The
service, region, labels, and other content will change depending on
the entry and the application doing the writing.
require "google/cloud/logging"
logging = Google::Cloud::Logging.new project: "my-gcp-project-id"
entry = logging.entry entry.log_name = "my_application_log"
entry.payload = "Log message"
entry.severity = :NOTICE
entry.resource.type = "gae_app"
entry.resource.labels[:module_id] = "default"
entry.resource.labels[:version_id] = "20160101t163030"
logging.write_entries entry
I wish to better understand the way the Log4cplus SocketAppender works with regard to the logging server that recieves this appender events.
I read the Log4cplus src code for loggingserver and socketappender and I will be glad to be clarified:
Can the SocketAppender only send events to the Log4cplus logging server, and not to any other server?
and if this is the case: does it mean that if I want to send log messages to remote machine, that machine must be installed with the Log4cplus lib?
I would also like to know- does this Log4cplus logging-server run as a service? and does it require special configuration and pre-setup in order to use it?
Can the SocketAppender only send events to the Log4cplus logging server, and not to any other server?
Yes and yes.
does it mean that if I want to send log messages to remote machine, that machine must be installed with the Log4cplus lib?
Well, sort of. If you want to use only SocketAppender, you will have to use the logging server. You could also use SysLogAppender and send to remote server using that. Obviously, you have to have syslog service and allow receiving from network in it. You could also write your own custom appender that sends the events to whatever server you desire.
I would also like to know- does this Log4cplus logging-server run as a service?
No, it is a simple executable that listens on a socket.
and does it require special configuration and pre-setup in order to use it?
It requires configuration file so that it knows where to log the events.
I just wanted to share how I used SocketAppender (this setup also works for docker containers being in the same network).
/usr/share/elasticsearch/config/log4j2.properties
status = error
appender.console.type = Console
appender.console.name = console
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%m%n
appender.socket.type=Socket
appender.socket.name=socket
appender.socket.port=601
appender.socket.host=api
appender.socket.reconnectDelayMillis=10000
appender.socket.layout.type = PatternLayout
appender.socket.layout.pattern = [%d{ISO8601}][%-5p][%-25c{1.}] %marker%m%n
rootLogger.level = info
rootLogger.appenderRef.console.ref = console
rootLogger.appenderRef.socket.ref = socket
in the second container I used syslog-ng:
apk add syslog-ng
vi /etc/syslog-ng/syslog-ng.conf
syslog-ng -f /etc/syslog-ng/syslog-ng.conf
/etc/syslog-ng/syslog-ng.conf
#version: 3.13
source s_network {
network(
transport(tcp)
port(601)
);
};
log {
source(s_network);
destination(d_network);
};
destination d_network {
file("/var/log/es_slowlog.log", template("${MSGHDR}${MESSAGE}\n"));
};
Notice that the #version: has to correspond to your version of syslog-ng. You can check it by invoking syslog-ng -V.
In Bluemix Blockchain service, where does the chaincode output go when I do
fmt.Printf("X: %v", len(someSlice))
in a deployed go chaincode?
I tried looking at the peer logs & chaincode logs on all peers, but I don't see X
As independently executed programs, user-provided chaincodes can use any appropriate technique to create their private logs - from simple print statements to fully-annotated and level-controlled logs.
You should be able to see the logs by clicking the logs button for the hash id that is associated with the chaincode you deployed.
Code:
Response after posting chaincode:
Getting logs of chaincode id:
My print statement in the log:
Thanks to #joe for pointing me in the right direction. It seems my fmt.Printf(..) statements did NOT include a newline in the string, so another log entry from a different output stream overwrote my stdout in the logs. Remember to add a \n in the fmt.Println() statements!
This problem did force me to set up logging according to the logging control section of the docs, which takes care of the newline problem.
I try to execute this example script (https://oss.trac.surfsara.nl/pbs_python/wiki/TorqueUsage/Scripts/Submit)
#!/usr/bin/env python
import sys
sys.path.append('/usr/local/build_pbs/lib/python2.7/site-packages/pbs/')
import pbs
server_name = pbs.pbs_default()
c = pbs.pbs_connect(server_name)
attropl = pbs.new_attropl(4)
# Set the name of the job
#
attropl[0].name = pbs.ATTR_N
attropl[0].value = "test"
# Job is Rerunable
#
attropl[1].name = pbs.ATTR_r
attropl[1].value = 'y'
# Walltime
#
attropl[2].name = pbs.ATTR_l
attropl[2].resource = 'walltime'
attropl[2].value = '400'
# Nodes
#
attropl[3].name = pbs.ATTR_l
attropl[3].resource = 'nodes'
attropl[3].value = '1:ppn=4'
# A1.tsk is the job script filename
#
job_id = pbs.pbs_submit(c, attropl, "A1.tsk", 'batch', 'NULL')
e, e_txt = pbs.error()
if e:
print e,e_txt
print job_id
But shell shows error "15025 Queue already exists". With qsub job submits normally. I have one queue 'batch' on my server. Torque version - 4.2.7. Pbs_python version - 4.4.0.
What I should to do to start new job?
There are two things going on here. First there is an error in pbs_python that maps the 15025 error code to "Queue already exists". Looking at the source of torque we see that 15025 actually maps to the error "Bad UID for job execution", this means that on the torque server, the daemon cannot determine if the user you are submitting as is allowed to run jobs. This could be because of several things:
The user you are submitting as doesn't exist on the machine running pbs_server
The host you are submitting from is not in the "submit_hosts" parameter of the pbs_server.
Solution For 1
The remedy for this depends on how you authenticate users across systems, you could use /etc/hosts.equiv to specify users/hosts allowed to submit, this file would need to be distributed to all the torque nodes as well as the torque server machine. Using hosts.equiv is pretty insecure, I haven't actually used it in this. We use a central LDAP server to authenticate all users on the network and do not have this problem. You could also manually add the user to all the torque nodes and the torque server, taking care to make sure the UID is the same on all systems.
Solution For 2
If #1 is not your problem (which I doubt it is), you probably need to add the hostname of the machine you're submitting from to the "submit_hosts" parameter on the torque server. This can be accomplished with qmgr:
[root#torque_server ]# qmgr -c "set server submit_hosts += hostname.example.com"
The pbs python library that you are using was written for torque 2.4.x.
The internal api's for torque were largely rewritten in torque 4.0.x. The library will most likely need to be written for thew new API.
Currently the developers of torque do not test any external libraries. It is possible that they could break at any time.