Strace can be used for tracing process by passing command for the process as below
strace -f -tt -o strace.log -D <SOME_COMMAND>
But below command fails to trace the syscalls of started daemon process
strace -f -tt -o strace.log -D service nginx start
In this case the strace just traces syscall for /usr/sbin/service and terminates. It does not trace syscalls on nginx process which is started as result of service nginx start
How do a I trace the process started by /usr/sbin/service? Specifically looking for solution with daemon process only!
Instead of running the nginx from service. Run service nginx stop and then run
strace nginx -g "daemon off;"
this will make sure that you get the trace of the process. The -g "daemon off;" will make sure the nginx is not run as a daemon process, else again the strace would end
Service command is just activating a process and if you want to strace it the best is to launch the process directly.
In case you are still interested in debugging the process started using the service command. Then do below
service nginx start
ps aux | grep nginx
Capture the pid from the nginx process and then attach to it using
strace -p <pid>
Forking Processes
To trace processes which fork, you need to use the -f flag
strace -f nginx
Service tracing
When you call service start nginx, assuming the system uses systemd, the call gets translated to systemctl start nginx. Now if you look at the source code of systemd
https://github.com/systemd/systemd/blob/cf45dd36282368d5cdf757cac2cf1fc2b562dab2/src/systemctl/systemctl.c#L3100
r = sd_bus_call_method_async(
bus,
NULL,
"org.freedesktop.systemd1",
"/org/freedesktop/systemd1",
"org.freedesktop.systemd1.Manager",
"Subscribe",
NULL, NULL,
NULL);
It doesn't spawn/fork the process. It sends the message to the systemd service which then starts nginx process.
So in short, NO you can't strace through your service nginx start command.
Change the ExecStart property of the service to include "strace". For example (tested on Debian Buster):
# grep ExecStart /lib/systemd/system/nginx.service
ExecStartPre=/usr/sbin/nginx -t -q -g 'daemon on; master_process on;'
ExecStart=/usr/sbin/nginx -g 'daemon on; master_process on;'
# cd /etc/systemd/system/
# mkdir nginx.service.d
# cat > nginx.service.d/strace.conf <<-EOD
[Service]
ExecStart=
ExecStart=/usr/bin/strace -f -tt -o /tmp/strace.log -D /usr/sbin/nginx -g 'daemon on; master_process on;'
EOD
# systemctl daemon-reload
# systemctl restart nginx.service
Related
I have a server machine that has RHL6 (Red Hat Linux 6) and is based on SysV initialization (does not have systemd package), and I want to make my prometheus node exporter collect metrics from this machine.
All I can find online is how to create a node exporter service with systemctl (systemd): basically you create a .service file under /etc/systemd/system and then write something like this in it:
[Unit]
Description=Node Exporter
After=network.target
[Service]
User=node_exporter
Group=node_exporter
Type=simple
ExecStart=/usr/local/bin/node_exporter
[Install]
WantedBy=multi-user.target
And then you start the service, enable it at startup, etc with systemctl command like this
sudo systemctl start node_exporter
sudo systemctl status node_exporter
sudo systemctl enable node_exporter
But the problem is that I don't have systemd installed and I don't have the right to update the server machine system so I am trying to find a way how to write an init script for node exporter to be placed under /etc/rd.d/init.d in my case.
It seems that all scripts under init.d are shell scripts that declare many methods like start(), stop(), restart(), reload(), force_reload(), ...
So it's not as easy as writing the service based on systemd.
Anyone have an idea how to do that with SysV init ???
Thanks,
I managed to found a solution for my problem.
Here is how the script looks like:
#!/bin/bash
#
# chkconfig: 2345 90 12
# description: node-exporter server
#
# Get function from functions library
. /etc/init.d/functions
# Start the service node-exporter
start() {
echo -n "Starting node-exporter service: "
/usr/sbin/node_exporter_service &
### Create the lock file ###
touch /var/lock/subsys/node-exporter
success $"node-exporter service startup"
echo
}
# Restart the service node-exporter
stop() {
echo -n "Shutting down node-exporter service: "
killproc node_exporter_service
### Now, delete the lock file ###
rm -f /var/lock/subsys/node-exporter
echo
}
### main logic ###
case "$1" in
start)
start
;;
stop)
stop
;;
status)
status node_exporter_service
;;
restart|reload)
stop
start
;;
*)
echo $"Usage: $0 {start|stop|restart|reload|status}"
exit 1
esac
exit 0
We place the above script under /etc/init.d named "node-exporter" (without .sh) and we place the binary for the node exporter under /usr/sbin (with systemd we place binaries under /usr/local/bin).
You can download the binary file for node exporter from here https://github.com/prometheus/node_exporter/releases.
Then we add the script file to the list of services with command chkconfig --add node-exporter (to check if it already exists use command chkconfig --list node-exporter).
Enable the service with command chkconfig node-exporter on.
And then to start/stop/restart ... we use command /etc/init.d/node-exporter start/stop/restart ....
In the start script we basically run the binary file and in the stop script we kill the process by its name.
I hope this will be useful.
I know that Zabbix can monitor any service on Linux machine via two options:
scan particular tcp or udp port, on which the service is bound
or count the service processes with proc.num[<processname>]
It is totally counter-intuitive, because I can spawn processes with the same executable name and they will deceive Zabbix. I'd prefer to use standard service <servicename> status or systemctl status name.service tool. But there are no standard way to use it from Zabbix except system.run[cmd]
Could you help me to write templates for monitoring a particular service state. We want to use different OSes like Centos 7 and Ubuntu 14.04 and 16.04 distributions. It is pity but service <servicename> status is completely different in listed operating systems.
You can also add the following UserParameters in zabbix_agentd.conf to monitor service status in systemd systems. For non-systemd the OS doesn't really monitor service status, the various bash script "status" arguments are often unreliable.
UserParameter=systemd.unit.is-active[*],systemctl is-active --quiet '$1' && echo 1 || echo 0
UserParameter=systemd.unit.is-failed[*],systemctl is-failed --quiet '$1' && echo 1 || echo 0
UserParameter=systemd.unit.is-enabled[*],systemctl is-enabled --quiet '$1' && echo 1 || echo 0
And then e.g. for sshd status create an item with a key like:
systemd.unit.is-active[sshd]
If Linux services are managed by systemd (Centos 7+, Ubuntu 16+, ...), then you can use https://github.com/cavaliercoder/zabbix-module-systemd. It uses standard systemd D-Bus communication - that's what systemctl does under the hood.
For centos 6 it can be done:
UserParameter=check_service_status_asterisk,sudo service asterisk status 2> /dev/null | grep -q "is running";echo $?
For centos 7 or similar it can be created with:
UserParameter=check_service_status_grafana,systemctl status grafana-server 2> /dev/null |sed -n 3p |grep -q "running";echo $?
or
UserParameter=check_service_status[*],systemctl status $1 2> /dev/null |sed -n 3p |grep -q "running";echo $?
I'm trying to run 3 services at my container startup (snmpd, sshd and centengine)
As runlevel is unknown in the container, services won't start.
I built an image with this Dockerfile :
FROM centos:6.7
MAINTAINER nael <me#mail>
# Update CentOS
RUN yum -y update
# Install wget
RUN yum install -y wget
# Get Centreon Repo
RUN wget http://yum.centreon.com/standard/3.0/stable/ces-standard.repo -O /etc/yum.repos.d/ces-standard.repo
# Install Packages (SSH, sudo, Centreon Poller & Engine, SNMP)
RUN yum install -y --nogpgcheck openssh-clients openssh-server centreon-poller-centreon-engine sudo net-snmp net-snmp-utils
# Install supervisord
RUN rpm -Uvh http://dl.fedoraproject.org/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm
RUN yum --enablerepo=epel install -y supervisor
RUN mv -f /etc/supervisord.conf /etc/supervisord.conf.org
ADD supervisord.conf /etc/
# For sshd & centengine
EXPOSE 22 5669
# Change user password
RUN echo -e "password" | (passwd --stdin user)
# Disable PAM (causing issues while ssh login)
RUN sed -ri 's/UsePAM yes/#UsePAM yes/g' /etc/ssh/sshd_config
RUN sed -ri 's/#UsePAM no/UsePAM no/g' /etc/ssh/sshd_config
# Start supervisord
CMD ["/usr/bin/supervisord"]
Here is the supervisord.conf file
[supervisord]
nodaemon=true
pidfile=/var/run/supervisord.pid
logfile=/var/log/supervisor/supervisord.log
[program:centengine]
command=service centengine start
[program:snmpd]
command=service snmpd start
[program:sshd]
command=service sshd start
But with this Dockerfile and supervisord.conf, when I start my container theses services aren't running.
What could be the problem ?
Not anymore using supervisord.
I just include a script with all the services ... start commands in the Dockerfile. When I create my container with docker run ... I just specify that I want to start it with my script.
& that's working very well.
Thanks #warmoverflow for trying to solve this.
You may find my dockerfy utility useful starting services, pre-running initialization commands before the primary command starts. See https://github.com/markriggins/dockerfy
For example:
RUN wget https://github.com/markriggins/dockerfy/releases/download/0.2.4/dockerfy-linux-amd64-0.2.4.tar.gz; \
tar -C /usr/local/bin -xvzf dockerfy-linux-amd64-*tar.gz; \
rm dockerfy-linux-amd64-*tar.gz;
ENTRYPOINT dockerfy
COMMAND --start bash -c "while false; do echo 'Ima Service'; sleep 1; done" -- \
--reap -- \
nginx
Would run a bash script as a service, echoing "Ima Service" every second, while the primary command nginx runs. If nginx exits, then the "Ima Service" script will automatically be stopped.
As an added benefit, any zombie processes left over by nginx will be automatically cleaned up.
You can also tail log files such as /var/log/nginx/error.log to stderr, edit nginx's configuration prior to startup and much more
I'm using runit to manage an HAProxy and want to do a safe restart to reload a configuration file (specifically: haproxy -f /etc/haproxy/haproxy.cfg -sf $OLD_PROCESS_ID). I figure that I could run sv restart haproxy and tried to add a custom script named /etc/service/haproxy/restart, but it never seems to execute. How do I have a special restart script? Is my approach even good here? How do I reload my config with minimal impact using runit?
HAProxy runit service script
/etc/service/haproxy/run
#!/bin/sh
#
# runit haproxy
#
# forward stderr to stdout for use with runit svlogd
exec 2>&1
PID_PATH=/var/run/haproxy/haproxy.pid
BIN_PATH=/opt/haproxy/sbin/haproxy
CFG_PATH=/opt/haproxy/etc/haproxy.cfg
exec /bin/bash <<EOF
$BIN_PATH -f $CFG_PATH -D -p $PID_PATH
trap "echo SIGHUP caught; $BIN_PATH -f $CFG_PATH -D -p $PID_PATH -sf \\\$(cat $PID_PATH)" SIGHUP
trap "echo SIGTERM caught; kill -TERM \\\$(cat $PID_PATH) && exit 0" SIGTERM SIGINT
while true; do # Iterate to keep job running.
sleep 1 # Wake up to handle signals
done
EOF
Graceful reload that keeps things up and running.
sv reload haproxy
Full stop and start.
sv restart haproxy
This solution was inspired by https://gist.github.com/gfrey/8472007
Morning,
I have monit I am testing on redhat 6.4 system. I setup a /etc/init/monit.conf:
description "Monit service manager"
limit core unlimited unlimited
start on runlevel [2345]
stop on runlevel [!2345]
expect daemon
respawn
exec /local/mis/monit/bin/monit -c /local/mis/monit/etc/monitrc
pre-stop exec /local/mis/monit/bin/monit -c /local/mis/monit/etc/monitrc quit
At the command line as root I can run stop monit and start monit just fine and it shows pid number. However, during reboot, it does not start. It shows start/running with no pid if I run initctl list, but if you check with ps -ef monit is not running. I can run stop monit and then run start monit just fine after a reboot. I am at a lose with how to troubleshoot. My system has /var/log/messages, but no /var/log/syslog. I see options to use log-priority info, but I am not sure how to set that as the level for logging during the reboot. The /var/log/message does not mention monit and /var/log/boot.log does not either. dmesg shows nothing.
this sounds like the same problem we just fixed.
We run monit as user "monit". upstart was trying to start it as root and the monit files where owned my user "monit". And we were getting identical symptoms you were getting.
To fix it i altered /etc/init/monit to
exec su -c "/web/bin/monit -c /web/etc/monitrc" monit
now when i start monitI see::
# start monit
monit start/running, process 3421
The final solution I ended up using: description "Monit service manager"
start on (net-device-up IFACE=eth0 and started networking and runlevel [2345])
stop on runlevel [!2345]
limit core unlimited unlimited
expect daemon
respawn
Had to do a pre-start script to loop until successful with nslookup of mail server listed in the monitrc file.
pre-start script
while [ 0 ]; do
i=/usr/bin/nslookup outlookwebapp.na.sas.com | grep Name
if [ ! -z "$i" ]; then
break
fi
sleep 4
done
end script
exec /local/mis/monit/bin/monit -c /local/mis/monit/etc/monitrc
pre-stop exec /local/mis/monit/bin/monit -c /local/mis/monit/etc/monitrc quit