Varnish DAEMON_OPTS Options Errors - daemon

When using inline C with Varnish I've not been able to get /etc/varnish/default
to be happy at startup.
I've tested inline C with varnish for two things: GeoIP detection and Anti-Site-Scraping functions.
The DAEMON_OPTS always complains even though I'm following what other seem
to indicate works fine.
My problem is that this command line start up works:
varnishd -f /etc/varnish/varnish-default.conf -s file,/var/lib/varnish/varnish_storage.bin,512M -T 127.0.0.1:2000 -a 0.0.0.0:8080 -p 'cc_command=exec cc -fpic -shared -Wl,-x -L/usr/include/libmemcached/memcached.h -lmemcached -o %o %s'
But it errors out with trying to start up from default start scripts:
/etc/default/varnish has this in it:
DAEMON_OPTS="-a :8080 \
-T localhost:2000 \
-f /etc/varnish/varnish-default.conf \
-s file,/var/lib/varnish/varnish_storage.bin,512M \
-p 'cc_command=exec cc -fpic -shared -Wl,-x -L/usr/include/libmemcached/memcached.h -lmemcached -o %o %s'"
The error is:
# /etc/init.d/varnish start
Starting HTTP accelerator: varnishd failed!
storage_file: filename: /var/lib/varnish/vbox.local/varnish_storage.bin size 512 MB.
Error:
Unknown parameter "'cc_command".
If I try change the last line to:
-p cc_command='exec cc -fpic -shared -Wl,-x -L/usr/include/libmemcached/memcached.h -lmemcached -o %o %s'"
It's error is now:
# /etc/init.d/varnish start
Starting HTTP accelerator: varnishd failed!
storage_file: filename: /var/lib/varnish/vbox.local/varnish_storage.bin size 512 MB.
Error: Unknown storage method "hared"
It's trying to interpret the '-shared' as -s hared and 'hared' is not a storage type.
For both GeoIP and the Anti-Site-Scrape I've used the exact recommended daemon options
plus have tried all sorts of variations like adding ' and '' but no joy.
Here is a link to the instruction I've followed that work fine except the DAEMON_OPTS part.
http://drcarter.info/2010/04/how-fighting-against-scraping-using-varnish-vcl-inline-c-memcached/
I'm using Debian and the exact DAEMON_OPTS as stated in the instructions.
Can anyone help with a pointer on what's going wrong here?

Even if Jacob will probably never read this, visitors from the future might appreciate what I'm going to write.
I believe I know what's wrong, and it looks like a Debian-specific problem, at least verified on Ubuntu 11.04 and Debian Squeeze.
I traced the execution from my /etc/default/varnish that contains the $DAEMON_OPTS to the init script.
In the init script /etc/init.d/varnish, the start_varnishd() function is:
start_varnishd() {
log_daemon_msg "Starting $DESC" "$NAME"
output=$(/bin/tempfile -s.varnish)
if start-stop-daemon \
--start --quiet --pidfile ${PIDFILE} --exec ${DAEMON} -- \
-P ${PIDFILE} ${DAEMON_OPTS} > ${output} 2>&1; then
log_end_msg 0
else
log_end_msg 1
cat $output
exit 1
fi
rm $output
}
So I modified it to print the full start-stop-daemon command line, like:
start_varnishd() {
log_daemon_msg "Starting $DESC" "$NAME"
output=$(/bin/tempfile -s.varnish)
+ echo "start-stop-daemon --start --quiet --pidfile ${PIDFILE} --exec ${DAEMON} -- -P ${PIDFILE} ${DAEMON_OPTS} > ${output} 2>&1"
if start-stop-daemon \
--start --quiet --pidfile ${PIDFILE} --exec ${DAEMON} -- \
-P ${PIDFILE} ${DAEMON_OPTS} > ${output} 2>&1; then
log_end_msg 0
So I got a command line echoed on STDOUT, and copied-pasted it into my shell. And, surprise! It worked. WTF?
Repeated again to be sure. Yes, it works. Mmh. Could it be another of those bash/dash corner cases?
Let's try feeding the start-stop-daemon command line to bash, and see how it reacts:
start_varnishd() {
log_daemon_msg "Starting $DESC" "$NAME"
output=$(/bin/tempfile -s.varnish)
if bash -c "start-stop-daemon \
--start --quiet --pidfile ${PIDFILE} --exec ${DAEMON} -- \
-P ${PIDFILE} ${DAEMON_OPTS} > ${output} 2>&1"; then
log_end_msg 0
else
log_end_msg 1
cat $output
exit 1
fi
rm $output
}
Yes, it works just fine, at least for my case.
Here's the relevant part of my /etc/default/varnish:
...
## Alternative 2, Configuration with VCL
#
# Listen on port 6081, administration on localhost:6082, and forward to
# one content server selected by the vcl file, based on the request. Use a 1GB
# fixed-size cache file.
#
DAEMON_OPTS="-a :6081 \
-T localhost:6082 \
-f /etc/varnish/geoip-example.vcl \
-S /etc/varnish/secret \
-s malloc,100M \
-p 'cc_command=exec cc -fpic -shared -Wl,-x -L/usr/include/GeoIP.h -lGeoIP -o %o %s'"
...
I've seen posts where someone tried to work around this problem by moving the compile command into a separated shell script. Unfortunately that doesn't change the fact that start-stop-daemon is going to pass the $DAEMON_OPTS var through dash, and that will result in mangled options.
Would be something along the lines of:
-p 'cc_command=exec /etc/varnish/compile.sh %o %s'"
And then the compile.sh script as:
#!/bin/sh
cc -fpic -shared -Wl,-x -L/usr/include/GeoIP.h -lGeoIP -o $#
but it doesn't work, so just patch your init scripts, and you're good to go!
Hope you can find this information useful.

You can try using :-
DAEMON_OPTS="-a :8080 \
-T localhost:2000 \
-f /etc/varnish/varnish-default.conf \
-s file,/var/lib/varnish/varnish_storage.bin,512M \
-p cc_command='exec cc -fpic -shared -Wl,-x -L/usr/include/libmemcached/memcached.h -lmemcached -o %o %s'"

Obviously, your startup script interpreting the DAEMON_OPTS is not prepared for whitespace (even within single quotes). At my Fedora (15) installation, the suggested solution works fine; the arguments get interpreted correctly because the "$*" bash parameter is passed in /etc/init.d/varnish and in /etc/init.d/functions in daemon().
Did you get your startup scripts from a package or did you make custom scripts?

This isn't directly related to the question, but you may find yourself here if you are working through the Varnish Tutorial - Put Varnish on port 80.
For recent installs of Varnish on Debian systems the configuration for varnishd startup options can be found in /etc/systemd/system/multi-user.target.wants/varnish.service. The documented way of changing the port via /etc/default/varnish still exists, but is no longer functional unless you change your system to use init scripts rather than systemd.
After you've changed your options in /etc/systemd/system/multi-user.target.wants/varnish.service, don't forget to run systemctl daemon-reload, which will catalog the changes for executing the program.

Related

Centos 7 sudo -u <user> mkdir -p <path> stopped working

I have an odd issue where as of recently I am unable to run mkdir -p for a different user running as root and get the following error:
[root#ip-192-168-1-146 ~]# sudo -u myuser mkdir -p /some/target/path
sudo: mkdir -p: command not found
When I test as the 'myuser' user the mkdir -p command works fine. I have additionally tried the following without success:
su - myuser -c "mkdir -p /some/target/path"
sudo -u myuser -i mkdir -p /some/target/path
sudo -u myuser -i -c "mkdir -p /some/target/path" <---dont think syntax is right on this but tried anyways.
Context: I am executing a script to setup my AWS EC2 instance that populates all defined directories. This has been working fine until recently. Not including my script here as the above command doesn't work by itself.
Env output for 'mysuser':
$ env
XDG_SESSION_ID=1
HOSTNAME=ip-192-168-1-146.ec2.internal
SHELL=/bin/bash
TERM=xterm-256color
HISTSIZE=1000
USER=myuser
LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
MAIL=/var/spool/mail/myuser
PATH=/sbin:/bin:/opt/home/myuser/.local/bin:/opt/home/myuser/bin
PWD=/opt/home/myuser
LANG=en_US.UTF-8
HISTCONTROL=ignoredups
SHLVL=1
HOME=/opt/home/myuser
LOGNAME=myuser
LESSOPEN=||/usr/bin/lesspipe.sh %s
_=/bin/env
Env output for 'root':
$ env
XDG_SESSION_ID=1
HOSTNAME=ip-192-168-1-146.ec2.internal
SHELL=/bin/bash
TERM=xterm-256color
HISTSIZE=1000
USER=root
LS_COLORS=rs=0:di=38;5;27:ln=38;5;51:mh=44;38;5;15:pi=40;38;5;11:so=38;5;13:do=38;5;5:bd=48;5;232;38;5;11:cd=48;5;232;38;5;3:or=48;5;232;38;5;9:mi=05;48;5;232;38;5;15:su=48;5;196;38;5;15:sg=48;5;11;38;5;16:ca=48;5;196;38;5;226:tw=48;5;10;38;5;16:ow=48;5;10;38;5;21:st=48;5;21;38;5;15:ex=38;5;34:*.tar=38;5;9:*.tgz=38;5;9:*.arc=38;5;9:*.arj=38;5;9:*.taz=38;5;9:*.lha=38;5;9:*.lz4=38;5;9:*.lzh=38;5;9:*.lzma=38;5;9:*.tlz=38;5;9:*.txz=38;5;9:*.tzo=38;5;9:*.t7z=38;5;9:*.zip=38;5;9:*.z=38;5;9:*.Z=38;5;9:*.dz=38;5;9:*.gz=38;5;9:*.lrz=38;5;9:*.lz=38;5;9:*.lzo=38;5;9:*.xz=38;5;9:*.bz2=38;5;9:*.bz=38;5;9:*.tbz=38;5;9:*.tbz2=38;5;9:*.tz=38;5;9:*.deb=38;5;9:*.rpm=38;5;9:*.jar=38;5;9:*.war=38;5;9:*.ear=38;5;9:*.sar=38;5;9:*.rar=38;5;9:*.alz=38;5;9:*.ace=38;5;9:*.zoo=38;5;9:*.cpio=38;5;9:*.7z=38;5;9:*.rz=38;5;9:*.cab=38;5;9:*.jpg=38;5;13:*.jpeg=38;5;13:*.gif=38;5;13:*.bmp=38;5;13:*.pbm=38;5;13:*.pgm=38;5;13:*.ppm=38;5;13:*.tga=38;5;13:*.xbm=38;5;13:*.xpm=38;5;13:*.tif=38;5;13:*.tiff=38;5;13:*.png=38;5;13:*.svg=38;5;13:*.svgz=38;5;13:*.mng=38;5;13:*.pcx=38;5;13:*.mov=38;5;13:*.mpg=38;5;13:*.mpeg=38;5;13:*.m2v=38;5;13:*.mkv=38;5;13:*.webm=38;5;13:*.ogm=38;5;13:*.mp4=38;5;13:*.m4v=38;5;13:*.mp4v=38;5;13:*.vob=38;5;13:*.qt=38;5;13:*.nuv=38;5;13:*.wmv=38;5;13:*.asf=38;5;13:*.rm=38;5;13:*.rmvb=38;5;13:*.flc=38;5;13:*.avi=38;5;13:*.fli=38;5;13:*.flv=38;5;13:*.gl=38;5;13:*.dl=38;5;13:*.xcf=38;5;13:*.xwd=38;5;13:*.yuv=38;5;13:*.cgm=38;5;13:*.emf=38;5;13:*.axv=38;5;13:*.anx=38;5;13:*.ogv=38;5;13:*.ogx=38;5;13:*.aac=38;5;45:*.au=38;5;45:*.flac=38;5;45:*.mid=38;5;45:*.midi=38;5;45:*.mka=38;5;45:*.mp3=38;5;45:*.mpc=38;5;45:*.ogg=38;5;45:*.ra=38;5;45:*.wav=38;5;45:*.axa=38;5;45:*.oga=38;5;45:*.spx=38;5;45:*.xspf=38;5;45:
MAIL=/var/spool/mail/root
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
PWD=/root
LANG=en_US.UTF-8
HISTCONTROL=ignoredups
SHLVL=1
HOME=/root
LOGNAME=root
LESSOPEN=||/usr/bin/lesspipe.sh %s
_=/bin/env
mkdir is on the system:
[root#ip-192-168-1-146 ~]# ls /bin/mkdir
/bin/mkdir
[root#ip-192-168-1-146 ~]# which mkdir
/bin/mkdir
[myuser#ip-192-168-1-146 ~]$ which mkdir
/bin/mkdir
I am at a loss, if anyone has any suggestions I'ld be greatful. Again this has been working for few months now.
Thanks!
so...no idea how it happened but somehow I pasted a special character M-BM- in place of a proper space char.
The M-BM- characters are an ASCII representation of byte sequence 0xc2 0xa0, which is the UTF8 encoding of unicode character A0 - a non-breaking space character. This character can be inserted in both LibreOffice and Microsoft Word documents using the key sequence Ctrl+Shift+SPACE.
¯\(ツ)/¯

lxc option "--" when calling lxc-start / lxc-create

What is the significance of -- in the command line of commands like lxc-create or lxc-start.
I tried to use Google in order to get an answer but without success.
// Example 1
lxc-create -t download -n u1 -- -d ubuntu -r DISTRO-SHORT-CODENAME -a amd64
// Example 1
application="/root/app.out"
start="/root/lxc-app/lxc-start"
$start -n LXC_app -d -f /etc/lxc/lxc-app/lxc-app.conf -- $application &
As explained in the references provided in the comments, the "--" indicates the end of the options passed to the command. The following parameters/options will be eventually used by a sub-command called by the command.
In your example:
lxc-create -t download -n u1 -- -d ubuntu -r DISTRO-SHORT-CODENAME -a amd64
lxc-create command will interpret "-t download -n u1" and the remaining "-d ubuntu -r DISTRO-SHORT-CODENAME -a amd64" will be passed to the template script which will configure/populate the container.
In this specific example, the "-t download" makes lxc-create run a template script named something like "/usr/share/lxc/templates/lxc-download" to which it will pass "-d ubuntu -r DISTRO-SHORT-CODENAME -a amd64".

How can I do "export $(dbus-launch) in booting

I'm using GDbus and make a dbus communication.
It using sesstion bus.
Problem is dbus-launch.
I was running dbus in Yocto with c++11.
And, I have to export $(dbus-launch).
But, I want to export $(dbus-launch) or same thing in booting time.
Because dbus start by systemd.
One solution is to have a recipe that adds environment variable:
SRC_URI += "file://dbus-env.sh"
do_install_append() {
install -d -m 0755 ${D}${sysconfdir}/profile.d
install -m 0755 ${WORKDIR}/dbus-env.sh ${D}${sysconfdir}/profile.d/
}
FILES_${PN} += "${sysconfdir}/profile.d/dbus-env.sh"
With dbus-env.sh
#!/bin/sh
export $(dbus-launch)
Use this command in /etc/profile or $HOME/.profile or $HOME/.bashrc :
eval \`dbus-launch --auto-syntax`
this will export "DBUS_SESSION_BUS_ADDRESS" and "DBUS_SESSION_BUS_PID" with proper values
you can also use this script:
[[ -n $SSH_CLIENT ]] && export $(cat /proc/$(command pgrep -u "$USER" -f -- "dbus-daemon --session" )/environ| tr '\0' '\n' | command grep "DBUS_SESSION_BUS_ADDRESS=")

How do I set maxitemsize in memcached when my init script uses start-stop-daemon

I need to temporarily increase the max item size in memcached while I work on a permanent fix for my problem. I found this guide http://www.alphadevx.com/a/387-Changing-the-maximum-item-size-allowed-by-Memcache, unfortunately it tells me to add
-I $MAXITEMSIZE to the line
daemon --pidfile ${pidfile} memcached -d -p $PORT -u $USER -I $MAXITEMSIZE -m $CACHESIZE -c $MAXCONN -P ${pidfile} $OPTIONS
but in my /etc/init.d/memcached I've got the line
start-stop-daemon --start --quiet --exec "$DAEMONBOOTSTRAP" -- /etc/${NAME}.conf $PIDFILE
instead. I tried adding the -I flag to that line, but I got the error:
Restarting memcached: start-stop-daemon: invalid IO scheduler policy
I haven't found any guides that tell me how to do it if my script uses start-stop-daemon instead of deamon, and I haven't found any documentation about start-stop-daemon. Does anyone know how to do this?
Thanks in advance.
Oops, my bad. I had MAXITEMSIZE=5m instead of MAXITEMSIZE=5M. start-stop-daemon does accept the -I flag, you just have to get it right.

Best init script for running an application as a separate user

I have an application that runs in a user account (Plack-based) and want an init script.
It seems as easy as "sudo $user start_server ...". I just wrote an LSB script using start-stop-daemon and it is really clumsy and verbose. It doesn't feel like the right way.
After scouring for a bit and looking at a log of examples, I'm still not sure what the best way to do this is and there isn't a cohesive guide that I've found.
Right now I have it working with:
start-stop-daemon --background --quiet --start --pidfile $PIDFILE \
--make-pidfile --chuid $DAEMONUSER \
--exec $DAEMON -- $DAEMON_OPTS
With DAEMON and DAEMON_OPTS as:
DAEMON="/home/mediamogul/perl5/perlbrew/perls/current/bin/start_server"
DAEMON_OPTS="--port $PORT -- starman --workers $WORKERS /home/mediamogul/MediaMogul/script/mediamogul.psgi"
This then requires me to adjust how to detect running, because it's a perl script so perl is showing up as the command and not "start_server".
(I'm running this out of a perlbrew on that user account so it is completely separate from the system perl, that's why the paths are pointing to a perl in the user dir)
Is this really the best way to go about doing this? It seems very clunky to me, but I'm not an admin type.
You can use the --pid option to starman to have it write the PID when the app starts, if you use the same filename as you give start-stop-daemon then it will work nicly.
For example, from one of my init.d scripts:
SITENAME=mysite
PORT=5000
DIR=/websites/mysite
SCRIPT=bin/app.pl
USER=davidp
PIDFILE=/var/run/site-$SITENAME.pid
case "$1" in
start)
start-stop-daemon --start --chuid $USER --chdir $DIR \
--pidfile=$PIDFILE \
--exec /usr/local/bin/starman -- -p $PORT $SCRIPT -D --pid $PIDFILE
;;
stop)
start-stop-daemon --stop --pidfile $PIDFILE
;;
*)
echo "Usage: $SCRIPTNAME {start|stop}" >&2
exit 3
;;
esac
It's very close to what you are already doing, and I'll admit it is a little clumsy, granted, but it works - having Starman write the PID file means that start-stop-daemon can reliably start & stop it.