Make current file & the extension with condor match output & error files? (to have PBS and Slurm have same output files) - hpc

How do I make condor name my files as follow:
meta_learning_experiments_submission.py.e451863 meta_learning_experiments_submission.py.o444375
$(FILENAME).e$(CLUSTER)
$(FILENAME).e$(CLUSTER)
I tried it but it doesn't seem to work.
e.g. so that it matches the default behaviour of PBS when I do qsub?
I also like to know how to make slurm also match it, it would be perfect (so all three clusters are working).
I tried:
#SBATCH --error="(filename).%j.%N.err"
but that didn't work
Bounty
I want to set the file name automatically without hardcoding it.
related:
https://www.quora.com/unanswered/How-do-I-make-a-current-file-the-extension-with-condor-match-output-error-files-to-have-PBS-and-Slurm-have-the-same-output-files
For context here is my current submit file:
####################
#
# Experiments script
# Simple HTCondor submit description file
#
#
# chmod a+x test_condor.py
# chmod a+x experiments_meta_model_optimization.py
# chmod a+x meta_learning_experiments_submission.py
# chmod a+x download_miniImagenet.py
# chmod a+x ~/meta-learning-lstm-pytorch/main.py
# chmod a+x /home/miranda9/automl-meta-learning/automl-proj/meta_learning/datasets/rand_fc_nn_vec_mu_ls_gen.py
# chmod a+x /home/miranda9/automl-meta-learning/automl-proj/experiments/meta_learning/supervised_experiments_submission.py
# chmod a+x /home/miranda9/automl-meta-learning/results_plots/is_rapid_learning_real.py
# condor_submit -i
# condor_submit job.sub
#
####################
Path = /home/miranda9/automl-meta-learning/
Path = /home/miranda9/ML4Coq/
# Executable = /home/miranda9/automl-meta-learning/automl-proj/experiments/meta_learning/supervised_experiments_submission.py
# Executable = /home/miranda9/automl-meta-learning/automl-proj/experiments/meta_learning/meta_learning_experiments_submission.py
# Executable = /home/miranda9/meta-learning-lstm-pytorch/main.py
# Executable = /home/miranda9/automl-meta-learning/automl-proj/meta_learning/datasets/rand_fc_nn_vec_mu_ls_gen.py
# Executable = /home/miranda9/automl-meta-learning/results_plots/is_rapid_learning_real.py
## Output Files
Log = experiment_output_job.$(CLUSTER).log.out
Output = experiment_output_job.$(CLUSTER).out.out
Error = experiment_output_job.$(CLUSTER).err.out
Output = %(FILENAME).o$(CLUSTER)
# Use this to make sure 1 gpu is available. The key words are case insensitive.
# REquest_gpus = 1
# requirements = (CUDADeviceName != "Tesla K40m")
# requirements = (CUDADeviceName == "Quadro RTX 6000")
# requirements = ((CUDADeviceName = "Tesla K40m")) && (TARGET.Arch == "X86_64") && (TARGET.OpSys == "LINUX") && (TARGET.Disk >= RequestDisk) && (TARGET.Memory >= RequestMemory) && (TARGET.Cpus >= RequestCpus) && (TARGET.gpus >= Requestgpus) && ((TARGET.FileSystemDomain == MY.FileSystemDomain) || (TARGET.HasFileTransfer))
# requirements = (CUDADeviceName == "Tesla K40m")
# requirements = (CUDADeviceName == "GeForce GTX TITAN X")
# Note: to use multiple CPUs instead of the default (one CPU), use request_cpus as well
# Request_cpus = 4
Request_cpus = 16
# E-mail option
Notify_user = me#gmail.com
Notification = always
Environment = MY_CONDOR_JOB_ID= $(CLUSTER)
# "Queue" means add the setup until this line to the queue (needs to be at the end of script).
Queue
Alternatively, if I can use my executable script as the submission script with the parameters at the top that would work too
I usually use my submission script be the same as my executable script, e.g. see my qsub example:
#!/homes/miranda9/.conda/envs/automl-meta-learning/bin/python
#PBS -V
#PBS -M mee#gmail.com
#PBS -m abe
#PBS -lselect=1:ncpus=112
import sys
import os
for p in sys.path:
print(p)
print(os.environ)

HTCondor doesn't set FILENAME by default, but you could set that yourself, e.g.
FILENAME = meta_learning_experiments_submission.py
output = $(FILENAME).o.$(CLUSTER)
error = $(FILENAME).e.$(CLUSTER)

Related

Replacing characters in a sh script

I am writing an sh script and need to replace the . and - with a _
Current:
V123_45_678_910.11_1213-1415.sh
Wanted:
V123_45_678_910_11_1213_1415.sh
I have used a few mv commands, but I am having trouble.
for file in /virtualun/rest/scripts/IOL_Extra/*.sh ; do mv $file ${file//V15_IOL_NVMe_01./V15_IOL_NVMe_01_} ; done
You don't need to match any of the other parts of the file name, just the characters you want to replace. To avoid turning foo.sh into foo-sh, remove the extension first, then add it back to the result of the replacement.
for file in /virtualun/rest/scripts/IOL_Extra/*.sh ; do
base=${file%.sh}
mv -i -- "$file" "${base//[-.]/_}".sh
done
Use the -i option to make sure you don't inadvertently replace one file with another when the modified names coincide.
This should work:
#!/usr/bin/env sh
# Fail on error
set -o errexit
# Disable undefined variable reference
set -o nounset
# Enable wildcard character expansion
set +o noglob
# ================
# CONFIGURATION
# ================
# Pattern
PATTERN="/virtualun/rest/scripts/IOL_Extra/*.sh"
# ================
# LOGGER
# ================
# Fatal log message
fatal() {
printf '[FATAL] %s\n' "$#" >&2
exit 1
}
# Info log message
info() {
printf '[INFO ] %s\n' "$#"
}
# ================
# MAIN
# ================
{
# Check directory exists
[ -d "$(dirname "$PATTERN")" ] || fatal "Directory '$PATTERN' does not exists"
for _file in $PATTERN; do
# Skip if not file
[ -f "$_file" ] || continue
info "Analyzing file '$_file'"
# File data
_file_dirname=$(dirname -- "$_file")
_file_basename=$(basename -- "$_file")
_file_name="${_file_basename%.*}"
_file_extension=
case $_file_basename in
*.*) _file_extension=".${_file_basename##*.}" ;;
esac
# New file name
_new_file_name=$(printf '%s\n' "$_file_name" | sed 's/[\.\-][\.\-]*/_/g')
# Skip if equals
[ "$_file_name" != "$_new_file_name" ] || continue
# New file
_new_file="$_file_dirname/${_new_file_name}${_file_extension}"
# Rename
info "Renaming file '$_file' to '$_new_file'"
mv -i -- "$_file" "$_new_file"
done
}
You can try this:
for f in /virtualun/rest/scripts/IOL_Extra/*.sh; do
mv "$f" $(sed 's/[.-]/_/g' <<< "$f")
done
The sed command is replacing all characters .- by _.
I prefer using sed substitute as posted by oliv.
However, if you have not familiar with regular expression, using rename is faster/easier to understand:
Example:
$ touch V123_45_678_910.11_1213-1415.sh
$ rename -va '.' '_' *sh
`V123_45_678_910.11_1213-1415.sh' -> `V123_45_678_910_11_1213-1415_sh'
$ rename -va '-' '_' *sh
`V123_45_678_910_11_1213-1415_sh' -> `V123_45_678_910_11_1213_1415_sh'
$ rename -vl '_sh' '.sh' *sh
`V123_45_678_910_11_1213_1415_sh' -> V123_45_678_910_11_1213_1415.sh'
$ ls *sh
V123_45_678_910_11_1213_1415.sh
Options explained:
-v prints the name of the file before -> after the operation
-a replaces all occurrences of the first argument with the second argument
-l replaces the last occurrence of the first argument with the second argument
Note that this might not be suitable depending on the other files you have in the given directory that would match *sh and that you do NOT want to rename.

How do I make the autosuggestions of a zsh function use syntax highlighting

I use zsh and wrote a function to replace cd function. With some help I got it to work like I want it to (mostly). This is a followup to one of my other question.
The function almost works like I want it to, but I still have some problems with syntax highlighting and autocompletion.
For the examples, lets say your directories look like this:
/
a/
b/
c/
d/
some_dir/
I am also assuming the following code has been sourced:
cl () {
local first=$( echo $1 | cut -d/ -f1 )
if [ -d $first ]; then
pushd $1 >/dev/null # If the first argument is an existing normal directory, move there
else
pushd ${PWD%/$first/*}/$1 >/dev/null # Otherwise, move to a parent directory or a child of that parent directory
fi
}
_cl() {
_cd
pth=${words[2]}
opts=""
new=${pth##*/}
local expl
# Generate the visual formatting and store it in `$expl`
_description -V ancestor-directories expl 'ancestor directories'
[[ "$pth" != *"/"*"/"* ]] && middle="" || middle="${${pth%/*}#*/}/"
if [[ "$pth" != *"/"* ]]; then
# If this is the start of the path
# In this case we should also show the parent directories
local ancestor=$PWD:h
while (( $#ancestor > 1 )); do
# -f: Treat this as a file (incl. dirs), so you get proper highlighting.
# -Q: Don't quote (escape) any of the characters.
# -W: Specify the parent of the dir we're adding.
# ${ancestor:h}: The parent ("head") of $ancestor.
# ${ancestor:t}: The short name ("tail") of $ancestor.
compadd "$expl[#]" -fQ -W "${ancestor:h}/" - "${ancestor:t}"
# Move on to the next parent.
ancestor=$ancestor:h
done
else
# $first is the first part of the path the user typed in.
# it it is part of the current direoctory, we know the user is trying to go back to a directory
first=${pth%%/*}
# $middle is the rest of the provided path
if [ ! -d $first ]; then
# path starts with parent directory
dir=${PWD%/$first/*}/$first
first=$first/
# List all sub directories of the $dir/$middle directory
if [ -d "$dir/$middle" ]; then
for d in $(ls -a $dir/$middle); do
if [ -d $dir/$middle/$d ] && [[ "$d" != "." ]] && [[ "$d" != ".." ]]; then
compadd "$expl[#]" -fQ -W $dir/ - $first$middle$d
fi
done
fi
fi
fi
}
compdef _cl cl
The problem:
I use syntax-highlighting, but the path i type is just white (when going to a parent directory. the normal cd functions are colored).
Example:
$ cd /a
$ cl c # 'c' is colored
$ pwd
/a/c
$ cl a/b # 'a/b' is not colored
$ cl a/[tab] # 'a/b', 'a/c' and 'a/some_dir' are not colored
How do I get these paths to be colored?
That's not possible out of the box with the zsh-syntax-highlighting plugin. It checks only whether what you've typed is
A) a valid absolute path or
B) a valid path relative to the present working dir.
This is not specific to your command. Highlighting also fails when specifying path arguments to other commands that are otherwise valid, but are not absolute paths or valid paths relative to the present working dir.
For example:
% cd /a/b
% cd b c # perfectly valid args, but will not get highlighted as valid paths
% pwd
/a/c

How do I make a zsh function autocomplet from the middle of a word?

I use zsh and wrote a function to replace cd function. With some help I got it to work like I want it to (mostly). This is a followup to one of my other question.
The function almost works like I want it to, but I still have some problems with syntax highlighting and autocompletion.
For the examples, lets say your directories look like this:
/
a/
b/
c/
d/
some_dir/
I am also assuming the following code has been sourced:
cl () {
local first=$( echo $1 | cut -d/ -f1 )
if [ -d $first ]; then
pushd $1 >/dev/null # If the first argument is an existing normal directory, move there
else
pushd ${PWD%/$first/*}/$1 >/dev/null # Otherwise, move to a parent directory or a child of that parent directory
fi
}
_cl() {
_cd
pth=${words[2]}
opts=""
new=${pth##*/}
local expl
# Generate the visual formatting and store it in `$expl`
_description -V ancestor-directories expl 'ancestor directories'
[[ "$pth" != *"/"*"/"* ]] && middle="" || middle="${${pth%/*}#*/}/"
if [[ "$pth" != *"/"* ]]; then
# If this is the start of the path
# In this case we should also show the parent directories
local ancestor=$PWD:h
while (( $#ancestor > 1 )); do
# -f: Treat this as a file (incl. dirs), so you get proper highlighting.
# -Q: Don't quote (escape) any of the characters.
# -W: Specify the parent of the dir we're adding.
# ${ancestor:h}: The parent ("head") of $ancestor.
# ${ancestor:t}: The short name ("tail") of $ancestor.
compadd "$expl[#]" -fQ -W "${ancestor:h}/" - "${ancestor:t}"
# Move on to the next parent.
ancestor=$ancestor:h
done
else
# $first is the first part of the path the user typed in.
# it it is part of the current direoctory, we know the user is trying to go back to a directory
first=${pth%%/*}
# $middle is the rest of the provided path
if [ ! -d $first ]; then
# path starts with parent directory
dir=${PWD%/$first/*}/$first
first=$first/
# List all sub directories of the $dir/$middle directory
if [ -d "$dir/$middle" ]; then
for d in $(ls -a $dir/$middle); do
if [ -d $dir/$middle/$d ] && [[ "$d" != "." ]] && [[ "$d" != ".." ]]; then
compadd "$expl[#]" -fQ -W $dir/ - $first$middle$d
fi
done
fi
fi
fi
}
compdef _cl cl
The problem:
In my zshrc, I have a line:
zstyle ':completion:*' matcher-list 'm:{a-z}={A-Za-z}' '+l:|=* r:|=*'
This should make autocompletions case insensitive and make sure I can start typing the last part of a directory name, and in will still finnish the full name
Example:
$ cd /a
$ cd di[tab] # replaces 'di' with 'some_dir/'
$ cl di[tab] # this does not do anything. I would like it to replace 'di' with 'some_dir'
How do get it to suggest 'some_dir' when I type 'di'?
The second matcher in your matcher-list never gets called, because _cl() returns "true" (exit status 0, actually) even when it has not added any matches. Returning "true" causes _main_complete() to assume that we're done completing and it will thus not try the next matcher in the list.
To fix this, add the following to the start of _cl():
local -i nmatches=$compstate[nmatches]
and this to the end of _cl():
(( compstate[nmatches] > nmatches ))
That way, _cl() will return "true" only when it has managed to actually add completions.

can't use '~' in zsh autocompletion

I use zsh and I want to use a function I wrote to replace cd.
This function gives you the ability to move to a parent directory:
$ pwd
/a/b/c/d
$ cl b
$ pwd
/a/b
You can also move into a subdirectory of a parent directory:
$ pwd
/a/b/c/d
$ cl b/e
$ pwd
/a/b/e
If the first part of the path is not a parent directory, it will just function as normal cd would. I hope that makes sense.
In summary, when in /a/b/c/d, I want to be able to move to /a, /a/b, /a/b/c, all subdirectories of /a/b/c/d and any absolute path starting with /, ~/ or ../ (or ./).
I hope that makes sense.
This is the function I wrote:
cl () {
local first=$( echo $1 | cut -d/ -f1 )
if [ $# -eq 0 ]; then
# cl without any arguments moves back to the previous directory
cd - > /dev/null
elif [ -d $first ]; then
# If the first argument is an existing normal directory, move there
cd $1
else
# Otherwise, move to a parent directory
cd ${PWD%/$first/*}/$1
fi
}
There is probably a better way to this (tips are welcome), but I haven't had any problems with this so far.
Now I want to add autocompletion. This is what I have so far:
_cl() {
pth=${words[2]}
opts=""
new=${pth##*/}
[[ "$pth" != *"/"*"/"* ]] && middle="" || middle="${${pth%/*}#*/}/"
if [[ "$pth" != *"/"* ]]; then
# If this is the start of the path
# In this case we should also show the parent directories
opts+=" "
first=""
d="${${PWD#/}%/*}/"
opts+="${d//\/// }"
dir=$PWD
else
first=${pth%%/*}
if [[ "$first" == "" ]]; then
# path starts with "/"
dir="/$middle"
elif [[ "$first" == "~" ]]; then
# path starts with "~/"
dir="$HOME/$middle"
elif [ -d $first ]; then
# path starts with a directory in the current directory
dir="$PWD/$first/$middle"
else
# path starts with parent directory
dir=${PWD%/$first/*}/$first/$middle
fi
first=$first/
fi
# List al sub directories of the $dir directory
if [ -d "$dir" ]; then
for d in $(ls -a $dir); do
if [ -d $dir/$d ] && [[ "$d" != "." ]] && [[ "$d" != ".." ]]; then
opts+="$first$middle$d/ "
fi
done
fi
_multi_parts / "(${opts})"
return 0
}
compdef _cl cl
Again, probably not the best way to do this, but it works... kinda.
One of the problems is that what I type cl ~/, it replaces it with cl ~/ and does not suggest any directories in my home folder. Is there a way to get this to work?
EDIT
cl () {
local first=$( echo $1 | cut -d/ -f1 )
if [ $# -eq 0 ]; then
# cl without any arguments moves back to the previous directory
local pwd_bu=$PWD
[[ $(dirs) == "~" ]] && return 1
while [[ $PWD == $pwd_bu ]]; do
popd >/dev/null
done
local pwd_nw=$PWD
[[ $(dirs) != "~" ]] && popd >/dev/null
pushd $pwd_bu >/dev/null
pushd $pwd_nw >/dev/null
elif [ -d $first ]; then
pushd $1 >/dev/null # If the first argument is an existing normal directory, move there
else
pushd ${PWD%/$first/*}/$1 >/dev/null # Otherwise, move to a parent directory or a child of that parent directory
fi
}
_cl() {
_cd
pth=${words[2]}
opts=""
new=${pth##*/}
local expl
# Generate the visual formatting and store it in `$expl`
_description -V ancestor-directories expl 'ancestor directories'
[[ "$pth" != *"/"*"/"* ]] && middle="" || middle="${${pth%/*}#*/}/"
if [[ "$pth" != *"/"* ]]; then
# If this is the start of the path
# In this case we should also show the parent directories
local ancestor=$PWD:h
while (( $#ancestor > 1 )); do
# -f: Treat this as a file (incl. dirs), so you get proper highlighting.
# -Q: Don't quote (escape) any of the characters.
# -W: Specify the parent of the dir we're adding.
# ${ancestor:h}: The parent ("head") of $ancestor.
# ${ancestor:t}: The short name ("tail") of $ancestor.
compadd "$expl[#]" -fQ -W "${ancestor:h}/" - "${ancestor:t}"
# Move on to the next parent.
ancestor=$ancestor:h
done
else
# $first is the first part of the path the user typed in.
# it it is part of the current direoctory, we know the user is trying to go back to a directory
first=${pth%%/*}
# $middle is the rest of the provided path
if [ ! -d $first ]; then
# path starts with parent directory
dir=${PWD%/$first/*}/$first
first=$first/
# List all sub directories of the $dir/$middle directory
if [ -d "$dir/$middle" ]; then
for d in $(ls -a $dir/$middle); do
if [ -d $dir/$middle/$d ] && [[ "$d" != "." ]] && [[ "$d" != ".." ]]; then
compadd "$expl[#]" -fQ -W $dir/ - $first$middle$d
fi
done
fi
fi
fi
}
compdef _cl cl
This is as far as I got on my own. It does works (kinda) but has a couple of problems:
When going back to a parent directory, completion mostly works. But when you go to a child of the paretn directory, the suggestions are wrong (they display the full path you have typed, not just the child directory). The result does work
I use syntax-hightlighting, but the path I type is just white (when using going to a parent directory. the normal cd functions are colored)
In my zshrc, I have the line:
zstyle ':completion:*' matcher-list 'm:{a-z}={A-Za-z}' '+l:|=* r:|=*'
Whith cd this means I can type "load" and it will complete to "Downloads". With cl, this does not work. Not event when using the normal cd functionality.
Is there a way to fix (some of these) problems?
I hope you guys understand my questions. I find it hard to explain the problem.
Thanks for your help!
This should do it:
_cl() {
# Store the number of matches generated so far.
local -i nmatches=$compstate[nmatches]
# Call the built-in completion for `cd`. No need to reinvent the wheel.
_cd
# ${PWD:h}: The parent ("head") of the present working dir.
local ancestor=$PWD:h expl
# Generate the visual formatting and store it in `$expl`
# -V: Don't sort these items; show them in the order we add them.
_description -V ancestor-directories expl 'ancestor directories'
while (( $#ancestor > 1 )); do
# -f: Treat this as a file (incl. dirs), so you get proper highlighting.
# -W: Specify the parent of the dir we're adding.
# ${ancestor:h}: The parent ("head") of $ancestor.
# ${ancestor:t}: The short name ("tail") of $ancestor.
compadd "$expl[#]" -f -W ${ancestor:h}/ - $ancestor:t
# Move on to the next parent.
ancestor=$ancestor:h
done
# Return true if we've added any matches.
(( compstate[nmatches] > nmatches ))
}
# Define the function above as generating completions for `cl`.
compdef _cl cl
# Alternatively, instead of the line above:
# 1. Create a file `_cl` inside a dir that's in your `$fpath`.
# 2. Paste the _contents_ of the function `_cl` into this file.
# 3. Add `#compdef cl` add the top of the file.
# `_cl` will now get loaded automatically when you run `compinit`.
Also, I would rewrite your cl function like this, so it no longer depends on cut or other external commands:
cl() {
if (( $# == 0 )); then
# `cl` without any arguments moves back to the previous directory.
cd -
elif [[ -d $1 || -d $PWD/$1 ]]; then
# If the argument is an existing absolute path or direct child, move there.
cd $1
else
# Get the longest prefix that ends with the argument.
local ancestor=${(M)${PWD:h}##*$1}
if [[ -d $ancestor ]]; then
# Move there, if it's an existing dir.
cd $ancestor
else
# Otherwise, print to stderr and return false.
print -u2 "$0: no such ancestor '$1'"
return 1
fi
fi
}
Alternative Solution
There is an easier way to do all of this, without the need to write a cd replacement or any completion code:
cdpath() {
# `$PWD` is always equal to the present working directory.
local dir=$PWD
# In addition to searching all children of `$PWD`, `cd` will also search all
# children of all of the dirs in the array `$cdpath`.
cdpath=()
# Add all ancestors of `$PWD` to `$cdpath`.
while (( $#dir > 1 )); do
# `:h` is the direct parent.
dir=$dir:h
cdpath+=( $dir )
done
}
# Run the function above whenever we change directory.
add-zsh-hook chpwd cdpath
Zsh's completion code for cd automatically takes $cdpath into account. No need to even configure that. :)
As an example of how this works, let's say you're in /Users/marlon/.zsh/prezto/modules/history-substring-search/external/.
You can now type cd pre and press Tab, and Zsh will complete it to cd prezto. After that, pressing Enter will take you directly to /Users/marlon/.zsh/prezto/.
Or let's say that there also exists /Users/marlon/.zsh/prezto/modules/prompt/external/agnoster/. When you're in the former dir, you can do cd prompt/external/agnoster to go directly to the latter, and Zsh will complete this path for you every step of the way.

How can we create two instances of memcached server in same server in different port?

I tried to add in the way
-l 11211
-l 11212
in memcached conf file. But it is just listening to first one i.e 1121
First I used mikewied's solution, but then I bumped into the problem of auto starting the daemon. Another confusing thing in that solution is that it doesn't use the config from etc. I was about to create my own start up scripts in /etc/init.d but then I looked into /etc/init.d/memcached file and saw this beautiful solution
# Usage:
# cp /etc/memcached.conf /etc/memcached_server1.conf
# cp /etc/memcached.conf /etc/memcached_server2.conf
# start all instances:
# /etc/init.d/memcached start
# start one instance:
# /etc/init.d/memcached start server1
# stop all instances:
# /etc/init.d/memcached stop
# stop one instance:
# /etc/init.d/memcached stop server1
# There is no "status" command.
Basically readers of this question just need to read the /etc/init.d/memcached file.
Cheers
Here's what memcached says the -l command is for:
-l <addr> interface to listen on (default: INADDR_ANY, all addresses)
<addr> may be specified as host:port. If you don't specify
a port number, the value you specified with -p or -U is
used. You may specify multiple addresses separated by comma
or by using -l multiple times
First off you need to specify the interface you want memcached to listen on if you are using the -l flag. Use 0.0.0.0 for all interfaces and use 127.0.0.1 is you just want to be able to access memcached from localhost. Second, don't use two -l flags. Use only one and separate each address by a comma. The command below should do what you want.
memcached -l 0.0.0.0:11211,0.0.0.0:11212
Keep in mind that this will have one memcached instance listen on two ports. To have two memcached instances on one machine run these two commands.
memcached -p 11211 -d
memcached -p 11212 -d
The answer from David Dzhagayev is the best one. If you don't have the correct version of memcache init script, here is the one he is talking about:
It should work with any linux distro using init.
#! /bin/bash
### BEGIN INIT INFO
# Provides: memcached
# Required-Start: $remote_fs $syslog
# Required-Stop: $remote_fs $syslog
# Should-Start: $local_fs
# Should-Stop: $local_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Start memcached daemon
# Description: Start up memcached, a high-performance memory caching daemon
### END INIT INFO
# Usage:
# cp /etc/memcached.conf /etc/memcached_server1.conf
# cp /etc/memcached.conf /etc/memcached_server2.conf
# start all instances:
# /etc/init.d/memcached start
# start one instance:
# /etc/init.d/memcached start server1
# stop all instances:
# /etc/init.d/memcached stop
# stop one instance:
# /etc/init.d/memcached stop server1
# There is no "status" command.
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin
DAEMON=/usr/bin/memcached
DAEMONNAME=memcached
DAEMONBOOTSTRAP=/usr/share/memcached/scripts/start-memcached
DESC=memcached
test -x $DAEMON || exit 0
test -x $DAEMONBOOTSTRAP || exit 0
set -e
. /lib/lsb/init-functions
# Edit /etc/default/memcached to change this.
ENABLE_MEMCACHED=no
test -r /etc/default/memcached && . /etc/default/memcached
FILES=(/etc/memcached_*.conf)
# check for alternative config schema
if [ -r "${FILES[0]}" ]; then
CONFIGS=()
for FILE in "${FILES[#]}";
do
# remove prefix
NAME=${FILE#/etc/}
# remove suffix
NAME=${NAME%.conf}
# check optional second param
if [ $# -ne 2 ];
then
# add to config array
CONFIGS+=($NAME)
elif [ "memcached_$2" == "$NAME" ];
then
# use only one memcached
CONFIGS=($NAME)
break;
fi;
done;
if [ ${#CONFIGS[#]} == 0 ];
then
echo "Config not exist for: $2" >&2
exit 1
fi;
else
CONFIGS=(memcached)
fi;
CONFIG_NUM=${#CONFIGS[#]}
for ((i=0; i < $CONFIG_NUM; i++)); do
NAME=${CONFIGS[${i}]}
PIDFILE="/var/run/${NAME}.pid"
case "$1" in
start)
echo -n "Starting $DESC: "
if [ $ENABLE_MEMCACHED = yes ]; then
start-stop-daemon --start --quiet --exec "$DAEMONBOOTSTRAP" -- /etc/${NAME}.conf $PIDFILE
echo "$NAME."
else
echo "$NAME disabled in /etc/default/memcached."
fi
;;
stop)
echo -n "Stopping $DESC: "
start-stop-daemon --stop --quiet --oknodo --retry 5 --pidfile $PIDFILE --exec $DAEMON
echo "$NAME."
rm -f $PIDFILE
;;
restart|force-reload)
#
# If the "reload" option is implemented, move the "force-reload"
# option to the "reload" entry above. If not, "force-reload" is
# just the same as "restart".
#
echo -n "Restarting $DESC: "
start-stop-daemon --stop --quiet --oknodo --retry 5 --pidfile $PIDFILE
rm -f $PIDFILE
if [ $ENABLE_MEMCACHED = yes ]; then
start-stop-daemon --start --quiet --exec "$DAEMONBOOTSTRAP" -- /etc/${NAME}.conf $PIDFILE
echo "$NAME."
else
echo "$NAME disabled in /etc/default/memcached."
fi
;;
status)
status_of_proc -p $PIDFILE $DAEMON $NAME && exit 0 || exit $?
;;
*)
N=/etc/init.d/$NAME
echo "Usage: $N {start|stop|restart|force-reload|status}" >&2
exit 1
;;
esac
done;
exit 0
In case someone else stumbles upon this question, there is a bug on the debian distribution of memcached (which means flavours like Ubuntu would also be affected).
https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=784357
Because of this bug, even when you have separate configuration files, when you run sudo service memcached restart, only the default configuration file in /etc/memcached.conf will be loaded.
As mentioned in the comment here, the temporary solution is to
Remove /lib/systemd/system/memcached.service
Run sudo systemctl daemon-reload (don't worry, it is safe to do
so)
Finally, run sudo service memcached restart if you are okay with losing all cache information. If not, run sudo service memcached force-reload
Ok, very good answer, Tristan CHARBONNIER.
Please replace code into file /usr/share/memcached/scripts/start-memcached:
#!/usr/bin/perl -w
# start-memcached
# 2003/2004 - Jay Bonci
# This script handles the parsing of the /etc/memcached.conf file
# and was originally created for the Debian distribution.
# Anyone may use this little script under the same terms as
# memcached itself.
use strict;
if($> != 0 and $< != 0)
{
print STDERR "Only root wants to run start-memcached.\n";
exit;
}
my $params; my $etchandle; my $etcfile = "/etc/memcached.conf";
# This script assumes that memcached is located at /usr/bin/memcached, and
# that the pidfile is writable at /var/run/memcached.pid
my $memcached = "/usr/bin/memcached";
my $pidfile = "/var/run/memcached.pid";
if (scalar(#ARGV) == 2) {
$etcfile = shift(#ARGV);
$pidfile = shift(#ARGV);
}
# If we don't get a valid logfile parameter in the /etc/memcached.conf file,
# we'll just throw away all of our in-daemon output. We need to re-tie it so
# that non-bash shells will not hang on logout. Thanks to Michael Renner for
# the tip
my $fd_reopened = "/dev/null";
sub handle_logfile
{
my ($logfile) = #_;
$fd_reopened = $logfile;
}
sub reopen_logfile
{
my ($logfile) = #_;
open *STDERR, ">>$logfile";
open *STDOUT, ">>$logfile";
open *STDIN, ">>/dev/null";
$fd_reopened = $logfile;
}
# This is set up in place here to support other non -[a-z] directives
my $conf_directives = {
"logfile" => \&handle_logfile,
};
if(open $etchandle, $etcfile)
{
foreach my $line (< $etchandle>)
{
$line ||= "";
$line =~ s/\#.*//g;
$line =~ s/\s+$//g;
$line =~ s/^\s+//g;
next unless $line;
next if $line =~ /^\-[dh]/;
if($line =~ /^[^\-]/)
{
my ($directive, $arg) = $line =~ /^(.*?)\s+(.*)/;
$conf_directives->{$directive}->($arg);
next;
}
push #$params, $line;
}
}else{
$params = [];
}
push #$params, "-u root" unless(grep "-u", #$params);
$params = join " ", #$params;
if(-e $pidfile)
{
open PIDHANDLE, "$pidfile";
my $localpid = <PIDHANDLE>;
close PIDHANDLE;
chomp $localpid;
if(-d "/proc/$localpid")
{
print STDERR "memcached is already running.\n";
exit;
}else{
`rm -f $localpid`;
}
}
my $pid = fork();
if($pid == 0)
{
reopen_logfile($fd_reopened);
exec "$memcached $params";
exit(0);
}else{
if(open PIDHANDLE,">$pidfile")
{
print PIDHANDLE $pid;
close PIDHANDLE;
}else{
print STDERR "Can't write pidfile to $pidfile.\n";
}
}
Simple solution to Centos 6
First copy /etc/sysconfig/memcached to /etc/sysconfig/memcached2 and write new settings to the new file.
Then copy /etc/init.d/memcached to /etc/init.d/memcached2 and change in the new file:
PORT to your new port (it should be reset from /etc/sysconfig/memcached2, so we do it just in case)
/etc/sysconfig/memcached to /etc/sysconfig/memcached2
/var/run/memcached/memcached.pid to /var/run/memcached/memcached2.pid
/var/lock/subsys/memcached to /var/lock/subsys/memcached2
Now you can use service memcached2 start, service memcached2 stop etc. Don't forget chkconfig memcached2 on to run it when machine boots up.
in /etc/memcached.conf you can just edit like below
-l 192.168.112.22,127.0.0.1
must use comma between two ip address