spark failed to launch org.apache.spark.deploy.worker.worker on master - apache-spark-standalone

I have setup Spark Standalone Cluster on two Ubuntu servers ( master and one slave).
I had config /conf/spark-env.sh (after copy it from spark-env.sh.template) as follows:
SPARK_MASTER_HOST="master"
I started spark-master successfully on master by command below .
sudo /opt/spark/sbin/start-master.sh
But starting spark-slave on the master results with error:
hadoop#master:/opt/spark$ sudo ./sbin/start-slave.sh --master spark://master:7077 --cores 2
/opt/spark/conf/spark-env.sh: line 28: SPARK_LOCAL_IP: command not found
starting org.apache.spark.deploy.worker.Worker, logging to /opt/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-master.out
failed to launch: nice -n 0 /opt/spark/bin/spark-class org.apache.spark.deploy.worker.Worker --webui-port 8081 --master spark://master:7077 --cores 2
Options:
-c CORES, --cores CORES Number of cores to use
-m MEM, --memory MEM Amount of memory to use (e.g. 1000M, 2G)
-d DIR, --work-dir DIR Directory to run apps in (default: SPARK_HOME/work)
-i HOST, --ip IP Hostname to listen on (deprecated, please use --host or -h)
-h HOST, --host HOST Hostname to listen on
-p PORT, --port PORT Port to listen on (default: random)
--webui-port PORT Port for web UI (default: 8081)
--properties-file FILE Path to a custom Spark properties file.
Default is conf/spark-defaults.conf.
full log in /opt/spark/logs/spark-root-org.apache.spark.deploy.worker.Worker-1-master.out
I had search a lot but I do not figure out what is my problem?

I found my fault, I should not use --master keyword and just run command
hadoop#master:/opt/spark$ sudo ./sbin/start-slave.sh spark://master:7077
following the steps of this tutorial: https://phoenixnap.com/kb/install-spark-on-ubuntu
Hint: make sure to install all dependencies before:
sudo apt install scala git -y

Related

Jconsole not connecting with jmx port of Keycloak

I tried to enable jmx to check on cache statistics, I tried to do this in local setup with following command
docker run -it --rm --name keycloak
--cap-add SYS_ADMIN
-p 8080:8080
-p 8787:8787
-p 8999:8999
-e KEYCLOAK_ADMIN="keycloak"
-e KEYCLOAK_ADMIN_PASSWORD="keycloak"
-e DEBUG="true"
-e DEBUG_PORT="*:8787"
-e JAVA_OPTS_APPEND="-Xmx1g
-Dcom.sun.management.jmxremote.port=8999 -Dcom.sun.management.jmxremote.rmi.port=8999
-Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.local.only=false -Djava.rmi.server.hostname="$(hostname)"
-XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp/keycloak.hprof"
quay.io/keycloak/keycloak:17.0.0 start-dev
--log-level=INFO
Jmx is enabled but not getting connected from jconsole
You should take a look in the log-files. Are there error messages or can you see, wheter the ARGS reach the startup process?
I had some trouble to add the JMX Options on keycloak-16, the ser failed to start afterward. Then I found this article. This works for me.
If you run the docker image on a remote host, you can copy the jboss-cli-client.jar to your local maschine. And after each startof the container you have to add the user again.

How do I connect bare metal worker to docker TSA host

I have followed the docker-compose concourse installation set up
Everything is up and running but I cant figure out what to use as --tsa-host value in command to connect worker to TSA host
Would be worth mentioning that docker concourse web and db are running on the same machine that I hope to use as bare metal worker.
I have tried 1. to use IP address of concourse web container but no joy. I cannot even ping the docker container IP from host.
1.
sudo ./concourse worker --work-dir ./worker --tsa-host IP_OF_DOCKER_CONTAINER --tsa-public-key host_key.pub
--tsa-worker-private-key worker_key
I have also tried using the 2. CONCOURSE_EXTERNAL_URL and 3. the ip address of the host but no luck either.
2.
sudo ./concourse worker --work-dir ./worker --tsa-host http://10.XXX.XXX.XX:8080 --tsa-public-key host_key.pub
--tsa-worker-private-key worker_key
3.
sudo ./concourse worker --work-dir ./worker --tsa-host 10.XXX.XXX.XX:8080 --tsa-public-key host_key.pub
--tsa-worker-private-key worker_key
Other details of setup:
Mac OSX Sierra
Docker For Mac
Please confirm you use the internal IP of the host, not public IP, not container IP.
--tsa-host <INTERNAL_IP_OF_HOST>
If you use docker-compose.yml as in its setup document, you needn't care of TSA-HOST, the environmen thas been defined already
CONCOURSE_TSA_HOST: concourse-web
I used the docker-compose.yml recently with the steps described here https://concourse-ci.org/docker-repository.html .
Please confirm that there is a keys directory next to the docker-compose.yml after you executed the steps.
mkdir -p keys/web keys/worker
ssh-keygen -t rsa -f ./keys/web/tsa_host_key -N ''
ssh-keygen -t rsa -f ./keys/web/session_signing_key -N ''
ssh-keygen -t rsa -f ./keys/worker/worker_key -N ''
cp ./keys/worker/worker_key.pub ./keys/web/authorized_worker_keys
cp ./keys/web/tsa_host_key.pub ./keys/worker
export CONCOURSE_EXTERNAL_URL=http://192.168.99.100:8080
docker-compose up

Create multiple Postgres instances on same machine

To test streaming replication, I would like to create a second Postgres instance on the same machine. The idea is that if it can be done on the test server, then it should be trivial to set it up on the two production servers.
The instances should use different configuration files and different data directories. I tried following the instructions here http://ubuntuforums.org/showthread.php?t=1431697 but I haven't figured out how to get Postgres to use a different configuration file. If I copy the init script, the scripts are just aliases to the same Postgres instance.
I'm using Postgres 9.3 and the Postgres help pages say to specify the configuration file on the postgres command line. I'm not really sure what this means. Am I supposed to install some client for this to work? Thanks.
I assume you can work your way out on using postgresql utilities.
Create the clusters
$ initdb -D /path/to/datadb1
$ initdb -D /path/to/datadb2
Run the instances
$ pg_ctl -D /path/to/datadb1 -o "-p 5433" -l /path/to/logdb1 start
$ pg_ctl -D /path/to/datadb2 -o "-p 5434" -l /path/to/logdb2 start
Test streaming
Now you have two instances running on ports 5433 and 5434. Configuration files for them are in data dirs specified by initdb. Tweak them for streaming replication.
Your default installation remains untouched in port 5432.
On Debian based distros you could use pg_createcluster instead of initdb:
$ pg_createcluster -u [user] -g [group] -d /path/to/data -l /path/to/log -p 5433
Also pg_ctlcluster is an alternative to pg_ctl.
Steps to create New Server Instance on PostgreSQL 9.5
On command prompt run:
initdb -D Instance_Directory_path -U username -W
(prompts for password)
Once the new Instance Directory is created. Run command prompt as Administrator
pg_ctl register -N service_name -D Instance_Directory_path -o "-p port_no"
After the service is registered, start server
pg_ctl start -D Instance_Directory_path -o "-p port_no"
To complete other answers, on CentOS 6 AND 7.
After running something like
$ initdb -D /path/to/newdb
You'll have to change at least port configuration option and, probably, listen_addresses in config file postgresql.conf.
Instead of starting inmediatly this new instance, which has been explained in previous answers, maybe you want new instance to run automatically on system start (in case of shutdown, e.g.). To do this, as CentOS doesn't have pg_ctl register option (only for Windows) you'll have to create a new service file and register it in order systemctl or service can start it up automatically.
Centos 6
Follow next commands to get service's init file:
[root#machine ~]# service postgresql-9.6 edit
Usage: /etc/init.d/postgresql-9.6 {start|stop|status|restart|upgrade|condrestart|try-restart|reload|force-reload|initdb|promote}
[root#machine ~]# cd /etc/init.d # Now we know where service file is
[root#machine init.d]# cp -p postgresql-9.6 postgresql-9.6_5433
[root#machine init.d]# vi postgresql-9.6_5433
Now you can change PGDATA directory with the one where new instance resides. If you're using Postgresql version previous to 9.4 (which you shouldn't by the time of this answer) you'll have to change PGPORT too with the value where new instance is listening to.
The name of the new service is up to you. I usually take original service name and add port number at the end.
Now you only have to register new service:
[root#machine init.d]# chkconfig postgresql-9.6_5433 on # service registered!
[root#machine init.d]# service postgresql-9.6_5433 start
Iniciando servicios postgresql-9.6_5433: [ OK ]
[root#machine init.d]# service postgresql-9.6_5433 status
Se está ejecutando postgresql-9.6_5433 (pid 120993)...
Centos 7
In CentOS 7 instead of service to control services running on the machine you have systemctl and commands and paths change a bit. But the process is the same: create new service file, edit with the new location/port, register and start:
[root#localhost ~]# locate postgresql.service
/etc/systemd/system/multi-user.target.wants/postgresql.service
/usr/lib/systemd/system/postgresql.service
[root#localhost ~]# cd /usr/lib/systemd/system
[root#localhost ~]# cp -p postgresql.service postgresql_5433.service
[root#localhost ~]# vi postgresql_5433.service
# Change PGDATA and maybe PGPORT if PG version <9.4
[root#localhost ~]# systemctl enable postgresql_5433.service
[root#localhost ~]# systemctl start postgresql_5433.service
[root#localhost ~]# systemctl list-unit-files | grep postgres
postgresql.service enabled
postgresql_5433.service enabled

memcached doesn't start on vagrant up with CentOS 6.5

I'm trying to provision a development box with Vagrant and a CentOS 6.5 base box. I want memcached to automatically start at system boot/vagrant up.
I have tried adding memcached -d -l localhost -p11211 to /etc/rc.d/rc.local and this does not work.
I have also tried adding to /etc/init/vagrant-mounted.conf
start on vagrant-mounted
memcached -d -l localhost -p11211
[EDIT]
I've updated /etc/rc.d/rc.local to now use the following
chkconfig memcached on
service memcached start
I'm not seeing anything in the /var/log/boot.log. It looks like rc.local is not being run at all. It has ugo+x permissions; so the file is definitely executable, but it doesn't appear to run at all.
Does memcached -d -l localhost -p11211 exit immediately or spawn a process?
If it keeps running, try: nohup memcached -d -l localhost -p11211 &
Also, try putting it in /etc/rc.local as
memcached -d -l localhost -p11211 >/var/log/memcached.log 2>&1
That will give you a log file with possible errors.
Lastly, does your install of memcached not have an init.d file in /etc/init.d ?
if it does, simply do chkconfig servicename on && service servicename start

Set Docker_Opts in centos

I need to set docker to listen to tcp://0.0.0.0/4243 on my host machine running amazon linux (centos). All the documentation I have seen has told me to run the following command
echo DOCKER_OPTS="-H=tcp://127.0.0.1:4243" >> /etc/default/docker
Which will write the correct docker_opts to /etc/default/docker. I've done this, but when I restart docker it does not listen to 127.0.0.1
I can make docker run correctly by typing
sudo /usr/bin/docker -H tcp://0.0.0.0:4243 -d &
That works, but I want the default option to be listening on tcp://0.0.0.0:4243 without having to specify it every time.
It seems that docker is completely ignoring my /etc/default/docker file so the settings are being ignored. I also tried writing the file to /etc/default/docker.io and /etc/default/docker-io (didn't really expect much to happen)
I need to be able to start docker with just
service docker start
or it will cause issues in my current deployment playbook.
Any thoughts on what I can do to set DOCKER_OPTS and not have to do it every time I restart docker?
In RHEL7, instead of modifying your docker.service unit file, you can also just edit your /etc/sysconfig/docker file:
# /etc/sysconfig/docker
# Modify these options if you want to change the way the docker daemon runs
OPTIONS=--selinux-enabled -H unix:///var/run/docker.sock -H tcp://0.0.0.0:4243
and then restart your docker service.
To me, this is more reliable than modifying the service script.
For CentOS 7 (RHEL 7):
Find the systemd docker.service unit file. Mine is located at: /usr/lib/systemd/system/docker.service
In this file, edit the line in the [Service] section beginning with ExecStart=. Add the "-H tcp://0.0.0.0:4243" into the line. (notice there's no "=" between the -H and the IP address as in your example DOCKER_OPTS line above.)
On my system, the entire contents of docker.service then looks like:
[Unit]
Description=Docker Application Container Engine
Documentation=http://docs.docker.com
After=network.target docker.socket
Requires=docker.socket
[Service]
Type=notify
EnvironmentFile=-/etc/sysconfig/docker
ExecStart=/usr/bin/docker -d -H tcp://127.0.0.1:4243 -H fd:// $OPTIONS
LimitNOFILE=1048576
LimitNPROC=1048576
[Install]
Also=docker.socket
(I only need Docker to listen on the loopback, instead of all IP addresses.)
After making this edit to the systemd unit file and restarting the Docker service via systemctl restart docker, I see the following process:
root 8574 0.0 0.2 321708 10564 ? Ssl 00:42 0:00 /usr/bin/docker -d -H tcp://127.0.0.1:4243 -H fd:// --selinux-enabled
As you can see, it does now listen on the configured TCP address, and will persist over reboots and service stop/starts.
I cannot believe how many answers there are for this. So here is another one for:
CentOS 7.3
Docker Version = 17.03.1-ce, API Version = 1.27
This answer is built upon an unbelievable playing around combination of this answer and this one and this one.
sudo vim /usr/lib/systemd/system/docker.service
insert " -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock"
sudo systemctl daemon-reload //refresh your file changes above
sudo systemctl restart docker
netstat -l | grep 4243 //verify port is open
connect to your docker host from somewhere, like Jenkins Docker Plugin, i.e. tcp://[server_ip]:4243
Editing /etc/docker/daemon.json seems to be the new, supported way.
With Docker 1.7.1 on CentOS 7 neither modifying /usr/lib/systemd/system/docker.service or /etc/sysconfig/docker worked for me. It seems that in systemd sets up the socket, so in order to change the group you have to edit SocketGroup in /usr/lib/systemd/system/docker.socket
[Unit]
Description=Docker Socket for the API
PartOf=docker.service
[Socket]
ListenStream=/var/run/docker.sock
SocketMode=0660
SocketUser=root
SocketGroup=jenkins
[Install]
WantedBy=sockets.target
I Think on CentOS, you can try setting the options as below in the file /etc/sysconfig/docker
other_args="-H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock"
Then restart the docker and try checking if the port is opening using
netstat -plt | grep 4243
This should list if listening
I needed to change the default bridge interface docker0 to use my own bridge interface br0 and putting the following content in that file solved my issue:
CentOS 7.2 and docker 1.10.3
/usr/lib/systemd/system/docker.service.d/docker.conf
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon --bridge=br0 -H fd://
and of course the following need to be performed after:
sudo systemctl daemon-reload
sudo systemctl restart docker
ip link del docker0
It worked for me when I followed how its shown in the posts above with drop-in replacement files in: /etc/systemd/system/docker.service.d
I am working on centos 7.
I just want to add insecure-registry in docker config file then I changed "DOCKER_OPTS=--insecure-registry=...." in /etc/sysconfig/docker while it did not work.
While I saw a INSECURE_REGISTRY in the config so I changed this variable and it WORKS!
So I guess DOCKER_OPTS does not work here!
But it worked on my unbuntu 14!
It is really frustrating when using docker!
Based on https://docs.docker.com/engine/admin/configuring/
sudo mkdir /etc/systemd/system/docker.service.d
sudo vi /etc/systemd/system/docker.service.d/docker.conf
[Service]
ExecStart=
ExecStart=/usr/bin/docker daemon -H fd:// -D -H tcp://127.0.0.1:4243
sudo systemctl daemon-reload
sudo systemctl restart docker
1、edit /usr/lib/systemd/system/docker.service to add two param in the service section:
# vim /usr/lib/systemd/system/docker.service
[Service]
ExecStart=
ExecStart=/usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix://var/run/docker.sock
2、reload the configuration,and then restart docker。
# systemctl daemon-reload
# systemctl restart docker
3、to check for success, see if the return the following response。
# ps -ef|grep docker
root 26208 1 0 23:51 ? 00:00:00 /usr/bin/dockerd -H tcp://0.0.0.0:2375 -H unix://var/run/docker.sock
reference from Expose the Docker Remote API on Centos 7?
I believe things have changed now, this answer by Brian Ogden had worked for me earlier but didn't work on my environment today, probably with the updated versions of the docker, kernel, and OS.
CentOS 7.4.1708 (on AWS)
Docker 17.03.2-ce
API 1.27
This is what worked after few hit and trials. I could not find it documented anywhere.
In file /etc/systemd/system/docker.service.d/execstart.conf, replace the last ExecStart (there are two) with below
ExecStart=/usr/bin/dockerd --graph=/var/lib/docker --storage-driver=overlay -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock
Now, my files looks like this
# cat /etc/systemd/system/docker.service.d/execstart.conf
[Service]
Restart=always
StartLimitInterval=0
RestartSec=15
ExecStartPre=-/sbin/ip link del docker0
ExecStart=
ExecStart=/usr/bin/dockerd --graph=/var/lib/docker --storage-driver=overlay -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock
#
Once, the above file is changed just the run the below command to activate the changes.
# systemctl daemon-reload && systemctl stop docker && rm -f /var/run/docker.sock && systemctl start docker
To verify if everything is working fine, you can run any (or all) of below commands
# systemctl status docker.service | grep tcp
├─21812 /usr/bin/dockerd --graph=/var/lib/docker --storage-driver=overlay -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock
#
# netstat -an | grep 4243
tcp6 0 0 :::4243 :::* LISTEN
#
# ps aux | grep [4]243
root 21812 1.0 0.8 1017512 67876 ? Ssl 15:11 0:06 /usr/bin/dockerd --graph=/var/lib/docker --storage-driver=overlay -H tcp://0.0.0.0:4243 -H unix:///var/run/docker.sock
#
# docker -H :4243 info