About
I'm trying to use presto plugin like wyukawa/presto-fluentd, it works on localhost(mac os x), but does not work on Amazon EMR.
Detail
on localhost
At first, I tried to work on localhost(mac os x) and just it works.
plugin dir
reizist ...plugin/presto-fluentd $ pwd
/usr/local/Cellar/presto/0.185/libexec/plugin/presto-fluentd
reizist ...plugin/presto-fluentd $ ls -1
fluency-1.3.0.jar
guava-21.0.jar
jackson-annotations-2.8.1.jar
jackson-core-2.7.1.jar
jackson-databind-2.7.1.jar
jackson-dataformat-msgpack-0.8.12.jar
jolokia-jvm-1.3.7-agent.jar
log-0.148.jar
msgpack-core-0.8.12.jar
phi-accural-failure-detector-0.0.4.jar
presto-fluentd-0.0.1.jar
slf4j-api-1.7.22.jar
properties
reizist ...libexec/etc $ pwd
/usr/local/Cellar/presto/0.185/libexec/etc
reizist ...libexec/etc $ ls -1
catalog
config.properties
event-listener.properties
jvm.config
log.properties
node.properties
reizist ...libexec/etc $ cat event-listener.properties
event-listener.name=presto-fluentd
event-listener.fluentd-host=localhost
event-listener.fluentd-port=24224
event-listener.fluentd-tag=presto.query
left: presto log, center: fluentd log, right: presto-cli
on EMR
Also I tried same on EC2 on EMR, but it did not work.
That plugin is correctly loaded, event listener registered, so I feel strange.
plugin dir
[hadoop#ip-172-31-29-54 plugin]$ pwd
/usr/lib/presto/plugin
[hadoop#ip-172-31-29-54 plugin]$ ls
accumulo cassandra jmx memory mysql redis tpch
atop example-http kafka ml postgresql resource-group-managers
blackhole hive-hadoop2 localfile mongodb presto-fluentd teradata-functions
[hadoop#ip-172-31-29-54 plugin]$ ls -1 presto-fluentd/
fluency-1.3.0.jar
guava-21.0.jar
jackson-annotations-2.8.1.jar
jackson-core-2.7.1.jar
jackson-databind-2.7.1.jar
jackson-dataformat-msgpack-0.8.12.jar
log-0.148.jar
msgpack-core-0.8.12.jar
phi-accural-failure-detector-0.0.4.jar
presto-fluentd-0.0.1.jar
slf4j-api-1.7.22.jar
properties
[hadoop#ip-172-31-29-54 presto]$ pwd
/etc/presto
[hadoop#ip-172-31-29-54 presto]$ tree .
.
├── conf -> /etc/alternatives/presto-conf
├── conf.dist
│ ├── catalog
│ │ ├── hive.properties
│ │ └── mysql.properties
│ ├── config.properties
│ ├── jvm.config
│ ├── log.properties
│ ├── node.properties
│ └── presto-env.sh
└── event-listener.properties
3 directories, 8 files
[hadoop#ip-172-31-29-54 presto]$ cat event-listener.properties
event-listener.name=presto-fluentd
event-listener.fluentd-host=localhost
event-listener.fluentd-port=24224
event-listener.fluentd-tag=presto.query
I also tested by inserting print debugging code, but it looks like not loaded.
How I should to work this plugin on EMR?
Thanks.
supplement
here is the fluentd configuration.
<source>
#type forward
</source>
<match *.**>
#type stdout
</match>
I resolved by my own.
Actually I have to locate event-listener.properties on /mnt/var/lib/presto/data/etc , so I did this:
$s3uri="s3://my-s3-bucket"
# make symbolic link
sudo mkdir /usr/lib/presto/etc
sudo ln -s /usr/lib/presto/etc /mnt/var/lib/presto/data
# download presto plugins
aws s3 sync $s3uri/jar/ /usr/lib/presto/plugin/
aws s3 sync $s3uri/properties/ /usr/lib/presto/etc/
# make sure all plugins are owned by presto user
chown -R presto:presto /usr/lib/presto/plugin
chown -R presto:presto /usr/lib/presto/etc
# restart presto
stop presto-server
start presto-server
finally my dir is like below:
[hadoop#ip-172-31-21-25 presto]$ pwd
/usr/lib/presto
[hadoop#ip-172-31-21-25 presto]$ ls -alh
total 228K
drwxr-xr-x 7 root root 4.0K Oct 19 06:52 .
dr-xr-xr-x 47 root root 4.0K Oct 19 06:30 ..
drwxr-xr-x 3 presto presto 4.0K Oct 19 06:30 bin
drwxr-xr-x 3 presto presto 4.0K Oct 19 06:52 etc
drwxr-xr-x 2 presto presto 12K Oct 19 06:30 lib
-rw-r--r-- 1 presto presto 188K Sep 22 22:54 NOTICE
drwxr-xr-x 24 presto presto 4.0K Oct 19 06:45 plugin
drwxr-xr-x 2 presto presto 4.0K Oct 19 06:30 presto-jdbc
-rw-r--r-- 1 presto presto 119 Sep 22 22:54 README.txt
[hadoop#ip-172-31-21-25 etc]$ pwd
/usr/lib/presto/etc
[hadoop#ip-172-31-21-25 etc]$ ls
event-listener.properties
[hadoop#ip-172-31-21-25 plugin]$ pwd
/usr/lib/presto/plugin
[hadoop#ip-172-31-21-25 plugin]$ ls
accumulo example-http localfile mysql redis tpcds
atop hive-hadoop2 memory postgresql resource-group-managers tpch
blackhole jmx ml presto-fluentd sqlserver
cassandra kafka mongodb presto-thrift teradata-functions
and it works correctly.
Related
I am trying to configure an user account using one cloud-init yaml file that include a call to write_files module, like this:
write_files:
#passwd file for vncserver
- path: /home/ubuntu/.vnc/passwd
owner: ubuntu:ubuntu
permissions: '0600'
defer: true
encoding: b64
content: bmtzZGN1eQo=
The file is created as expected, but the problem is that the parent directory is owned by root, and not by ubuntu user.
$ ls -la .vnc/
total 12
drwxr-xr-x 2 root root 4096 Dec 20 16:24 .
drwxr-x--- 5 ubuntu ubuntu 4096 Dec 20 16:24 ..
-rw------- 1 ubuntu ubuntu 8 Dec 20 16:24 passwd
I tried to manually create the /home/ubuntu/.vnc/ directory prior to create the passwd file to be able to set the ownership of the directory, just to find that documentation of write_files does not explain how to create (empty) directories.
I know that I could do this using runcmd module to insert a command like this:
runcmd:
- mkdir --mode 0600 --parents /home/ubuntu/.vnc
- echo bmtzZGN1eQo | base64 -d > /home/ubuntu/.vnc/passwd
- chmod 0600 /home/ubuntu/.vnc/passwd
but this seems to be too complex to do such small task.
It is possible to use write_files module to create directories or change ownership/permission of existing directories?
I am having weird issues with official postgres docker image. Most of the time it works fine, if I shut down the container and launch it again, I sometimes get this error but it's not every time:
PostgreSQL Database directory appears to contain a database; Skipping initialization
postgres: could not access the server configuration file "/var/lib/postgresql/data/postgresql.conf": No such file or directory
I am launching postgres image using this command:
export $(grep -v '^#' .env | xargs) && docker run --rm --name postgres \
-e POSTGRES_USER=$POSTGRES_USER \
-e POSTGRES_DB=$POSTGRES_DB \
-e POSTGRES_PASSWORD=$POSTGRES_PASSWORD \
-p $POSTGRES_PORT:$POSTGRES_PORT \
-v $POSTGRES_DEVELOPMENT_DATA:/var/lib/postgresql/data \
postgres
I keep variables in .env file, they look like this:
POSTGRES_USER=custom-db
POSTGRES_DB=custom-db
POSTGRES_PASSWORD=12345678
POSTGRES_PORT=5432
POSTGRES_DEVELOPMENT_DATA=/tmp/custom-db-pgdata
When I try to echo variables the values are there so I don't think I'm passing null values to docker env variables.
The directory on my host machine looks something like this:
/tmp/custom-db-pgdata
├── base
│ ├── 1
│ ├── 13407
│ ├── 13408
│ └── 16384
├── global
├── pg_logical
├── pg_multixact
│ ├── members
│ └── offsets
├── pg_notify
├── pg_stat
├── pg_stat_tmp
├── pg_subtrans
├── pg_wal
│ └── archive_status
└── pg_xact
If it's inconsistent in how it works between executions on the same machine and same session (aka without rebooting) then something isn't mapping your directories properly. Finding what it is that's breaking will be difficult, more so since you're on a Mac. Docker on a Mac you has the extra bonus of running through a VM, so docker is mapping your local drive/path through to the VM and then mapping that into the container image, so there are two different layers where things can go wrong.
Dario has the right idea in his clarifying comments, you shouldn't rely on /tmp since that also has Mac Magic to it. It's actually /var/private/somegarbagestring and is different every bootup. Try switching to a /Users/$USER/dbpath folder and move your data to that, so at least you're debugging with one less layer of magic between data and database.
After configuring a standalone Concourse 2.4.0 per the instructions, everything seems to be up and running. However, when trying to run the "hello world" example, I can see the following error in the Concourse UI:
runc create: exit status 1: rootfs ("/volumes/live/a72f9a0d-3506-489b-5b9b-168744b892c1/volume") does not exist
"web" start command:
./concourse web \
--basic-auth-username admin \
--basic-auth-password admin \
--session-signing-key session_signing_key \
--tsa-host-key host_key \
--tsa-authorized-keys authorized_worker_keys \
--external-url http://myconcoursedomain:8080 \
--postgres-data-source postgres://user:pass#mydbserver/concourse
"worker" start command:
./concourse worker \
--work-dir worker \
--tsa-host 127.0.0.1 \
--tsa-public-key host_key.pub \
--tsa-worker-private-key worker_key
I'm wondering if the problem occurs since the "missing" directory is created in the directory specified in the "start worker" command, instead of at the actual root directory:
~/concourse# ls -la worker
total 145740
drwxr-xr-x 5 root root 4096 Nov 15 23:07 .
drwxr-xr-x 3 root root 4096 Nov 15 23:07 ..
drwxr-xr-x 3 root root 4096 Nov 15 23:07 2.4.0
drwxr-xr-x 2 root root 4096 Nov 15 23:09 depot
drwxr-xr-x 1 root root 24 Nov 15 23:07 volumes
-rw-r--r-- 1 root root 42142052352 Nov 15 23:15 volumes.img
Concourse is installed on Ubuntu 14.04:
uname -r
4.4.0-47-generic
uname -a
Linux ubuntu-2gb-nyc3-01 4.4.0-47-generic #68~14.04.1-Ubuntu SMP Wed Oct 26 19:42:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
For reasons that I still do not understand, it appears that if you specify the --work-dir value to be /opt/concourse/worker, then the worker will work with this kernel version without issue.
I was using a relative path to a worker directory within a dir in my user folder as my --work-dir value.
I have a yum repository I've set up where I store custom rpms.
I have no problem finding information about other packages that were built and stored in this custom repo.
#yum --disablerepo=rhui-us-east-rhel-server-1y,epel,epel-testing --enablerepo=customrepo install php53-pecl-xdebug
php53-pecl-xdebug x86_64 2.2.1-2 customrepo 132 k
No problem.
Now I drop somerpm.rpm in centos/repo/5/noarch, run createrepo --update . in this directory and try the same command, and yet it shows no results.
I tried running createrepo --update in the root of the repo as well, but that did not work either (I'm actually not sure where to run it and if it needs a repodata directory in each subdir).
[root#reposerver mnt]# ls -l /var/www/repo/
total 12
-rw-r--r-- 1 root root 203 Jun 8 00:13 REPO_README
drwxr-xr-x 3 root root 4096 Jun 10 2011 centos
drwxr-xr-x 2 root root 4096 Oct 18 20:02 repodata
[root#reposerver mnt]# ls -l /var/www/repo/centos/5/
SRPMS/ i386/ noarch/ repodata/ x86_64/
[root#reposerver mnt]# ls -l /var/www/repo/centos/5/noarch/
total 7324
-rw-r--r-- 1 root root 1622 Jun 28 2011 compat-php-5.1.6-1.noarch.rpm
drwxr-xr-x 2 root root 4096 Oct 18 19:55 repodata
-rw-r--r-- 1 root root 1066928 Oct 18 19:54 salt-0.10.3-1.noarch.rpm
-rw-r--r-- 1 root root 6363197 Oct 18 19:54 salt-0.10.3-1.src.rpm
-rw-r--r-- 1 root root 21822 Oct 18 19:54 salt-master-0.10.3-1.noarch.rpm
-rw-r--r-- 1 root root 14294 Oct 18 19:54 salt-minion-0.10.3-1.noarch.rpm
I also tried adding the exactarch=0 flag to my repo config to ignore arch restrictions and this did not work either, it was a shot in the dark, since my rpm is noarch, it should show regardless.
# cat /etc/yum.repos.d/mycompany.repo
[mycompany]
name=mycompany custom repo
baseurl=http://config/repo/centos/5/$basearch
enabled=1
exactarch=0
I'm at a loss at this point. Usually createrepo --update does the trick, but for some reason it cannot find the new rpms.
repo]# find . -type f -name "*.gz" | xargs zcat | grep salt-minion
returns results as well, so it's definitely in the repo data.
yum clean all on the server I was trying to install on worked.
Also make sure to do createrepo --update on the specific subdir instead of the root of the repo.
By following the official instructions http://www.mongodb.org/display/DOCS/Quickstart+Unix and this post http://blog.phy5ics.com/2010/03/27/installing-mongodb-on-mediatemple-dv/ I've just about managed to get mongodb installed on MediaTemples DV 4.0 server (I think).
I am however having problems installing the PHP driver http://www.mongodb.org/display/DOCS/PHP+Language+Center
In SSH I get this:
[root#xxx]# cd /var/tmp
[root#xxx]# pecl install mongo
downloading mongo-1.1.4.tgz ...
Starting to download mongo-1.1.4.tgz (68,924 bytes)
.................done: 68,924 bytes
18 source files, building
running: phpize
Configuring for:
PHP Api Version: 20090626
Zend Module Api No: 20090626
Zend Extension Api No: 220090626
/usr/bin/phpize: /var/tmp/mongo/build/shtool: /bin/sh: bad interpreter: Permission denied
Cannot find autoconf. Please check your autoconf installation and the $PHP_AUTOCONF environment variable. Then, rerun this script.
ERROR: `phpize' failed
I am logged in as the root user - I don't understand why it's failing and what steps I need to take to install the PHP driver?
Thanks
Run the following commands on your server's command line:
$ mkdir /root/tmp
$ mount --bind /root/tmp /tmp
$ umount /tmp; umount /var/tmp
$ pecl install mongo
A few things:
/root/tmp is just an arbitrary temp directory. You can use whatever you want, provided it exists.
Some instructions say to use --host instead of --bind. On RHEL/CentOS mount says --host is an unrecognized option.
If you're on a VM, it's likely that you'll have to do this each time you restart your VM/Container.
For Media Temple customers, I can confirm that this works on both (dv) and (ve) servers with CentOS 5 and 6.
From media temple support: Need to create a temporary directory (/root/tmpz):
$ mkdir /root/tmpz
$ mount --host /root/tmpz /tmp
$ umount /tmp; umount /var/tmp
$ pecl install mongo
Build complete.
Don't forget to run 'make test'.
running: make INSTALL_ROOT="/var/tmp/pear-build-root/install-mongo-1.1.4" install
Installing shared extensions: /var/tmp/pear-build-root/install-mongo-1.1.4/usr/lib64/php /modules/
running: find "/var/tmp/pear-build-root/install-mongo-1.1.4" | xargs ls -dils
69094140 4 drwxr-xr-x 3 root root 4096 Feb 22 13:40 /var/tmp/pear-build-root/install-mongo-1.1.4
69275176 4 drwxr-xr-x 3 root root 4096 Feb 22 13:40 /var/tmp/pear-build-root/install-mongo-1.1.4/usr
69275177 4 drwxr-xr-x 3 root root 4096 Feb 22 13:40 /var/tmp/pear-build-root/install-mongo-1.1.4/usr/lib64
69290445 4 drwxr-xr-x 3 root root 4096 Feb 22 13:40 /var/tmp/pear-build-root/install-mongo-1.1.4/usr/lib64/php
69290447 4 drwxr-xr-x 2 root root 4096 Feb 22 13:40 /var/tmp/pear-build-root/install-mongo-1.1.4/usr/lib64/php/modules
69290448 676 -rwxr-xr-x 1 root root 684126 Feb 22 13:40 /var/tmp/pear-build-root/install-mongo-1.1.4/usr/lib64/php/modules/mongo.so
Build process completed successfully
Installing '/usr/lib64/php/modules/mongo.so'
install ok: channel://pecl.php.net/mongo-1.1.4
configuration option "php_ini" is not set to php.ini location
You should add "extension=mongo.so" to php.ini
Do you have php-dev installed? phpize is basically "compiling" the MongoDB driver, but unless you have the -dev installed, this may not work.