I'm new on GCP and on linux and I try to mount a bucket on my centos instance using gcsfuse.
I tried with a script running at boot but it was not working so I tried with fstab (peoples told me it is much better)
But I got this error when I tried to ls my mounted point :
ls: reading directory .: Input/output error
here is my fstab file :
#
# /etc/fstab
# Created by anaconda on Tue Mar 26 23:07:36 2019
#
# Accessible filesystems, by reference, are maintained under'/dev/disk'
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info
#
UUID=de2d3dce-cce3-47a8-a0fa-5bfe54e611ab / xfs defaults 0 0
mybucket /mount/to/point gcsfuse rw,allow_other,uid=1001,gid=1001
According : https://github.com/GoogleCloudPlatform/gcsfuse/blob/master/docs/mounting.md
Thanks for your time.
Okay so I just had to wait 2 minutes due to google auth granting my key. Basically it works
Related
I'm trying to upload a file to cloud object storage from IBM Analytics Engine:
$ hadoop fs -copyFromLocal LICENSE-2.0.txt \
cos://xxxxx/LICENSE-2.0.txt
However, I'm receiving warnings about failure to create disks:
18/01/26 17:47:47 WARN fs.LocalDirAllocator$AllocatorPerContext:
Failed to create /disk1/s3a 18/01/26 17:47:47 WARN
fs.LocalDirAllocator$AllocatorPerContext: Failed to create /disk2/s3a
Note even though I recieve this warning, the file is still uploaded:
$ hadoop fs -ls cos://xxxxx/LICENSE-2.0.txt
-rw-rw-rw- 1 clsadmin clsadmin 11358 2018-01-26 17:49 cos://xxxxx/LICENSE-2.0.txt
The problem seems to be:
$ grep -B2 -C1 'disk' /etc/hadoop/conf/core-site.xml
<property>
<name>fs.s3a.buffer.dir</name>
<value>/disk1/s3a,/disk2/s3a,/tmp/s3a</value>
</property>
$ ls -lh /disk1 /disk2
ls: cannot access /disk1: No such file or directory
ls: cannot access /disk2: No such file or directory
What are the implications of these warnings? The /tmp/s3a folder does exist, so can we ignore the warnings about these other folders?
The hadoop property 'fs.s3a.buffer.dir' supports list (comma separated values)and points to a local path. When the path is missing, the warnings do appear but they can be safely ignored since they are harmless.If the same command had been run from within the data node, the warning would not show up.Regardless of the warning, the file will be copied to Cloud Object Store, hence does not have any other impact.
Idea to have multiple values for fs.s3a.buffer.dir to'/disk1/s3a,/disk2/s3a,/tmp/s3a' is that when hadoop jobs are run on cluster with Cloud Object Store, the map-reduce tasks are scheduled on data nodes which has additional disks viz /disk1 and /disk2 which has more disk capacity compared to management nodes.
I tried 30 day trial of dashDB Local. I followed the steps described in the link:
https://www.ibm.com/support/knowledgecenter/en/SS6NHC/com.ibm.swg.im.dashdb.doc/admin/linux_deploy.html
I did not create a node configuration file because mine is a SMP setup.
Logged into my docker hub account and pulled the image.
docker login -u xxx -p yyyyy
docker pull ibmdashdb/local:latest-linux
The pull took 5 minutes or so. I waited for the image download to complete.
Ran the following command. It completed successfully.
docker run -d -it --privileged=true --net=host --name=dashDB -v /mnt/clusterfs:/mnt/bludata0 -v /mnt/clusterfs:/mnt/blumeta0 ibmdashdb/local:latest-linux
ran logs command
docker logs --follow dashDB
This showed dashDB did not start but exited with error code 130
# docker ps -a
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
0f008f8e413d ibmdashdb/local:latest-linux "/usr/sbin/init" 16 seconds ago Exited (130) 1 seconds ago dashDB
#
logs command shows this:
2017-05-17T17:48:11.285582000Z Detected virtualization docker.
2017-05-17T17:48:11.286078000Z Detected architecture x86-64.
2017-05-17T17:48:11.286481000Z
2017-05-17T17:48:11.294224000Z Welcome to dashDB Local!
2017-05-17T17:48:11.294621000Z
2017-05-17T17:48:11.295022000Z Set hostname to <orion>.
2017-05-17T17:48:11.547189000Z Cannot add dependency job for unit systemd-tmpfiles-clean.timer, ignoring: Unit is masked.
2017-05-17T17:48:11.547619000Z [ OK ] Reached target Timers.
<snip>
2017-05-17T17:48:13.361610000Z [ OK ] Started The entrypoint script for initializing dashDB local.
2017-05-17T17:48:19.729980000Z [100209.207731] start_dashDB_local.sh[161]: /usr/lib/dashDB_local_common_functions.sh: line 1816: /tmp/etc_profile-LOCAL.cfg: No such file or directory
2017-05-17T17:48:20.236127000Z [100209.713223] start_dashDB_local.sh[161]: The dashDB Local container's environment is not set up yet.
2017-05-17T17:48:20.275248000Z [ OK ] Stopped Create Volatile Files and Directories.
<snip>
2017-05-17T17:48:20.737471000Z Sending SIGTERM to remaining processes...
2017-05-17T17:48:20.840909000Z Sending SIGKILL to remaining processes...
2017-05-17T17:48:20.880537000Z Powering off.
So it looks like start_dashDB_local.sh is failing at /usr/lib/dashDB_local_common_functions.sh 1816th line? I exported the image and this is the 1816th line of dashDB_local_common_functions.sh
update_etc_profile()
{
local runtime_env=$1
local cfg_file
# Check if /etc/profile/dashdb_env.sh is already updated
grep -q BLUMETAHOME /etc/profile.d/dashdb_env.sh
if [ $? -eq 0 ]; then
return
fi
case "$runtime_env" in
"AWS" | "V1.5" ) cfg_file="/tmp/etc_profile-V15_AWS.cfg"
;;
"V2.0" ) cfg_file="/tmp/etc_profile-V20.cfg"
;;
"LOCAL" ) # dashDB Local Case and also the default
cfg_file="/tmp/etc_profile-LOCAL.cfg"
;;
*) logger_error "Invalid ${runtime_env} value"
return
;;
esac
I also see /tmp/etc_profile-LOCAL.cfg in the image. Did I miss any step here?
I also created /mnt/clusterfs/nodes file ... but it did not help. The same docker run command failed in the same way.
Please help.
I am using x86_64 Fedora25.
# docker version
Client:
Version: 1.12.6
API version: 1.24
Package version: docker-common-1.12.6-6.gitae7d637.fc25.x86_64
Go version: go1.7.4
Git commit: ae7d637/1.12.6
Built: Mon Jan 30 16:15:28 2017
OS/Arch: linux/amd64
Server:
Version: 1.12.6
API version: 1.24
Package version: docker-common-1.12.6-6.gitae7d637.fc25.x86_64
Go version: go1.7.4
Git commit: ae7d637/1.12.6
Built: Mon Jan 30 16:15:28 2017
OS/Arch: linux/amd64
#
# cat /etc/fedora-release
Fedora release 25 (Twenty Five)
# uname -r
4.10.15-200.fc25.x86_64
#
Thanks for bringing this to our attention. I reached out to our developer team. It seems this is happening because inside the container, tmpfs gets mounted on to /tmp and wipes out all the scripts
We have seen this issue and moving to the latest version of docker seems to fix it. Your docker version commands shows it is an older version.
So please install the latest docker version and retry the deployment of dashdb Local and update here.
Regards
Murali
I posted this as an answer on the original question posed on
postgreSQL permission denied when reading from file with \i command
But I now realise I should have posted as a separate question.
My OS is Fedora 21, I have installed PostgreSQL-9.4
I am trying to read a file using the \i command and error message is Permission denied. I have chmod a+r on the file.
On my system there are 3 users [root, damo, postgres] and I have established a group [project] that has 2 members [damo, postgres]. I have used chgrp on all relevant directories which (I believe) should grant permission to user [postgres] to access various files.
I access pqsql with the user [postgres] but all of my other work is under user [damo]. Within pgsql using the tab to navigate from the command \i stops after
../../home/damo
and I can go no further. This obviously important but I don't know what it means. So I moved the file to that location and it still does not load. This is the output from ls -l and stat
$ ls -l testScript.sql
-rw-r--r--. 1 damo project 76 Nov 5 18:18 testScript.sql
$ stat testScript.sql
File: ‘testScript.sql’
Size: 76 Blocks: 8 IO Block: 4096 regular file
Device: fd02h/64770d Inode: 2623547 Links: 1
Access: (0644/-rw-r--r--) Uid: ( 1000/ damo) Gid: ( 1001/ project)
Context: unconfined_u:object_r:user_home_t:s0
Access: 2015-11-05 18:18:06.082928881 +0000
Modify: 2015-11-05 18:18:06.084928866 +0000
Change: 2015-11-05 18:25:57.041183384 +0000
Birth: -
Can anyone advance some suggestions?
You're on Fedora 21, so you likely have SELinux enabled by default. The postgres user probably doesn't have the rights to access the security context user_home_t. The security context is shown by your stat output. You can include it in ls using ls -lZ.
The selinux boolean postgresql_selinux_unconfined_dbadm looks like what you want for that, per getsebool -a.
Check
getsebool postgresql_selinux_unconfined_dbadm
and if it's off, try:
setsebool postgresql_selinux_unconfined_dbadm on
Also, to determine whether selinux is what's denying access, check the system logs or run the sealert tool (SELinux alert browser).
Apart from that, it could well be that one of the directories in the absolute path of the file denies access to your operating system user.
I'm the operator of the XMPP server on darkness.su.The server runs on Centos 6.
I installed TOR and configured it to provide a hidden service access to the server.It was working fine at first,but ever since an update a few months ago it started giving me these errors:
799 May 25 14:19:37.060 [warn] Permissions on directory /var/lib/tor/hidden_service are too permissive.
800 May 25 14:19:37.060 [warn] Failed to parse/validate config: Failed to configure rendezvous options. See logs for details.
801 May 25 14:19:37.060 [err] Reading config failed--see warnings above.
I tried to check the logs,but I can't find them,and setting one doesn't seem to work.I've tried removing TOR and wiping all its folder,then reinstalling it.Same thing.
I'm installing through yum from TOR Project's repository.
With chmod 700 on the hidden service directory(owned by TOR):
Jul 24 21:39:05.573 [warn] Directory /var/lib/tor/hidden_service/ cannot be read: Permission denied
Jul 24 21:39:05.573 [warn] Failed to parse/validate config: Failed to configure rendezvous options. See logs for details.
Jul 24 21:39:05.573 [err] Reading config failed--see warnings above
After changing directory owner to root:
Jul 24 22:11:36.236 [warn] /var/lib/tor/hidden_service/ is not owned by this user (_tor, 496) but by root (0). Perhaps you are running Tor as the wrong user?
Jul 24 22:11:36.236 [warn] Failed to parse/validate config: Failed to configure rendezvous options. See logs for details.
Jul 24 22:11:36.236 [err] Reading config failed--see warnings above.
Permissions on directory /var/lib/tor/hidden_service are too permissive.
This means, that too many users have access to this directory. Try to change it:
chmod 700 /var/lib/tor/hidden_service
I assume here that the user running TOR is also the owner of the directory.
Your initial problem with permission issues (I had these after cloning a virtual hdd in VirtualBox) was caused by broken labels in selinux. On CentOS/Linux this is fixed with:
restorecon -r -v /var/lib/tor
It is all about file and directory permissions. I wrote this in Dockerfile
FROM osminogin/tor-simple:0.4.6.7
ARG source=.
USER tor
COPY $source/torrc /etc/tor/torrc
RUN mkdir /var/lib/tor/sc && chmod 700 /var/lib/tor/sc
COPY --chown=tor:nogroup $source/private/* /var/lib/tor/sc
RUN chmod -R 400 /var/lib/tor/sc/*
In my sc directory I have hostname and key pair.
After restarting the container tor domain name persists
sudo chown _tor:_tor /var/lib/tor/site/
fixed it for me.
I'm new to web development and I wanted to get started with some RoR (using Locomotive CMS).
One of the things Locomotive asks for is to have Mongodb. I installed using homebrew by following this link http://docs.mongodb.org/manual/tutorial/install-mongodb-on-os-x/
It installs fine but then im not able to run it!
When I type 'mongo' on terminal I get the following output :
"MongoDB shell version: 2.4.3
connecting to: test
Mon May 6 11:12:28.927
JavaScript execution failed:
Error: couldn't connect to server
127.0.0.1:27017 at src/mongo/shell/mongo.js:L112
exception: connect failed"
BACKGROUND TO HELP DEBUGGING ( on Terminal) :
1.When I type in mongod I get the following :
"all output going to: /usr/local/var/log/mongodb/mongo.log"
Ownership of mongo.log :
-rw-r--r-- 1 username admin 22133 May 6 11:13 mongo.log
2.When I input mongod --fork I get the following :
about to fork child process, waiting until server is ready for connections.
forked process: 77566
all output going to: /usr/local/var/log/mongodb/mongo.log
ERROR: child process failed, exited with error number 100
3.Typing mongod --help gives the following warning:
* WARNING: soft rlimits too low. Number of files is 256, should be at least 1000
4.I have a folder called data (which acts as amongodb database, is this where it should be?)in root (PATH : /data) Ownership of data folder :
"drwxr-xr-x 3 username wheel 102 Apr 23 21:38 data"
5.Checking if ports are free: lsof -i :27017. Ive also tried to check for a running mongo process using activity montior and found zilch!
No output
6.Ive also tried : mongo --repair. Dint help!
Ive been stuk on this for a while, I've looked at most responses on stackoverflow and searched around to find a solution to this but nothing has helped so far!
UPDATE:
When I tried to start the mongo shell, I was getting the following l
log message from mongo.log:
5/6/13 1:33:27.616 PM com.apple.launchd:
(org.mongodb.mongod[79133])
open("/private/var/log/mongodb/output.log", ...): Permission denied
So I did a chmod777 for the particular folder and the shell launches!
Although I still get a warning when it launches as:
Server has startup warnings:
Mon May 6 13:33:27.693 [initandlisten]
Mon May 6 13:33:27.693 [initandlisten]
** WARNING: soft rlimits too low.
Number of files is 256, should be at least 1000
Any idea how I can silence these warnings?
To get the information you need to determine the cause of failure you need to look in (and post for us) the output from /usr/local/var/log/mongodb/mongo.log when it is trying to start.
However, the most common reason for the failure is the lack of the default database path - at /data/db. Either create that folder (and don't forget to make sure your user has permission to read/write to it) or specify a different path with the --dbpath option.
UPDATE: as you have since found, bad permissions on the log file can cause the issue, in a similar way to bad permissions on the data path.
In terms of the warning, the information you need is here:
https://superuser.com/questions/433746/is-there-a-fix-for-the-too-many-open-files-in-system-error-on-os-x-10-7-1
It is just that though, a warning - you can run MongoDB without an issue with those limits as long as it is not under heavy load. So, if this is a development environment, unless you plan on load testing, you should be fine