I'm using CentOS 7 via AWS.
I'd like to store MongoDB data on an attached EBS instead of the default /var/lib path.
However, when I edit /etc/mongod.conf to point to a new dbpath, I'm getting a permission denied error.
Permissions are set correctly to mongod.mongod on the dir.
What gives?
TL;DR - The issue is SELinux, which affects what daemons can access. Run setenforce 0 to temporarily disable.
You're using a flavour of Linux that uses SELinux.
From Wikipedia:
SELinux can potentially control which activities a system allows each
user, process and daemon, with very precise specifications. However,
it is mostly used to confine daemons[citation needed] like database
engines or web servers that have more clearly defined data access and
activity rights. This limits potential harm from a confined daemon
that becomes compromised. Ordinary user-processes often run in the
unconfined domain, not restricted by SELinux but still restricted by
the classic Linux access rights
To fix temporarily:
sudo setenforce 0
This should disable SELinux policies and allow the service to run.
To fix permanently:
Edit /etc/sysconfig/selinux and set this:
SELINUX=disabled
Then reboot.
The service should now start-up fine.
The data dir will also work with Docker, i.e. something like:
docker run --name db -v /mnt/path-to-mounted-ebs:/data/db -p 27017:27017 mongo:latest
Warning: Both solutions DISABLE the security that SELinux provides, which will weaken your overall security. A better solution is to understand how SELinux works, and create a policy on your new data dir that works with mongod. See https://wiki.centos.org/HowTos/SELinux for a more complete tutorial.
Related
I am attempting to debug some C# / .NET 5 code in WSL 2 with Ubuntu on Windows. I have WSL 2 setup with Windows 10 and want to test out creating a Systemd service. Unfortunately, it appears Systemd is not enabled with WSL 2 by default, even though a standard Ubuntu install does have it enabled by default. Is there any way to get Systemd enabled in WSL 2?
Note: See footnote at bottom of this answer for background on this Community Wiki.
There are several possible paths to enabling Systemd on WSL2 (but not WSL1). These are summarized here, with more detail provided below.
Option 1: Upgrade WSL to the latest application release (if supported by your system) and opt-in to the Systemd feature
Option 2: Run a Systemd-helper script designed for WSL2
Option 3: Manually run Systemd in its own namespace
And while not part of this question, for those simply looking to run certain applications that require Systemd, there are alternatives:
On WSL1 and WSL2:
Alternative 1: SysVInit scripts (e.g. sudo service <service_name> start) where available
Alternative 2: Manually configuring and running the service
On WSL2-only:
Alternative 3: Docker
Should you enable Systemd in WSL?
First, consider whether you should or need to enable Systemd in WSL. Enabling Systemd will automatically start a number of background services and tasks that you really may not need under WSL. As a result, it will also increase WSL startup times, although the impact will be dependent on your system. Check the Alternatives section below to see if there may be a better option that fits your needs. For example, the service command may do what you need without any additional effort.
More detail on each answer:
Option 1: Upgrade WSL to the latest application release (if supported by your system) and opt-in to the Systemd feature
Microsoft has now integrated Systemd support in the WSL2 application release (as opposed to the older "Windows feature" implementation).
Starting with WSL Application Release 1.0.0, this feature is available on both Windows 10 and Windows 11. Windows 10 users do need to be on UBR (update build revision) 2311 or later. The UBR is the last 4 digits of your full Windows build number (e.g. 10.0.19045.2311 for Windows 10 22H2). 2311 is installed with KB5020030, an optional Preview update, although if you are reading this later, it will likely be a later (non-Preview) monthly servicing update.
If you are on a supported Windows release, the WSL application with Systemd support can be installed:
Through the Microsoft Store (as "Windows Subsystem for Linux").
Or from the Releases page in the Github repo. To install a release manually:
Reboot (to make sure that WSL is not in use at all). A simple wsl --shutdown may work, but often will not.
Download the 1.0.0 (or later) release from the link above.
Start an Administrator PowerShell and:
Add-AppxPackage <path.to>/Microsoft.WSL_1.0.0.0_x64_ARM64.msixbundle
wsl --version # to confirm
To enable, start your Ubuntu (or other Systemd) distribution under WSL (typically just wsl ~ will work).
sudo -e /etc/wsl.conf
Add the following:
[boot]
systemd=true
Exit Ubuntu and again:
wsl --shutdown
Then restart Ubuntu.
sudo systemctl status
... should show your Systemd services.
Option 2: Run a Systemd-helper script designed for WSL2
There are a number of Systemd-enablement scripts available from various sources. Given the complexities involved in running Systemd under WSL, it is recommended that you:
Use one that is actively maintained
Attempt to understand, as much as possible, how they operate, and how they may impact other features and applications in your distribution(s) under WSL
When asking questions here or on any other site, disclose in the question which script you are using so that others can attempt to understand and/or reproduce your issue in the proper context
Several of the more popular projects that enable Systemd under WSL2 are:
Genie: 1.8k stars, last commit September, 2022
Distrod: 1.4k stars, last commit July 2022
WSL2-Hacks: 1.1k stars, mostly instructional, with a supporting script example. Last commit January, 2022
At the core, all of them operate on the same principles covered in the next option ...
Option 3: Manually run Systemd in its own namespace
One of the main issues with running Systemd in earlier versions of WSL is that both inits need to be PID 1. To get around this, it is possible to create a new namespace or container where Systemd can run as PID 1.
To see how this is done (at a very basic level):
Run:
sudo -b unshare --pid --fork --mount-proc /lib/systemd/systemd --system-unit=basic.target
This starts Systemd in a new namespace with its own PID mapping. Inside that namespace, Systemd will be PID1 (as it must, to function) and own all other processes. However, the "real" PID mapping still exists outside that namespace.
Note that this is a "bare minimum" command-line for starting Systemd. It will not have support for, at least:
Windows Interop (the ability to run Windows .exe)
The Windows PATH (which isn't necessary without Windows Interop anyway)
WSLg
The scripts and projects listed above do extra work to get these things working as well.
Wait a few seconds for Systemd to start up, then:
sudo -E nsenter --all -t $(pgrep -xo systemd) runuser -P -l $USER -c "exec $SHELL"
This enters the namespace, and you can now use ps -efH to see that systemd is running as PID 1 in that namespace.
At this point, you should be able to run systemctl.
And after proving to yourself that it's possible, it is recommended that you exit all WSL instances completely, then doing wsl --shutdown. Otherwise, some things will be "broken" until you do. They can likely be "fixed", but that's beyond the scope this answer. If you are interested, please refer to the projects listed above to see how they handle these situations.
Alternative 1: SysVInit scripts (e.g. sudo service <service_name> start) where available
In Ubuntu, Debian, and some other distributions on WSL, many of the common system services still have the "old" init.d scripts available to be used in place of systemctl with Systemd units. You can see these by using ls /etc/init.d/.
So, for example, you can start ssh with sudo service ssh start, and it will run the /etc/init.d/ssh script with the start argument.
Even some non-default packages such as MySql/MariaDB will install both the Systemd unit files and the old init.d scripts, so you can still use the service command for them as well.
On the hand, some packages, like Elasticsearch, only install Systemd units. And some distributions only provide Systemd units for most (if not all) packages in their repositories.
Alternative 2: Manually configuring and running the service
For those services that don't have a init-script equivalent, it can be possible to run them "manually".
For simplicity, let's assume that the ssh init.d script wasn't available.
In this case, the "answer" is to figure out what the Systemd unit files are doing and attempt to replicate that manually. This can vary widely in complexity. But I'd start with looking at the Systemd unit file that you are trying to run:
less /lib/systemd/system/ssh.service
# Trimmed
[Service]
EnvironmentFile=-/etc/default/ssh
ExecStartPre=/usr/sbin/sshd -t
ExecStart=/usr/sbin/sshd -D $SSHD_OPTS
RuntimeDirectory=sshd
RuntimeDirectoryMode=0755
Some of the less relevant lines have been trimmed to make it easier to parse, but you can man systemd.exec, man systemd.service, and others to see what most of the options do.
In this case, when you sudo systemctl start ssh, it:
Reads environment variables (the $SSHD_OPTS) from /etc/default/ssh
Tests the config, exits if there is a failure
Makes sure the RuntimeDirectory exists with the specified permissions. This translates to /run/sshd (from man systemd.exec). This also removes the runtime directory when you stop the service.
Runs /usr/sbin/sshd with options
So, if you don't have any environment-based config, you could just set up a script to:
Make sure the runtime directory exists. Note that, since it is in /run, which is a tmpfs mount, it will be deleted after every restart of the WSL instance.
Set the permissions to 0755
Start /usr/sbin/sshd as root
... And you would have done the same thing manually without Systemd.
Again, this is probably the simplest example. You might have much more to work through for more complex tasks.
Alternative 3: Docker
Many packages/services are available as Docker images. Docker typically runs very well under Ubuntu on WSL2 (specifically WSL2; it will not run on WSL1). If there's not a SysVinit "service" script for the service you are trying to start, there may very well be a Docker image available that runs in a containerized environment.
Example: Elasticsearch, as in this question.
Bonus #1: Doesn't interfere with other packages already installed (no dependency issues).
Bonus #2: The Docker images themselves pretty much never use Systemd, so you can often inspect the Dockerfile to see how the service is started without Systemd. For more information see the next option - "The manual way."
Microsoft recommends Docker Desktop for Windows for running Docker containers under WSL2.
Footnote This answer is being posted as a Community Wiki because it can apply to multiple Stack Overflow questions. It is originally based on answers to this Ask Ubuntu question. However, it is hoped that this wiki-answer can be continuously updated by the community as Systemd evolves on WSL.
This question has been chosen since:
It appears to be the most canonical, straightforward, "How do I enable Systemd on WSL?" question.
It is on-topic, as *creating Systemd services is (or at least can-be) unique to programming.
I am building a docker image off postgres image, and I would like to seed it with some data.
I am following the initialization-scripts section of the documentation.
But the problem I am facing now, is that my initialisation scripts needs to ssh to a remote database and dumb data from there. Basically something like this:
ssh remote.host "pg_dump -U user -d somedb" > some.sql
but this fails with the error that ssh: command not found
Question now is, in general, how do I ssh from a docker container to a remote server. In this case, specifically how do I ssh from a docker container to a remote database server as part of the initialisation step of seeding a postgres database?
As a general rule you don't do things this way. Typical Docker images contain only the server they're running and some core tools, but network clients like ssh or curl generally aren't part of this. In the particular case of ssh, securely managing the credentials required is also tricky (not impossible, but not obvious).
In your particular case, I might rearrange things so that your scripts didn't have the hard assumption the database was running locally. Provision an empty database container, then run your script from the host targeting that empty database. It may even work to set the PGHOST and PGPORT environment variables to point to your host machine's host name and the port you publish the database interface on, and then run that script unmodified.
Looking closer at that specific command, you also may find it better to set up a cron job to run that specific database dump and put the contents somewhere. Then a developer can get a snapshot of the data without having to make a connection to the live database server, and you can limit the number of people who will have access. Once you have this dump file, you can use the /docker-entrypoint-initdb.d mechanism to cause it to be loaded at first startup time.
I am using Ubuntu 14.04 LTS and I have installed MongoDB 3.2.1. I had various problems with it that I fixed by either finding help from the internet (thanks Google) or by uninstalling and re-installing MongoDb.
One persistent problem that I cannot fix, unless I re-install, is by running mongod on the terminal. Currently my mongodb installation is working properly. With sudo service mongod start I start mongodb (I commented out start on xxxx line at /etc/init/mongod.conf so it doesn't auto start.) and with sudo service mongod stop I can stop it normally atm. And it correctly uses /var/lib/mongodb path for saving my collections.
If however after I stop mongodb with sudo service mongod stop I hit, on the terminal, mongod then mongodb breaks. I have gotten all kinds of errors like
Unusable mongod.lock. I have gotten around it, in a previous install, by doing sudo mongod but that was not a proper solution.
/data/db folder not found !!. Why look for it in the first place? The /etc/mongod.conf specifies the dbpath /var/lib/mongodb which is also the default when mongodb gets installed !!?? I have gotten around it as well with the mongod --dbpath /var/lib/mongodb option.
I think there was one more error that I don't remember but I also fixed/gotten_around it by finding solutions online.
I know that by re-installing and by never running mongod I can - for now? - not face those problems again (which looks a much better solution that the "workarounds" I did whenever those problems arose. I am wondering however what it is that is going wrong when I type mongod instead of sudo service mongod start ?!
In case somebody tries to replicate the problem know that I ve done only two modifications on my system after installing:
Commenting out the start on xxxx line at /etc/init/mongod.conf
Disabling transparent hugepages as described on the answer here.
I don't think anyone of those should interfere with my installation.
Can anyone help me understand what is going on? Aren't those commands supposed to do the same thing??
Thanks for your help.
Ok, I will try to answer.
First, unless you know what you are doing, you should not start mongod manually.
In general (a bit simplified), calling [sudo] service mongod start, you instruct the system to read the according file in /etc/init and start the executable according to the configuration described in said file.
When you started mongod by hand, however, you actually called the mongod binary, the server software itself – while the name is the same as the service, the two commands have few things in common. The binary does not use the /etc/mongod.conf by default, falling back to its default values for the various settings. Actually, you can see that the config file is explicitly defined in the init script. This is why mongod tried to find /data/db.
You can find said binary by issuing
which mongod
Regarding the lock file: When mongod is started by the system, user root actually assumes the effective user id of mongod (or mongodb I don't know for Ubuntu of the top of my head). When you tried to start it from your user id, you do not have the privileges to overwrite the lock file. When you used sudo mongod afterwards, you assumed the effective user id of root which on the other hand is allowed to overwrite said file. However, mongod will the run as root which is a security no-no.
An init script defines which environment to use, which user to run under and a lot of other stuff. Unless you really know what you are doing, you should not even fiddle with them, much less skip it.
And now, with the finger up
sudo is not the UNIX way of saying "I mean it!"
It has security implications, and you should be very aware of those implications before using it – aka read the man pages of every command you issue before you use it until you have at least a fact based idea of what the command is doing.
And again: unless you really know (as opposed to assume) what you are doing, do not fiddle with the system configuration.
I am currently deploying to Digital Ocean using Meteor Up. If I don't specify a MONGO_URL in the mup.json, can I get the value from the command line while the website is running, i.e. I don't want to shutdown the site?
If I go to the app directory and run meteor mongo --url, I get the following error:
mongo: Meteor isn't running a local MongoDB server.
This command only works while Meteor is running your application
locally. Start your application first. (This error will also occur if
you asked Meteor to use a different MongoDB server with $MONGO_URL when
you ran your application.)
If you're trying to connect to the database of an app you deployed
with 'meteor deploy', specify your site's name with this command.
Even if I run the app from the app directory, it will only give the localhost MONGO_URL. I need the MONGO_URL for the deployed app.
I have also taken a look at a similar question as suggested by some of the answers. I disagree that it is "impossible" to get the MONGO_URL without some other program running on the server. It's not as if we are defying the laws of physics here, folks. Fundamentally, there should be a way to access it. Just because no one has yet figured it out doesn't mean it is impossible.
meteor mongo --url should return the URL.
Try opening another shell in the app directory and running that command.
Meteor Up packages your app in production mode with meteor build so that it runs via node rather than the meteor command line interface. Among other things, this means meteor foo won't work on the remote server (at least not by default). So what you're really looking for is a way to access mongo itself remotely.
I recently set up mongo on an AWS EC2 instance and listed some lessons learned here: https://stackoverflow.com/a/28846703/2669596. Some details of how you do it are going to be different on Digital Ocean, but these are the main things you have to take care of once mongo itself is installed:
Public IP/DNS Address: This is probably fine already since you can deploy to the server.
Port Security Rules: You need to make sure port 27017 is open for TCP access, at least from your IP address. MongoDB also has an http interface you can set up; if you want to use that you'll need to open 28017 as well.
/etc/mongod.conf (file location may differ depending on Linux flavor):
Uncomment port=27017 to make sure you have the default port (I don't think this is actually necessary, but it made me feel better and it's good to know where to change the default port...).
Comment out bind_ip=127.0.0.1 in order to listen to external interfaces (e.g. remote connections).
Uncomment httpinterface=true if you want to use the http interface.
You may have to restart the mongod host via sudo service mongod restart. That's a problem if you can't have downtime, but I don't know of a way around that if you change the config file.
Create User: You need to create an admin and/or user to access the database remotely.
Once you've done all of that, you should be able to access the database from your local machine (assuming you have the mongo client installed locally) by running
mongo server.url.com:27017/mup-app-name -u username -p
where server.url.com is the URL or IP address of your remote server, mup-app-name is the appName parameter from your mup.json file, username is the user you created to access the database, and you'll be prompted for that user's password after you run the command (or you could put it after -p on the same line, depending on the password).
There may also be a way to do this by setting up nginx to reverse-proxy 127.0.0.1:27017 on your remote server, but I've never done it and that's just me speculating.
I am trying to COPY a file into a table in PostgreSQL. The table owner is postgres and the file owner is postgres.
The file is in /tmp.
Still I am getting the error message:
could not open file "/tmp/file" for reading: Permission denied
I don't understand what I am doing wrong as all the posts I've found say that if I have the file in /tmp and owner is postgres then the COPY command should work.
A guess: You are using Fedora, Red Hat Enterprise Linux, CentOS, Scientific Linux, or one of the other distros that enable SELinux by default.
Either and on your particular OS/version the SELinux policies for PostgreSQL do not permit the server to read files outside the PostgreSQL data directory, or the file was created by a service covered by a targeted policy so it has a label that PostgreSQL isn't allowed to read from.
You can confirm whether or not this is the problem by running, as root:
setenforce 0
then re-testing. Run:
setenforce 1
to re-enable SELinux after testing. setenforce isn't permanent; SELinux will be automatically re-enabled on reboot anyway. Disabling SELinux permanently is not usually a good solution for issues like this; if you confirm the issue is SELinux it can be explored further.
Since you have not specified the OS or version you are using, the PostgreSQL version, the exact command you're running, ls -al on the file, \d+ on the table, etc, it's hard to give any more detail, or to know if this is more than a guess. Try updating your answer to include all that and an ls --lcontext of the file too.
COPY with a file name instructs the PostgreSQL server to directly read from or write to a file. The file must be accessible by the PostgreSQL user (the user ID the server runs as) and the name must be specified from the viewpoint of the server. (source: postgresql documentation)
So the file should be readable (or writable) by the unix user under which postgresql server is running (i.e not your user!). To be absolutly sure, you can try to run sudo -u postgres head /tmp/test.csv (assuming you are allowed to used sudo and assuming the database user is postgres).
If that fails, it might be an issue related to SELinux (as mentioned by Craig Ringer). Under the most common SELinux policy (the "targeted" reference policy), used by Red Hat/Fedora/CentOS, Scientific Linux, Debian and others... the postgresql server process is confined : it can only read/write a few file types.
The denial might not be logged in auditd's log file (/var/log/audit/audit.log) due to a donaudit rule. So the usual SELinux quick test apply e.g: stop SELinux from confining any process by running getenforce;setengorce 0;getenforce, then test postgresql's COPY. Then re-activate SELinux by running setenforce 1 (this command modify the running state, not the configuration file, so SELinux will be active (Enforcing) after reboot.
The proper way to fix that is to change the SELinux context of the file to load. A quick hack is to run:
chcon -t postgresql_tmp_t /tmp/a.csv
But this file labelling will not survive if hte filesystem is relabel or if you create a new file. You will need to create a directory with an SELinux file context mapping :
which semanage || yum install policycoreutils-python
semanage fcontext -a -t postgresql_tmp_t '/srv/psql_copydir(/.*)?'
mkdir /srv/psql_copydir
chmod 750 /srv/psql_copydir
chgrp postgres /srv/psql_copydir
restorecon -Rv /srv/psql_copydir
ls -Zd /srv/psql_copydir
Any file created in that directory should have the proper file context automatically so postgresql server can read/write it.
(to check the SELinux context under which postgres is running, runps xaZ | grep "postmaste[r]" | grep -o "[a-z_]*_t", which should print postgresql_t. To list the context types to which postgresql_t can write, use sesearch -s postgresql_t -A | grep ': file.*write'. the command sesearch belong to the setools-console RPM package).