So i have an instance that supposedly has a lot of free space.
I have installed Postgres and when i try to import a dump it tells me my disk is full. I did a df -h and saw that in fact xvda1 is full, but what about `xvdb ?
it has a lot of free space, how can i use that for my Postgres database??
df -h
Filesystem Size Used Avail Use% Mounted on
/dev/xvda1 7.9G 7.5G 1.1M 100% /
udev 1.9G 8.0K 1.9G 1% /dev
tmpfs 376M 184K 375M 1% /run
none 5.0M 0 5.0M 0% /run/lock
none 1.9G 0 1.9G 0% /run/shm
/dev/xvdb 394G 199M 374G 1% /mnt
xvdb is most likely an the Amazon EC2 Instance Store volume (also called an ephemeral volume, in contrast to an Amazon EBS volume) and you almost certainly do not want to use it if you value your data, except if you exactly know what you are doing and are prepared to always have point in time backups etc.:
Ephemeral storage will be lost on stop/start cycles and can
generally go away, so you definitely don't want to put anything of
lasting value there, i.e. only put temporary data there you can
afford to lose or rebuild easily, like a swap file or strictly
temporary data in use during computations. Of course you might store
huge indexes there for example, but must be prepared to rebuild these
after the storage has been cleared for whatever reason (instance
reboot, hardware failure, ...).
See my answers to the following related questions for more details:
Will moving data from EBS to ephemeral storage improve MySQL query performance?
how to take backup of aws ec2 instance/ephemeral storage?
Related
I'm using some FIWARE components (Orion 2.4.0-bext and Cygnus 2.3.0). Also I'm using mongoDB (version 3.6) for current context information and MySQL (version 5.7) for saving historical changes using Cygnus. All component are dockerized, each one on its own container.
In my current solution, I receive lots of messages (between 100 and 500 per second) that are stored into mongoDB (through Orion) and MySQL (through Cygnus). The problem that I found is that
Mongo and Cygnus are using a lot of CPU power, even tough the amount of data is not huge.
Cygnus is growing insanely in the amount of disk used by the container.
Cygnus has been configured to store 1 row per attribute change in MySQL. These are the docker-compose environment variables of Cygnus regarding the configuration of the mapping in MySQL:
environment:
- ...
- "CYGNUS_LOG_LEVEL=DEBUG" #
- "CYGNUS_MYSQL_DATA_MODEL=dm-by-entity-type"
- "CYGNUS_MYSQL_ATTR_NATIVE_TYPES=true"
- "CYGNUS_MYSQL_ATTR_PERSISTENCE=column"
To ilustrate point 2), I'll show you how much space is using Cygnus after and bofere running for a while.
When I check the disk used:
[root#maaster-docker-test docker]# du -h --max-depth=1
4.0K ./trust
20K ./builder
11M ./image
12G ./overlay2
4.0K ./runtimes
72K ./buildkit
100K ./network
4.0K ./tmp
807M ./volumes
4.0K ./swarm
20K ./plugins
38G ./containers
50G .
We can see that containers folder is using 38G of space. Inspecting a little more:
[root#maaster-docker-test containers]# du -h --max-depth=1
35G ./dd1c103dd7d1a5ecbab0e3a2e039a6c4d3fd525837a766a9bab28b29ba923a32
1.4M ./ebf1c3978077a727d414d3e2de5974b03a236999cc66c29f66c0e517d6bbe055
40K ./2c92baf1368ee292b7cf33db86369d0b6f7941753f54f145a14e5793eac6eba2
40K ./047c082e1f1e8f26be0bb1a063e93907f55b4736ad88fd29736e383c6e03d559
175M ./63c715643cfbd7695dc538081e4a963e270dc03a4ffdb528bb375cf57438a477
152M ./cbb5fe85b16411dc94c8ab00dcd7b40b728acaa422398445be4130aa0197d287
1.4M ./e7c01dca2246f17c5f0aa9f413772b7b73be2d962ae772e062c5257ce95252fa
2.6G ./534b92fe6f1dac9f56d59b1e9feeb0d013199d17fae0af1df150dd20b96c9f70
38G .
container starting with dd1c103dd7d1 is taking 35G of space. When I check to whom this container corresponds:
[root#maaster-docker-test containers]# docker ps
...
dd1c103dd7d1 fiware/cygnus-ngsi:latest "/cygnus-entrypoint.…" 2 days ago Up 2 days (healthy) 0.0.0.0:5050-5051->5050-5051/tcp, 0.0.0.0:5080->5080/tcp fiware-cygnus
...
Turns out that is the Cygnus. At this point, I had been inserted around 12000 different entities in MongoDB, reflecting around 300.000 changes (rows) in MySQL. So the amount of data is not that insane.
To finalize, after restarting the services defined in docker-compose, this is stopping Cygnus, removing containers and starting them again):
[root#maaster-docker-test containers]# docker ps
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
...
0fb5ab78b70b fiware/cygnus-ngsi:latest "/cygnus-entrypoint.…" 3 minutes ago Up 3 minutes (healthy) 0.0.0.0:5050-5051->5050-5051/tcp, 0.0.0.0:5080->5080/tcp fiware-cygnus
...
And checking again for space used
[root#maaster-docker-test containers]# du -h --max-depth=1
48K ./e8d32c563ad0a22f70a83b576592b32b6336b0b39cc401cfe451896e6b76a897
40K ./043069752e112cdce051d3792b8a261133b869e939bed263b9e0eb7dc265c379
80K ./f1f40e29a8d620fc38b7b12cfcd4986f3ffcc2f970f08d582cb826c2f2a41ff0
1.1M ./0fb5ab78b70bcc856c91b9b99266049b20f05f5d1d6d7d224cdfde9eec47b801
40K ./047c082e1f1e8f26be0bb1a063e93907f55b4736ad88fd29736e383c6e03d559
128K ./aa3c63cafe57c6ea1b4ed01bee9415b6a87bf538d6d7a700750be21c738d39d5
68K ./11388b7090bcd08ac0235129b7354e5c9b68f1d4855abdc97f025ad597b73e68
68K ./84a6aaf4221fde5cb0e21b727e33e21ff58d275597b121ba22f87296148b38db
1.6M .
We can see that the container used by Cygnus (0fb5ab78b70b), is taking only 1.1M of space.
So, does anyone know if there is a problem with Cygnus, with Docker, or with Cygnus running in a Docker environment?
Is there any configuration change I should perform on Cygnus to be more efficient and also consume less disk space?
Also, I'd be nice to know if there is any configuration change I must to do mongoDB (or maybe ORION) in order to make mongodb more efficient? I don't think that inserting ~200 new entities per second should not impact in the CPU usage of mongo process.
Thanks a lot in advance
I've discovered that the configuration
"CYGNUS_LOG_LEVEL=DEBUG"
was generating an enormous amount of data that was putting down the performance of the service, increasing the disk usage.
Changing that value solved my issue.
I want to perform actions before my disk is running out of space.
For that, I have to periodically check the size of my DB.
I can query the size of the DB (for example by running pg_database_size( current_database() ). The problem is, that the size of the postgres folder itself, (/var/lib/postgresql/data) is much bigger. (almost twice the size of the response that I get from my query.
edit: I run the du -h /var/lib/postrgesql/data and pg_database_size() on my data base few times, when the database was with different data.
Here are the results :
du -h /var/lib/postgresql/data | pg_databse_size()
-----------------------------------------------------------
3.7G |(1582MB)
5.8G |(2490MB)
6.3G |(2699MB)
7.8G |(4245MB)
8.0G |(4400MB)
8.9G |(5330MB)
58G |(41GB)
Those are the file from my du -h command (when running this command, pg_database_size()=41 GB
du -h data/
4.0K data/pg_replslot
4.0K data/pg_wal/archive_status
16G data/pg_wal
4.0K data/pg_stat
12K data/pg_multixact/offsets
12K data/pg_multixact/members
28K data/pg_multixact
4.0K data/pg_logical/snapshots
4.0K data/pg_logical/mappings
16K data/pg_logical
12K data/pg_notify
168K data/pg_subtrans
7.4M data/base/12993
7.4M data/base/1
7.6M data/base/12994
20K data/base/pgsql_tmp
43G data/base/16384
43G data/base
40K data/pg_stat_tmp
4.0K data/pg_twophase
584K data/global
4.0K data/pg_snapshots
4.0K data/pg_commit_ts
20K data/pg_xact
4.0K data/pg_serial
4.0K data/pg_tblspc
4.0K data/pg_dynshmem
58G data/
Is there any correlation between the size of the DB and the size of /var/lib/posgresql/data?
How can I predict the size of the data folder based on the DB size?
There is a correlation between database size and the space used by the data directory, but “correlation” is a statistical term, and statistics won't tell you if you are currently running out of disk space.
What you actually want to know is if you can determine the size of the PostgreSQL data directory based on the size of the databases, right?
The answer to that is “no”: there are many other data in the PostgreSQL data directory, and they are not related to the database size.
The most obvious example are the transaction logs (WAL files), which are in the pg_wal subdirectory (pg_xlog in older PostgreSQL versions).
The amount of WAL generated is proportional to the amount of data modifications to the database, which has nothing to do with the size of the database.
Normally, WAL size is limited by max_wal_size (checkpoint_segments in older versions), but that is no hard limit. For example, if WAL archiving has a problem, old and un-archived WAL file will not get deleted. They can accumulate pretty quickly if you have a lot of data modification activity and can fill your disk. This is completely unrelated to the size of your database.
A second example are temporary files, which are stored in the base/pgsql_tmp subdirectory of your data directory. Such files are generated to store intermediary query data if they don't fit into work_mem RAM.
I've noticed that Ubuntu (14.04 and 16.04) creates a /run partition depending on the available RAM, and in 3 different servers i've noticed that this value is always equal to 10% of installed RAM on the server.
How to correctly bypass this value and increase this tmpfs partition on Ubuntu?
Thanks.
I have mongodb running on a linux server with 90% disk usage (as confirmed by df). Every operation is causing the following error:
db.dropDatabase() { "ok" : 0, "errmsg" : "Can't take a write lock
while out of disk space", "code" : 14031 }
'df -k' shows:
Filesystem 1K-blocks Used Available Use% Mounted on
/dev/xvda1 16380820 14622652 1657920 90% /
devtmpfs 2015972 56 2015916 1% /dev
tmpfs 2024984 0 2024984 0% /dev/shm
There seems to be plenty of disk space. Is it possible that the database is corrupted? The disk had originally ran out of disk space but then I deleted many files and cleared 10% but the error persists.
Can anyone recommend what next steps to take?
Thanks
So it turns out that mongodb requires as much space as the size of the database you are dealing with. This is crazy. What I had to do, is to clear as much disk space as the size of the db.
SWAP memory of my solaris server is used more than its threshold, is it possible to free some space by restarting the processes which are using most of SWAP memory?
$ swap -s
total: 10820256k bytes allocated + 453808k reserved = 11274064k used, 11911648k available
$ swap -l
swapfile dev swaplo blocks free
/dev/md/dsk/d210 85,210 16 20972720 20971152
You are only using 800 KB of swap out of 10 GB (i.e. 0.008 %) which is insignificant. There is then probably nothing to worry about according to the reported statistics.
You might have a look to /tmp (or other tmpfs backed directories) to free some virtual memory.