How can I compact Firebird 2.1 database, like we do in MS Access (discarding erased data, remaking index, etc)?
There's a way to do it?
Thanks!
Usually there is no need to compact a Firebird Database: see fb release notes about garbage collection and an automatic (per-database configurable) operation named "sweep".
In few words, fb reuses space in pages when records are deleted or oldest record version are freed asking for disk space chunks only when free space becomes too small (i.e. under a defined percent).
Sweep is performed as default after a predefined number of commited transactions, bur it's an expensive task.
Backup and restore must be intended as last resort to optimize and shrink, as this rebuilds and optimize indexes too, but usually this is not needed as there are commands and tools to rebuild indexes.
The only way to do it is to make a backup and a restore.
From the official faq
Many users wonder why they don't get their disk space back when they
delete a lot of records from database.
The reason is that it is an expensive operation, it would require a
lot of disk writes and memory - just like doing refragmentation of
hard disk partition. The parts of database (pages) that were used by
such data are marked as empty and Firebird will reuse them next time
it needs to write new data.
If disk space is critical for you, you can get the space back by
doing backup and then restore. Since you're doing the backup to
restore right away, it's wise to use the "inhibit garbage collection"
or "don't use garbage collection" switch (-G in gbak), which will make
backup go A LOT FASTER. Garbage collection is used to clean up your
database, and as it is a maintenance task, it's often done together
with backup (as backup has to go throught entire database anyway).
However, you're soon going to ditch that database file, and there's no
need to clean it up.
Related
Our postgresql db has a no-usage window of 2am to 6am.
one of the daily cron jobs already does a VACUUM FULL during this period. i am seeing no real performance hit with the ~200 odd users who use the web site. but the db is what i would classify as 'light' at this time.
however, there is forecast of data surge in the upcoming months due to some process changes in the org. my specific question:
is there a performance gain to be expected if I dump the entire db to a text file (already happens as part of db backup), drop the database, recreate it and reload the dump back. if the answer is 'yes', how significant is the gain?
or will VACUUM FULL do the job and no action is needed?
vacuum full would fo it for you. no need to manually reload data
https://www.postgresql.org/docs/current/static/sql-vacuum.html
VACUUM FULL
can reclaim more space, but takes much longer and exclusively locks the table. This method also requires extra disk
space, since it writes a new copy of the table and doesn't release the
old copy until the operation is complete.
I am familiar both with the MongoDB repairDatabase and compact commands, but these both seem to lock the database and/or collection. Is there another way to reclaim deleted disk space without essentially shutting down the database? What are best practices in this area? Thanks!
Best practice would probably depend on your schema and what your application does. Here's my use case, perhaps you can learn something... My application is storing very large amounts of time stamped data samples. Deleting data from a very large store is a very expensive operation, this gets more complicated when you try doing this on live systems. MongoDB had several issues in the past with reclaiming the disk space back to OS and we had to dance around this, not sure how good it works now. But what we did solved everything for good - we partitioned the data in such way so that we could dispose of old stuff by simply dumping entire database. Dropping mongodb database is a very cheap and efficient operation, almost instantaneous even when you drop a TB. Note that dropping collection is not as effective as dropping database, this was actually a key to the solution. For doing this we had to redesign the schema.. Your case of course could be different, but the lesson learned is that deleting data from large storage is very expensive.
The best method currently is to run a Master Slave Setup.
Shutdown 1 mongod instance and let it resync.
More details here: Reducing MongoDB database file size
I am using python, scrapy, MongoDB for my web scraping project. I used to scrape 40Gb data daily. Is there a way or setting in mongodb.conf file so that MongoDB will exit normally before applying a write lock on db due to disk full error ?
Because every time i face this problem of disk full error in MongoDB. Then I have to manually re-install MongoDB to remove the write lock from db. I cant run repair and compact command on the database because for running this command also I need free space.
MongoDB doesn't handle disk-full errors very well in certain cases, but you do not have to uninstall and then re-install MongoDB to remove the lock file. Instead, you can just mongod.lock file from this. As long as you have journalling enabled, your data should be good. Of course, at that moment, you can't add more data to the MongoDB databases.
You probably wouldn't need repair and compact only helps if you actually have deleted data from MongoDB. compact does not compress data, so this is only useful if you indeed have deleted data.
Constant adding, and then deleting later can cause fragmentation and lots of disk space to be unused. You can prevent that mostly by using the userPowerOf2Sizes option that you can set on collections. compact mitigates this by rewriting the database files as well, but as you said you need free disk space for this. I would advice you to also add some monitoring to warn you when your data size reaches 50% of your full disk space. In that case, there is still plenty of time to use compact to reclaim unused space.
I use PostgreSQL on an embedded system with limited drive space. Now the DB-drive is full. When I delete data, it doesn't seem to free up any space. I tried to VACUUM FULL, but that requires space. So does deleting the last remaining index.
Any ideas on how to free up space without randomly deleting stuff? I can afford to lose some of the data from back when, but I can't seem to actually do it, since there isn't enough space to VACUUM FULL.
PostgreSQL uses MVCC model which means that deleted records mark their space as free (after the transaction which deleted them had been committed) but it is still reserved by the table.
Prior to PostgreSQL 9.0, VACUUM FULL used to move the data inside the table without need for additional space.
In PostgreSQL 9.0, behavior of VACUUM FULL had changed and now it requires additional space for the full copy of the table.
You may try to drop the indexes from the tables and vacuum them one by one, starting from the least one.
The easiest answer at this point would be to dump the database to a different drive/computer (for instance, using pg_dump, or pg_dumpall if you have more than one db, and keeping in mind things like Large Objects that need special backup/restore processes) then drop and recreate the database.
If there's a tiny bit of space left, you might try vacuum full smallesttable, which might be able to finish and free up some space to vacuum the next smallest table, and so on.
If you end up filling the drive completely, the database server will probably refuse to start and you won't be able to do either of those. In that case, you could move the entire data directory to another computer with the same CPU architecture and more disk space, then start postgresql there to perform the vacuum.
In certain situations VACUUM (not full) can reclaim some disk space. (I think it will return pages that are totally dead to the OS.) That might free up enough space to begin with the VACUUM FULL. But it's not a good idea to let one table grow to more than the amount of disk free space.
This one is interesting to me - despite the almost inane title. I have used Firebird for a long time, but not until recently noticed an interesting behavior.
I am using embedded Firebird 1.5, and noticed that if I stuff the database full of blobs (lets say 10mb worth), the size of the database increases. I can then delete all the fields in the database, and the file size of the DB remains at its expanded size. Currently it is at 20mb and is completely empty.
I know that Firebird has this built into its architecture (for quick indexing, speed issues etc), but I always thought it would decrease back down to its original ~2mb default.
Does anyone have any suggestions to 'deflate' the file size? The reason being is that this is a space conscious issue. If I had tons of space to work with, I wouldn't care. However that is not the case, and I need things to be as optimal as possible
The only way to free unused space in a firebird database is to do a backup then an immediate restore of that backup (Reference: Firebird FAQ).
Here is a good technical explanation of why this is so.
Note that Firebird will reuse the currently unused space - ie. if you put another 10MB of blobs in now, the database should not grow to 30MB.