I have a database with roughly 4,000 members. I need to decrypt all their passwords so that I may re-encrypt them using MD5 and put into a new system. I have one account with before encryption and after encryption password strings so with this I assume I should be able to figure out a method and key range. I do not have access to the original source code or I would not be here.
I should mention that existing passwords vary dramatically.
User1 - qwDyP1db9iOPI
User2 - $1$.WXtDyLg$yxBPAWxzd.srEWJZfqZAY/
User3 - do8sLMoVVqxKQ
User4 - TKmnAlPe
How do I approach such a problem?
Typically, you can't decrypt the passwords, because the passwords are not stored. What you are seeing is hashes of the password - there's no efficient way to find a password that matches a particular hash.
You can either force everyone to change their password once you have the configuration changed to use MD5 hashing, or you can generate new temporary passwords for everyone to get all the hashes replaced at once, but then they should all change them again anyway.
Those look like they probably came from Linux (the one that starts with $1$ is already an MD5 hash, the others look like the older crypt() DES variant style).
That's not always the case when dealing with application passwords - for example, some databases use considerably weaker credential storage - but Unix/Linux flavors have not stored actual passwords for a very long time, if ever.
let's think. Suppose I can hack my way to your computer / DB (not an impossible task). So I have what you have - a list of all users' passwords, but encrypted. But most probably I can register to your system myself, and I know my own password. I can find its encrypted form in the DB, either by my username, or by the latest entry, so now I have the clear + encrypted pair! If I could do what you are asking, I'd have all passwords and full access to all accounts. Meaning that encrypting passwords, or data in general has virtually no additional security effect, and all I should care about is preventing unauthorized access to the data. But this is not true, so you can't :)
BTW, MD5 is not an encryption but a hash, or more correctly a digest.
Looks like twalberg is right for the most part, but I'd like to go a step further to give you some practical advice:
1) Look at http://www.openwall.com/john/ (John the Ripper) for help getting the original passwords out of the database you have. Note that the passwords generated may not be the original passwords (it is likely, but not certain).
Once you've gotten all of the passwords using this tool, think to yourself: Anyone could do this. Your users might use these passwords somewhere else. If someone broke into your site, they could expose the people who trust you. Your users could blame you if someone used this information to break into their other accounts. Having the cleartext passwords around is risky for you.
I recommend you take this opportunity to have your users change their passwords. Those that no longer use their accounts should probably have to go through a small verification step to regain access, so you can disable their accounts now.
2) When you generate the new password database, I recommend you at a minimum use a secure hash (E.g. SHA1 or SHA2) along with a per-user salt (random value prepended to the generated hash output in a standard way) to help secure the database from accidental exposure.
While this answer isn't efficient, it is probably practical for the most part for your current database if your actual password database has even approximately the proportions you represent.
If you're switching away from your current system don't switch to plain MD5 - there are "rainbow tables" to crack a lot of common passwords; instead use salted/double MD5 or SHA
Thanks all for your input. I will just have everyone change their passwords. I have no intention of keeping anyone's password. The goal was to not have all 4000 users have to change their passwords after migrating to the new system but it seems unavoidable. As davka pointed out, I have an account in this database and know my password and can see the encrypted version so thought that would be sufficient to develop a script that decrypts and renters the password into the new database. Thanks again.
Related
Why is it not a security hole that PostgreSQL by default stores user passwords in an MD5 hash? I am studying the internals of PostgreSQL and have gotten to the system catelog pg_authid and when I read about the MD5 hash encryption it appears that it is regarded as antiquated. In my thinking if an admin or a user is able to access the underlying file store then they could hypothetically crack the passwords and do whatever said credentials would enable.
I ask why it is not a security hole because apparently PostgreSQL has been "Common Criteria Certified" which seems to be military grade secure according to it's wiki which notes it's provenance from western defense organizations.
Thanks!
First, PostgreSQL 10 adds SCRAM-SHA256 based on SASL, making this a moot point.
For older versions: It's a weakness, but it's not a large security hole for a number of reasons:
Internet-deployed PostgreSQL instances should be using SSL, preventing eavesdropping on the protocol. This greatly reduces the chances of successful password theft.
Passwords are twice-salted. The password stored on disk is hashed with a salt and the md5 digest is taken. But the password sent on the wire is re-salted and re-hashed with an authentication-exchange specific salt, so if you capture a hashed password on the wire you cannot simply replay it in a later authentication.
If you manage to get a few copies of the same twice-salted password by eavesdropping a plaintext connection, you could potentially exploit weaknesses in MD5 to find the once-salted version that's stored on disk, and use that to authenticate with the DB.
But it's a lot of work, and it's pretty much totally prevented by using SSL.
Personally speaking, I think the "Common Criteria" is close to a pile bureaucratic nonsense. It applies only to one specific install with very narrow, specific versions of everything from hardware on up. It should help exclude total snake-oil, but it sure doesn't prove anything is secure. (Hell, look at Government systems...)
I'm studying a migration from an old LDAP to a new database.
Today I just found that the passwords are all saved as {crypt} passwords and the new system does not support that.
Is it possible to decode the {crypt} passwords (in PERL if possible) ?
No, crypt implements a one way hash.
However, you have the hash values and (hopefully) the salt. Whenever a user submits a password, it's encrypted with the same salt, and if the hashes match, the user is authenticated.
In general, it's a very bad practice to have human readable passwords, but you don't need to read a raw input password to authenticate a user.
The answer should be no. The whole point of hashes such as crypt is that you can't do that. However, if it's using the old-fashion unix crypt(), it's often actually quite brute-forceable (it's rather weak cryptography). Question is, should you really want that.
This is sounding like an XY problem though. What LDAP servers are we talking about, and what formats do they support? There's a reasonable chance there's some kind of overlap that can be used for migrational purposes.
If implemented correctly a password stored as crypt can not be decoded into cleartext because it is not encrypted but hashed.
Depending on your target system you might be able to migrate the hashed passwords "as-is", but you have to determine which hash was used and if the same is supported by your target system.
You need to disclose more about the LDAP server and the database you use.
On our message board, we use password matching to help detect members with multiple registrations and enforce our rules against malicious puppet accounts. It worked well when we had SHA256 hashes and a per-site salt. But we recently had a humbling security breach in which a number of password hashes fell to a dictionary attack. So we forced a password change, and switched to bcrypt + per-user salts.
Of course, now password matching doesn't work anymore. I don't have a formal education in cryptography or computer science so I wanted to ask if there's a secure way to overcome this problem. Somebody I work with suggested a second password field using a loose hashing algorithm which intentionally has lots of collisions, but it seems to me that this would either lead to tons of false positives, or else reduce the search space too much to be secure. My idea was to stick with bcrypt, but store a second password hash which uses a per-site salt and an extremely high iteration count (say 10+ seconds to generate on modern hardware). That way users with the same password would have the same hash, but it couldn't be easily deduced with a dictionary attack.
I'm just wondering if there's an obvious problem with this, or if someone more knowledgeable than me has any suggestions for a better way to approach things? It seems to me like it would work, but I've learned that there can be a lot of hidden gotchas when it comes to security. :P Thanks!
Short Answer
Any algorithm that would allow you to detect whether or not 2 users had the same password would also allow an attacker to detect whether or not 2 users had the same password. This is, effectively, a precomputation attack. Therefore, your problem is not securely solvable.
Example
Assume I've compromised your password database.
Assume I've figured out how your hashes are calculated.
If I can apply your password transformation algorithm to "password" and quickly tell which users use "password" as their password, then the system is vulnerable to a form of precomputation attack.
If I must do an expensive calculation to determine the password for each individual user and work spent to calculate User A's password does not make calculating User B's password easier, then the system is secure (against these type of attacks).
Further Consideration
Your idea of using a per-site salt with bcrypt and a high iteration count may seem attractive at first, but it just can't scale. Even at 10 seconds, that's 6 password guesses per minute, 360 per hour, 8640 per day, or 3M per year (that's a lot). And that's just one machine. Throw a botnet of machines at that problem, or some GPU's and suddenly that number goes through the roof. Just 300 machines/cores/GPU's could knock out 2.5M guesses in a day.
Because you would be using the same salt for each one, you're allowing the attacker to crack all of your user's passwords at once. By sticking with a per-user salt only, the attacker can effectively only attempt to crack a single user's password at a time.
The short answer given above makes the assumption that the attacker has the same access as the server at all times, which is probably not reasonable. If the server is compromised in a permanent manner (owned by the attacker) then no scheme can save you - the attacker can retrieve all passwords as they are set by the user. The model is more normally that an attacker is able to access your server for a limited period of time, some point after it has gone live. This introduces an opportunity to perform the password matching that you've asked about without providing information that is useful to an attacker.
If at sign-up or password change your server has access to the password in plain text, then the server could iterate through all the user accounts on the system, hashing the new password with each user's individual salt, and testing to see if they were the same.
This doesn't introduce any weaknesses, but it would only be useful to you if your algorithm for preventing multiple fake accounts can use this as a one-time input ("this password matches these accounts").
Storing that information for later analysis would obviously be a weakness (for if an attacker can obtain your database of passwords, they can probably also obtain this list of accounts with the same password). A middle ground might be to store the information for daily review - reducing the total useful information available to an attacker who temporarily compromises your storage.
All of this is moot if the salting and hashing occurs client-side - then the server can't carry out the test.
I came across a table the other day on our enterprise system -- dba_users (oracle).
I was able to find a hashed password in this table for each employee, and also their username.
As far as I know (from googling) the username + password is concatenated, then hashed.
Question: knowing the "salt" (my own username), the "original value" (my own password), and also the hashed value...is there a danger here of being able to figure-out the hash?
Also -- googling oracle 10g hash -- seems like some folks think they've figured the hash algorythm out. And I've read about "rainbow tables" and "offline dictionary attacks"... And finally, I've googled oracle 11g, and one of the features in that version is they hide the hashed password in dba_users so end-users can't see it.
Anyway, I'm scratching my head over why I (i.e., end-users) have access to this table, and why the DBA dosen't seem too worried about it.
The whole point of hashing the password and then storing the hash to database is that if you do so you won't have to worry about who can see the password in the table.
To emphasize: a (properly calculated) hash value of a password which is stored in database is completely useless without the original.
As far as I know, for algorithms like md5, there isn't a way to reverse engineer the original password from hash whatsoever. That's why most services nowadays don't send you your password when you click 'I forgot' link - instead they offer to set a new one (as soon as you provide the old password and the service compares hashes).
Elaborating on what #Goran Jovic said, by concatenating (ie salting) each hashed password with the username.There is no danger in making this available. To make sense of this, you have to understand how a rainbow table work. The way a series of passwords is cracked with a rainbow table is by loading up a precomputed series of hashes from passwords into memory. Then, searching through the table of hashes associated unknown passwords to see if you can find a match in the rainbow table (of which you know the password corresponding with a specific hash). By salting the hashes with a unique identifier (ie a username), however, you defeat this attack because even a password that's already been seen will hash differently depending n the user associated with it. Therefore now, instead of a rainbow table having to handle the millions of different password combinations possible, it must now contain a hash of every possible password plus every possible username. The results space of this is simply too large to search and unless there was a quantum leap in computing technology impossible to beat using a bruteforce tactic.
In a web application written in Perl and using PostgreSQL the users have username and password. What would be the recommended way to store the passwords?
Encrypting them using the crypt() function of Perl and a random salt? That would limit the useful length of passswords to 8 characters and will require fetching the stored password in order to compare to the one given by the user when authenticating (to fetch the salt that was attached to it).
Is there a built-in way in PostgreSQL to do this?
Should I use Digest::MD5?
Don't use SHA1 or SHA256, as most other people are suggesting. Definitely don't use MD5.
SHA1/256 and MD5 are both designed to create checksums of files and strings (and other datatypes, if necessary). Because of this, they're designed to be as fast as possible, so that the checksum is quick to generate.
This fast speed makes it much easier to bruteforce passwords, as a well-written program easily can generate thousands of hashes every second.
Instead, use a slow algorithm that is specifically designed for passwords. They're designed to take a little bit longer to generate, with the upside being that bruteforce attacks become much harder. Because of this, the passwords will be much more secure.
You won't experience any significant performance disadvantages if you're only looking at encrypting individual passwords one at a time, which is the normal implementation of storing and checking passwords. It's only in bulk where the real difference is.
I personally like bcrypt. There should be a Perl version of it available, as a quick Google search yielded several possible matches.
MD5 is commonly used, but SHA1/SHA256 is better. Still not the best, but better.
The problem with all of these general-purpose hashing algorithms is that they're optimized to be fast. When you're hashing your passwords for storage, though, fast is just what you don't want - if you can hash the password in a microsecond, then that means an attacker can try a million passwords every second if they get their hands on your password database.
But you want to slow an attacker down as much as possible, don't you? Wouldn't it be better to use an algorithm which takes a tenth of a second to hash the password instead? A tenth of a second is still fast enough that users won't generally notice, but an attacker who has a copy of your database will only be able to make 10 attempts per second - it will take them 100,000 times longer to find a working set of login credentials. Every hour that it would take them at a microsecond per attempt becomes 11 years at a tenth of a second per attempt.
So, how do you accomplish this? Some folks fake it by running several rounds of MD5/SHA digesting, but the bcrypt algorithm is designed specifically to address this issue. I don't fully understand the math behind it, but I'm told that it's based on the creation of Blowfish frames, which is inherently slow (unlike MD5 operations which can be heavily streamlined on properly-configured hardware), and it has a tunable "cost" parameter so that, as Moore's Law advances, all you need to do is adjust that "cost" to keep your password hashing just as slow in ten years as it is today.
I like bcrypt the best, with SHA2(256) a close second. I've never seen MD5 used for passwords but maybe some apps/libraries use that. Keep in mind that you should always use a salt as well. The salt itself should be completely unique for each user and, in my opinion, as long as possible. I would never, ever use just a hash against a string without a salt added to it. Mainly because I'm a bit paranoid and also so that it's a little more future-proof.
Having a delay before a user can try again and auto-lockouts (with auto-admin notifications) is a good idea as well.
The pgcrypto module in PostgreSQL has builtin suppotr for password hashing, that is pretty smart about storage, generation, multi-algorithm etc. See http://www.postgresql.org/docs/current/static/pgcrypto.html, the section on Password Hashing Functions. You can also see the pgcrypto section of http://www.hagander.net/talks/hidden%20gems%20of%20postgresql.pdf.
Use SHA1 or SHA256 hashing with salting. Thats the way to go for storing passwords.
If you don't use a password recovery mechanism (Not password reset) I think using a hashing mechanism is better than trying to encrypt the password. You can just check the hashes without any security risk. Even you don't know the password of the user.
I would suggest storing it as a salted md5 hash.
INSERT INTO user (password) VALUES (md5('some_salt'||'the_password'));
You could calculate the md5 hash in perl if you wish, it doesn't make much difference unless you are micro-optimizing.
You could also use sha1 as an alternative, but I'm unsure if Postgres has a native implementation of this.
I usually discourage the use of a dynamic random salt, as it is yet another field that must be stored in the database. Plus, if your tables were ever compromised, the salt becomes useless.
I always go with a one-time randomly generated salt and store this in the application source, or a config file.
Another benefit of using a md5 or sha1 hash for the password is you can define the password column as a fixed width CHAR(32) or CHAR(40) for md5 and sha1 respectively.