I'm using the PostgreSQL timestamp to determine the end date of a row and would like to populate high values such as "9999-12-31 00.00.00.000000".
How can I do that using a query?
The greatest possible value for a timestamp in PostgreSQL is the special value infinity (+infinity or positive infinity).
The greatest non-infinite value depends on the date/time representation and the data types on the platform. That's part of why it's often best to use infinity or if more appropriate, null.
If you use infinity, note that many programming languages don't define an infinite value for their date/time types. So the database driver must pick a sentinel value on the client language side to represent infinity. Then it has the problem that it doesn't know, when writing data back to the server, if the client application meant that value or its use as a placeholder for infinity. So even though infinity is the correct choice when you want a "higher than everything" value, it's not necessarily a practical choice.
If you really must have a high sentinel value, choose one that's the lowest common maximum among all the client applications and languages you want to use, for both their timestamp and date types. Then add a check constraint that prevents any higher values from being inserted.
Related
Our current PostgreSQL database is using GUID's as primary keys and storing them as a Text field.
My initial reaction to this is that trying to perform any kind of minimal cartesian join would be a nightmare of indexing trying to find all the matching records. However, perhaps my limited understanding of database indexing is wrong here.
I'm thinking that we should be using UUID as these are stored as a binary representation of the GUID where a Text is not and the amount of indexing that you get on a Text column is minimal.
It would be a significant project to change these, and I'm wondering if it would be worth it?
When dealing with UUID numbers store them as data type uuid. Always. There is simply no good reason to even consider text as alternative. Input and output is done via text representation by default anyway. The cast is very cheap.
The data type text requires more space in RAM and on disk, is slower to process and more error prone. #khampson's answer provides most of the rationale. Oddly, he doesn't seem to arrive at the same conclusion.
This has all been asked and answered and discussed before. Related questions on dba.SE with detailed explanation:
Would index lookup be noticeably faster with char vs varchar when all values are 36 chars
What is the optimal data type for an MD5 field?
bigint?
Maybe you don't need UUIDs (GUIDs) at all. Consider bigint instead. It only occupies 8 bytes and is faster in every respect. It's range is often underestimated:
-9223372036854775808 to +9223372036854775807
That's 9.2 millions of millions of millions positive numbers. IOW, nine quintillion two hundred twenty-three quadrillion three hundred seventy-two trillion thirty-six something billion.
If you burn 1 million IDs per second (which is an insanely high number) you can keep doing so for 292471 years. And then another 292471 years for negative numbers. "Tens or hundreds of millions" is not even close.
UUID is really just for distributed systems and other special cases.
As #Kevin mentioned, the only way to know for sure with your exact data would be to compare and contrast both methods, but from what you've described, I don't see why this would be different from any other case where a string was either the primary key in a table or part of a unique index.
What can be said up front is that your indexes will probably larger, since they have to store larger string values, and in theory the comparisons for the index will take a bit longer, but I wouldn't advocate premature optimization if to do so would be painful.
In my experience, I have seen very good performance on a unique index using md5sums on a table with billions of rows. I have found it tends to be other factors about a query which tend to result in performance issues. For example, when you end up needing to query over a very large swath of the table, say hundreds of thousands of rows, a sequential scan ends up being the better choice, so that's what the query planner chooses, and it can take much longer.
There are other mitigating strategies for that type of situation, such as chunking the query and then UNIONing the results (e.g. a manual simulation of the sort of thing that would be done in Hive or Impala in the Hadoop sphere).
Re: your concern about indexing of text, while I'm sure there are some cases where a dataset produces a key distribution such that it performs terribly, GUIDs, much like md5sums, sha1's, etc. should index quite well in general and not require sequential scans (unless, as I mentioned above, you query a huge swath of the table).
One of the big factors about how an index would perform is how many unique values there are. For that reason, a boolean index on a table with a large number of rows isn't likely to help, since it basically is going to end up having a huge number of row collisions for any of the values (true, false, and potentially NULL) in the index. A GUID index, on the other hand, is likely to have a huge number of values with no collision (in theory definitionally, since they are GUIDs).
Edit in response to comment from OP:
So are you saying that a UUID guid is the same thing as a Text guid as far as the indexing goes? Our entire table structure is using Text fields with a guid-like string, but I'm not sure Postgre recognizes it as a Guid. Just a string that happens to be unique.
Not literally the same, no. However, I am saying that they should have very similar performance for this particular case, and I don't see why optimizing up front is worth doing, especially given that you say to do so would be a very involved task.
You can always change things later if, in your specific environment, you run into performance problems. However, as I mentioned earlier, I think if you hit that scenario, there are other things that would likely yield better performance than changing the PK data types.
A UUID is a 128-bit data type (so, 16 bytes), whereas text has 1 or 4 bytes of overhead plus the actual length of the string. For a GUID, that would mean a minimum of 33 bytes, but could vary significantly depending on the encoding used.
So, with that in mind, certainly indexes of text-based UUIDs will be larger since the values are larger, and comparing two strings versus two numerical values is in theory less efficient, but is not something that's likely to make a huge difference in this case, at least not usual cases.
I would not optimize up front when to do so would be a significant cost and is likely to never be needed. That bridge can be crossed if that time does come (although I would persue other query optimizations first, as I mentioned above).
Regarding whether Postgres knows the string is a GUID, it definitely does not by default. As far as it's concerned, it's just a unique string. But that should be fine for most cases, e.g. matching rows and such. If you find yourself needing some behavior that specifically requires a GUID (for example, some non-equality based comparisons where a GUID comparison may differ from a purely lexical one), then you can always cast the string to a UUID, and Postgres will treat the value as such during that query.
e.g. for a text column foo, you can do foo::uuid to cast it to a uuid.
There's also a module available for generating uuids, uuid-ossp.
I have stored date in unix timestamp format in mongodb >> 1449060622
now I want to add milliseconds as well so if records inserted in same seconds can be sorted properly.
can someone suggest me that using js new Date() is more better or simply (new Date).getTime() ?
Whenever you store times in MongoDB you should really consider using the native Date type instead. Not only does it provide you with millisecond precision, it also unlocks a lot of features which are unavailable for simple integer, like date aggregation operators for example.
If you really don't want to use native dates for some obscure reason (I couldn't think of a good one) or don't want to convert your whole database (really, you should) and need a higher precision, you might consider to add new values as floating point values. This ensures interoperability with the old data because integers and floating point values usually can be converted and compared between each other easily.
I have an existing PostgreSQL database with lots of columns with data types like "numeric(27,10)" or "numeric(17,2)". In the Sails models I've set up for each table, I list the type of each attribute as either integer or float. However, when Sails outputs JSON for these tables, the data are represented as strings.
I can always run something like parseInt() on the client side, but is there a better way to fix this by reconfiguring sails?
is there a better way to fix this by reconfiguring sails?
Not really. The issue is that the numeric data type in Postgres handles numbers with up to 131,072 digits before the decimal point--waayyyy bigger than any number your computer (or probably any computer) can handle in memory. So Waterline would have to decide what to do with cases where it couldn't cast the value to a float or integer (return an error, or a string?). Instead, it just takes the safe, predictable route and returns a string every time, which you'll have to handle in userland code.
I am storing enums for things such as ranks (administrator, moderator, user...) and achievements for each user in my Mongo database. As far as I know Mongo does not have an enum data type which means I have to store it using another type.
I have thought of storing it using integers which I would assume uses less space than storing strings for everything that could easily be expressed as an integer. Another upside I see of using integers is that if I wanted to rename an achievement or rank I could easily change it without even having to touch the database. A benefit I see for using strings is that the data requires less processing before it is used and is more human readable which could help in tracking down bugs.
Are there any better ways of storing enums in Mongo? Is there an strong reason to use either integers or strings? (trying to stay away from a which is better question)
TL;DR: Strings are probably the safer choice, and the performance difference should be negligible. Integers make sense for huge collections where the enum must be indexed. YMMV.
I have thought of storing it using integers which I would assume uses less space than storing strings for everything that could easily be expressed as an integer
True.
other upside I see of using integers is that if I wanted to rename an achievement or rank I could easily change it without even having to touch the database.
This is a key benefit of integers in my opinion. However, it also requires you to make sure the associated values of the enum don't change. If you screw that up, you'll almost certainly wreak havoc, which is a huge disadvantage.
A benefit I see for using strings is that the data requires less processing before it is used
If you're actually using an enum data type, it's probably some kind of integer internally, so the integer should require less processing. Either way, that overhead should be negligible.
Is there an strong reason to use either integers or strings?
I'm repeating a lot of what's been said, but maybe that helps other readers. Summing up:
Mixing up the enum value map wreaks havoc. Imagine your Declined states are suddenly interpreted as Accepted, because Declined had the value '2' and now it's Accepted because you reordered the enum and forgot to assign values manually... (shudders)
Strings are more expressive
Integers take less space. Disk space doesn't matter, usually, but index space will eat RAM which is expensive.
Integer updates don't resize the object. Strings, if their lengths vary greatly, might require a reallocation. String padding and padding factor should alleviate this, though.
Integers can be flags (not yet queryable (yet), unfortunately, see SERVER-3518)
Integers can be queried by $gt / $lt so you can efficiently implement complex $or queries, though that is a rather arcane requirement and there's nothing wrong with $or queries...
Need suggestion on which Datatype would give better performance if we set one of these as primary key in DB2 - BIGINT or Decimal(13,0) type?
I suspect Decimal(13,0) will have issues once the key grows to a very big size but I wanted a better answer/understanding for this.
Thanks.
Decimal does not have issues. The only thing, is that DB2 has to do more operations to retrieve the data, once is read. I mean, DB2 read the data and then it should find the decimal part (the precision) even if is 0.
On the other hand, DB2 will read the BigInt and it does not need any further process. The number is on the bufferpool.
If you are going to use integers of 13 positions (most of them), probably Decimal will be better because you are not going to use extra bytes, however decimals have extra bytes for the precision. By using decimal in this way, you are going to optimize the storage, and this will be translated in better IO, better performance. However, it depends on the other columns of your table. You have to test which of them gives you better performance.
When using compression, there are more CPU cycles to recover the information. You have to test if the performance is affected.
Use BIGINT:
Can store ~19 digits (versus 13)
Will take 8 bytes (versus maybe either 7 or 13 - see next)
Depending on platform, DECIMAL will be stored as a form of Binary Coded Decimal - for example, on the iSeries (and I can't remember if it's Packed or Zoned). Can't speak to other deployments, unfortunately.
You aren't doing math on these values (things like "next entry" don't count) - save DECIMAL/NUMERIC for measurements/values.
Note that, really, ids are just a sequence of bits - the fact that it happens to be an integer (usually) is irrelevant. It's best to consider them random data; sequential assignment is an optimization detail, there's often gaps (rollbacks, system crashes, whatever), and they're meaningless for anything other than joining.