I'd like to import some data into a Redshift database using COPY. For reasons passing understanding one of the columns in the data is a timestamp that's given in seconds since 2000-01-01 00:00:00. Is there any way to turn these into proper timestamps on import?
Unfortunately, you cannot transform data in a Redshift COPY load. I think you will have to stage these to a load table and then do the transform in the insert to the final table.
Worth noting though that you could do this if they had used the standard Unix epoch (seconds since 1970-01-01 00:00:00) by adding TIMEFORMAT 'epochsecs' to your COPY.
Related
I have a postgresql database that I use to store datetimes that includes timezones (utc offsets). I used the timestampz type as it seems to be what I need, but when I insert a datetime, the UTC offset get converted to UTC+00:
For example, if I insert 2022-10-20 00:00:00+01 the actual data stored becomes 2022-10-19 23:00:00+00.
This doesn't look like mutch but we lost some information in the process. Is there a way to keep the offset without adding a column to store that information (or the timezone)?
No, and in that respect PostgreSQL arguably diverges from the SQL standard. If you want to retain the time zone information, you need an additional column. If you do that, consider storing IANA time zone names like Europe/Paris rather than UTC offsets.
What is the best way to store a timestamp value in Postgresql in a specific format.
For example I would like to store a TIMESTAMP '2020-07-09 17:29:30.873513Z' down to the minute and ignore seconds value.
I can drop the seconds by using date_trunc('minute', TIMESTAMP '2020-07-09 17:29:30.873513Z') Is there anyway for me to specify this format in the column itself when I create a table?
Don't store formatted timestamps in the database, use timestamp with time zone or timestamp without time zone. You would lose powerful datetime arithmetic, value checking and waste storage space.
To have the values truncated to minute precision, use a BEFORE INSERT trigger that uses date_trunc on the value.
If you want to ascertain that only such values are stored, add a check constraint.
I would like to recommend not to drop seconds or anything from the stored data. Because it will create issues while you process the data later. And if you have to eliminate anything, you may eliminate it while retrieving the data.
Use the following code while creation of table
col_name timestamp without time zone DEFAULT timezone('gmt'::text, now())
This will give you a result as shown in the following image:
Good Luck.
Converting epoch time to SQL datetime format. I am trying to convert the 35 000 records received from another table (other db) with epoch timestemp to a new table with SQL datetime format. Also I will need to have this updated on a daily basis so one time conversion is good but I am also open to other suggestions.
I never worked with epoch and I am not sure how to go about it.
Also later on I want to use it in SSRS with correct datetime so I am not sure should I convert it before the transfer to new table or to do it in SSRS?
Any ideas?
Presuming that the epoch timestamp you have is in seconds:
DATEADD(SECOND, epoch_col, '19700101')
This will add the epoch seconds to the start of 'epoch time' (01-01-1970 00:00:00) and give you a DATETIME.
Example with output:
SELECT DATEADD(SECOND, 1571994774, '19700101')
2019-10-25 09:12:54.000
If you have an epoch timestamp in milliseconds just use this variation:
DATEADD(MILLISECOND, epoch_col, '19700101')
In terms of your other question about when to convert the value; My view is that it would be preferable to store the value in a DATETIME column at point of insertion rather than storing the epoch value and converting it upon use.
This is just an opinion though and not a recommendation.
I have a table that I am using to store iso dates with timezones. I realize that dates should "always" be stored as utc but I have an exception to that rule. The timestamps aren't in any way related to the server they are running on. I want to be able to store an iso date like this:
2016-03-06T01:15:52-06:00
And regardless of the time zone of the server or anything else I want the timestamp returned as:
2016-03-06T01:15:52-06:00
Currently if I insert an iso date it automatically converts it to whatever the server timezone is. My above date gets converted to:
2016-03-06 07:15:52+00 (server is utc)
The only thing I can think of is storing the timezone offset in a separate column, storing my date as utc and then converting using the offset column, horribly messy. Surely there is a way to store my date in one column and get it out the way it was originally created?
Your proposed solution is correct. Or more precisely, it is one of several correct implementations. Any of the following would work:
Store the UTC timestamp in one field, store the offset in another.
Store the local timestamp in one field, store the offset in another.
Store the local date in one field, and store a time with time zone in another. (though time with time zone is generally discouraged...)
Store the UTC timestamps in one field and the local timestamp in another.
The easiest by far is the first one, which you already proposed.
I'd avoid against storing timestamps in text fields, as they tend not to be very efficiently searchable.
Also note - if you're coming from a SQL Server background, you might recall its datetimeoffset type, which stores the local datetime and offset in the field, and uses the UTC equivalent during indexing. It's common to think that Postgres and MySQL's timestamp with time zone would have the same behavior, but they don't. They simply use the session time zone to convert to/from UTC. SQL Server has no concept of a session time zone, and thus the discrepancy.
Be sure to read this part of the Postgres docs.
Redshift only support TIMESTAMP format, I want to dump some data that is originally stored in Postgres in time without time zone format. My data looks like this: 15:30:00
When I COPY my data into Redshift, it says Invalid timestamp format or value [YYYY-MM-DD HH24:MI:SS], error code 1206.
My current workaround is to create that column in Redshift as CHAR(8)
Thanks
There's no other solution that to use the char(8).
Redshift does not support timezones.
Workaround would be to parse data to UTC and store time difference in separate column. You will be able to calculate local time by adding/substracting hours to the UTC data.