I am loading a data into IBM cloud and it has a date and time format that looks like this, 08/28/2004 05:50:56 PM. what would be the correct time format? i have tried given available formats but they do not match. If i need to create one, how do i go about it? Thank you.
I believe that the Db2 Console for Db2 on Cloud currently uses LOAD statements to get data into Db2. The Db2 Console for Db2 Warehouseon Cloud uses both LOAD and EXTERNAL TABLE (for Netezza/PDA sourced data)
The timestamp format options for LOAD are less flexible than those available with EXTERNAL TABLE based loads in Db2 11.5
You need to use this format
MM/DD/YYYY HH:MM:SS TT
as per the available elements listed under the timestampformat option on this page https://www.ibm.com/support/knowledgecenter/en/SSEPGG_11.5.0/com.ibm.db2.luw.admin.cmd.doc/doc/r0008305.html
Here is an client command line example.
$ cat file.csv
08/28/2004 05:50:56 PM
$ db2 "create table ts(ts timestamp)"
DB20000I The SQL command completed successfully.
$ db2 'import from file.csv of del modified by timestampformat="MM/DD/YYYY HH:MM:SS TT" insert into ts'
SQL3109N The utility is beginning to load data from file "file.csv".
SQL3110N The utility has completed processing. "1" rows were read from the
input file.
SQL3221W ...Begin COMMIT WORK. Input Record Count = "1".
SQL3222W ...COMMIT of any database changes was successful.
SQL3149N "1" rows were processed from the input file. "1" rows were
successfully inserted into the table. "0" rows were rejected.
Number of rows read = 1
Number of rows skipped = 0
Number of rows inserted = 1
Number of rows updated = 0
Number of rows rejected = 0
Number of rows committed = 1
$ db2 "select * from ts"
TS
--------------------------
2004-08-28-17.50.56.000000
1 record(s) selected.
The following works:
values to_date('08/28/2004 05:50:56 PM', 'MM/DD/YYYY HH:MI:SS AM')
So, try this format.
Related
I have a pyspark script that processes a dataframe containing a date & time, called startdate, for which I use the following code to convert it into a timestamp format:
## Convert startdate to timestamp
df = df.withColumn('startdate', regexp_replace('startdate', 'T', ' ')) \
.withColumn("startdate",expr("substring(startdate, 1, length(startdate)-1)")) \
.withColumn('startdate',to_timestamp(col('startdate'), 'yyyy-MM-dd HH:mm:ss'))
I'm creating a type 2 slowly changing dimension, so for every ID, I'm taking the start date and "lagging it" to create an end date. For this, I run the following code:
# Get end date
df = df.withColumn("rank",
dense_rank().over(Window.partitionBy('id').orderBy(desc("startdate"))))
partition = Window().partitionBy().orderBy(col('id'))
df = df.withColumn("enddate",when(df.rank == 1,lit(None)).otherwise(lag("startdate").over(partition)))
This all works fine and the script outputs to parquet files in data lake Gen 2, and when I display my output I can see the right output. But the next step that I'm doing which is that I have an ADF pipeline that copies data from parquet files into an Azure SQL database fails because it isn't turning the 'undefined' (or the output of the lit(none) part of the script) values into NULL in the database.
My question is, what do I need to do in my script above, where I'm currently using lit(none) to get an output that will be compatible to turn into a NULL 'value' in the SQL database as part of my pipeline?
The column in my database is of datetime2 type, and is nullable. The startdate column is working fine, but it will never be NULL / empty, which is why I've concluded the issue here is with the 'empty' values.
The question was asked in error as something else was obstructing my pipeline, but to answer the question, of how to produce an "empty" value in parquet files, that becomes a NULL value in a SQL database, the following values can be used:
lit(None)
None
So the following line in my code was actually working:
df = df.withColumn("enddate",when(df.rank == 1,lit(None)).otherwise(lag("startdate").over(partition)))
I am a SQL DBA trying to get some information in Cognos.
The column name is [Packingdate] column.
In order to retrieve last 7 days of packing data information I use the below.
where [Packingdate] >_add_days( current_date, -7 ) -This is working fine.
However when i tried to get November month data I tried the below
[Packingdate] between '2021-11-01' and '2021-11-30'
It is giving me the below error
RQP-DEF-0177 An error occurred while performing operation 'sqlPrepareWithOptions' status='-126'.
Ami missing something?
Remove the quotes:
[Packingdate] between 2021-11-01 and 2021-11-30
When dealing with date or datetime data, YYYY-MM-DD (without quotes) is the format Cognos understands.
I have a varchar column in redshift table where the time is stored in 24 hours format, e.g, 17:00, I want to query the table and convert the format to 12 hours format showing AM or PM in time. When I test like to_char('17:00'::time,'HH12:MI AM') it works fine but when I put column name in place of hardcoded value querying the table,
SELECT to_char(prepoll_start::time,'HH12:MI AM')
FROM votecast.poll_hours AS ph
WHERE ph.prepoll_start is not null
and state = 'AL'
AND tab_elec_type = 'primary'
It won't work, gives an error
Invalid operation: Specified types or functions (one per INFO message) not supported on Redshift tables.;
Postgres version is 8.0.2
Please let me know what am doing wrong :(
First I had to create a timestamp value out of the time available. Then fetch the time in a 12 hour format.
select to_char( to_timestamp('1900-01-01 '||prepoll_start||':00' ,'YYYY/MM/DD HH:MI:SS') , 'HH12:MI AM')
from votecast.poll_hours;
Amazon Redshift does not support a TIME data type.
See: Datetime Types - Amazon Redshift
However, you are correct that it seems to support TIME for non-table related operations.
I tried playing around with string manipulation but was unable to get beyond the error you experienced. I think it is because TIME is recognized on the leader node, but fails to run on the compute nodes. (This is similar to the behaviour of time_series().)
Thus, you won't be able to use the TIME data type for anything that relies on table data.
I have IOT data with a Unix time. I need to show this to the second in a Data Studio table report. Cannot make this work. In the PubSub function I have:
d = new Date();
d.setTime(res[0] * 1000); // 1000 is for JS time
console.log( d.toISOString());
and later set the field with:
timestamp:d.toISOString(),
Using BigQuery the table shows the field as:
2018-07-06 23:44:49 UTC
which is correct. The 'timestamp' field in Data Studio appears in the table as just:
July 6, 2018
I need to get more resolution, down to the second, for the table and eventually graphs. I've tried custom queries for the data source but cannot get Data Studio to show better resolution.
I tried creating a new field using TODATE(timestamp, 'RFC_3339', "%Y%m%d%H") but cannot that also just shows the July 7, 2018 format. Cannot get it to go to greater resolution.
Yes, I've tried all the other approaches suggested in other questions but none match what I am trying, or they don't succeed.
From what I can see on the DataStudio documentation, this product doesn't support seconds for their date output formats, it only supports up to the minute for output formats and for the 'mm' date type.
I suggest to vote on the related feature requests to support more specific time resolution. 1 2
I have a talend solution where inside it rests a tMap --> tPostgreSQLOutput.
Inside the schema is a integer(key field) and a Date(Timestamp) in the format of "dd-MM-yyyy HH:mm:ss". The intent is to update the date field with the current time/date (Timestamp).
the date is set with this talend function call in the tMap:
TalendDate.parseDate("yyyy-MM-dd HH:mm:ss", TalendDate.getDate("yyyy-MM-dd HH:mm:ss"))
I have confirmed the date(timestamp) format, and confirmed that the timestamp data type in the PostgreSQL database. However, I'm getting this error upon runtime:
Batch entry 0 UPDATE "bitcoin_options" SET "last_notified" = 2017-04-08 12:02:40.000000 -05:00:00 WHERE "id" = 3 was aborted. Call getNextException to see the cause.
I took the query it errored and manually ran it into PostgreSQL. I got this response:
ERROR: syntax error at or near "11"
LINE 1: ...bitcoin_options" SET "last_notified" = 2017-04-08 11:53:11.0...
^
Again, I checked the format, the datatype, and compared them against other tables and their UPSERTS. same format. same datatype.
In addition, I attempted to add a second space between date and time, with no avail.
UPDATE 1
I updated the tMap output to:
TalendDate.getCurrentDate();
and got the same error.
Thanks
UPDATE 2
Here's my layout for Talend:
I figured it out. after much trial and error.
The tPostgresSQLCommit x3 was redundant. When I removed the first two and placed just one, it gave me the proper output.
LESSONS LEARNED: You only need 1 commit.
Notice your timestamp is not properly formatted:
UPDATE "bitcoin_options" SET "last_notified" = '2017-04-08 12:02:40.000000 -05:00:00' WHERE "id" = 3
It's missing the single quotes surrounding the timestamp. If you add those you should be good to go.