how do I change the date-time stamp to unix timestamp in Oracle Golden Gate? There is no inbuilt function in Oracle for it and the function that I am writing might cause delays in GG replication. I cannot afford delays, since GG replicates real-time data
You can use the following solutions:
use SQLEXEC and call custom code (stored procedure) that would make the transformation
write an USER EXIT custom function in C code and call it from GoldenGate
The second solution would be fastest since there is no round trip when SQL code is being called in the database.
Related
I'm trying to implement an Extractor pipeline in ADF, with several Copy Data activities (SAP ERP Table sources). To save some processing time, I'd like to have some deltas (incremental load). What's the best way to implement this?
What I'm trying at the moment is just to use the "RFC table options" in each Copy Data activity. However, this seems to be quite limited (only very simple queries allowed). Also, each SAP ERP table requires a different query. I found 3 different situations, regarding table field formats:
Timestamp in miliseconds (e.g. COVP);
Timestamp in YYYYMMDDHHMMSS (e.g. FAGLFLEXA);
Last change date and last change time, in separated fields (e.g. CATSDB)
Has anyone ever tried this? What would you advise?
Thanks!
This is a hybrid question & public service announcement. The basic question is whether there are convenient and/or efficient workarounds to the limitations listed below. I spent half the morning discovering that one cannot just transplant SQL code from Access to Matlab. I've boiled it down to 3 points so far.
Can't use double quotes in SQL statements to avoid collision with
Matlab's string delimiter. The Matlab code for the SQL code strings
can become quite complicated, especially if the SQL strings already
use repeated single-quotes to represent a single quote within a
string constant.
Must always specify a source table from which to query. What won't
work is "SELECT #2015-07-28#". One basically needs to create a 1-row dummy table.
Must always select at least one field in the table being queried.
An asterisk does not seem to suffice.
The above limitations do not exist when submitting SQL code using the Access Query Designer (either in SQL 92 mode or not), nor do these limitations exist when submitting SQL code using VBA via CurrentProject.Connection.Execute.
Hopefully, this saves someone else some time in learning about these differences. And if anyone has found a workaround, that would be appreciated. Note that the above is in the context of using the JDBC/ODBC bridge (3rd of 3 illustrated configurations in the drivers documentation. The database toolbox documentation for directly connecting to an Access file using code (rather than setting up a data source using the GUI) only describes a code pattern that uses the JDBC/ODBC bridge. This is described in Example "Connect to Microsoft Access Using a File DSN" in the "Connect to database" page. I'd like to stick to this approach because I want to quickly be able to directly specify the source *.accdb file without jumping through GUI hoops of setting up a data source.
I've posted this to:
Stack overflow
Usenet
While I have not connected Matlab to Access, I have connected Access databases to PHP, Python, SAS, R, even Excel without any issues with quotes or asterisk. As for the needed source table in queries that is simply due to the Jet/ACE SQL dialect as each dialect have their particular rules. But your query above should work for one-row, one-column output of such date. SQL Server does not require a source table but Oracle does. Access dates are in the MM/DD/YYYY format, requiring # in conditional statements. MySQL use dates in YYYY-MM-DD format, requiring single quotes.
Are you using Windows installed Jet/ACE ODBC drivers? Do note these drivers are softwares external to any application (not part of Access) and can be used by any client interfacing to any data source. Aside - many are not aware that Access' backend database engine, Jet/ACE is actually a Windows technology and not restricted to Access; PC users without Access installed can still use this engine. Hence, connection strings are standard across ODBC calls. And as I see you can apply same principles to Matlab database connections.
With that said, my suggestions/workarounds:
properly learn the SQL dialect for connecting database and check if query runs; even optimize or re-write as needed
try escaping various punctuation not compliant in Matlab strings or use the ASCII counterparts: chr(34) for double quote; chr(39) for single quote; chr(42) for asterisk.
if select statements are limited in Matlab strings, create a temp table from query (use make-table query SELECT * INTO newtable FROM query outside of Matlab), then connect to this new table
use intermediary coding language (VBA, Python, C#) by user trigger or
command line to connect to database and export needed dataset, then
automate Matlab to import
export query to flatfile format (csv, txt, etc.) for Matlab import
I am trying to run RAWSQL_REAL("select sum(amount_us)from gbsa_dpo_itg.Fact_tblHistoryData_new where qtr_data='Q42014'") in calculated field and I am getting error message ERROR 2133: Aggregate function calls cannot contain subqueries.
I am using tableau 8.3.3 and HP Vertica database live connection to tableau
When I run the same query in custom sql it is working fine
pleas help in this
thanks in advance
Read the manual about these functions, look under reference, functions
You don't generally pass an entire SQL string to execute in isolation. Instead, they are useful for writing expressions or calling non standard functions that your server may provide, which are embedded within the SQL that Tableau generates. So first learn to use Tableau to get the effect you want, and then resort to Raw SQL functions in the rare case where you need to access some database server specific feature.
There is no reason that you would need Raw SQL to get the information above using Tableau. You could put amount_us on the row shelf and qtr_data on the filter shelf, and Tableau would generate a similar query.
If you are doing this to combine data from multiple queries, first learn about calculated fields and data blending.
I have written a VBScript to extract data from Active Directory into a record set. I'm now wondering what the most efficient way is to transfer the data into a SQL database.
I'm torn between;
Writing it to an excel file then firing an SSIS package to import it or...
Within the VBScript, iterating through the dataset in memory and submitting 3000+ INSERT commands to the SQL database
Would the latter option result in 3000+ round trips communicating with the database and therefore be the slower of the two options?
Sending an insert row by row is always the slowest option. This is what is known as Row by Agonizing Row or RBAR. You should avoid that if possible and take advantage of set based operations.
Your other option, writing to an intermediate file is a good option, I agree with #Remou in the comments that you should probably pick CSV rather than Excel if you are going to choose this option.
I would propose a third option. You already have the design in VB contained in your VBscript. You should be able to convert this easily to a script component in SSIS. Create an SSIS package, add a DataFlow task, add a Script Component (as a datasource {example here}) to the flow, write your fields out to the output buffer, and then add a sql destination and save yourself the step of writing to an intermediate file. This is also more secure, as you don't have your AD data on disk in plaintext anywhere during the process.
You don't mention how often this will run or if you have to run it within a certain time window, so it isn't clear that performance is even an issue here. "Slow" doesn't mean anything by itself: a process that runs for 30 minutes can be perfectly acceptable if the time window is one hour.
Just write the simplest, most maintainable code you can to get the job done and go from there. If it runs in an acceptable amount of time then you're done. If it doesn't, then at least you have a clean, functioning solution that you can profile and optimize.
If you already have it in a dataset and if it's SQL Server 2008+ create a user defined table type and send the whole dataset in as an atomic unit.
And if you go the SSIS route, I have a post covering Active Directory as an SSIS Data Source
We have a DB2 running on z/OS and some tables use a timestamp as a Primary Key.
My opinion is, that it might be possible that two transactions calling CURRENT TIMESTAMP in the same nanosecond can have exactly the same Timestamp returned.
My colleague thinks that the CURRENT TIMESTAMP function on the same database is always unique.
The DB2 documentation here is not very clear.
Is there an offical statement from IBM, which proofs the one or the other thesis? I found only a statement for UNIX DB2, which is maybe not applicable for z/OS.
Thank you.
There are instances when it won't be unique. They are:
Datetime special registers are stored in an internal format. When two or more of these registers are implicitly or explicitly specified in a single SQL statement, they represent the same point in time.
If the SQL statement in which a datetime special register is used is in a user-defined function or stored procedure that is within the scope of a trigger, DB2 uses the timestamp for the triggering SQL statement to determine the special register value.
Source: http://publib.boulder.ibm.com/infocenter/dzichelp/v2r2/topic/com.ibm.db2.doc.sqlref/xfbb68.htm#xfbb68
You should use GENERATE_UNIQUE() if you want a unique timestamp. Good example here: http://www.mainframesupport.dk/tips/tip0925.html
There is no guarantee that CURRENT TIMESTAMP will return a unique value.
I have seen many examples of DB/2 SQL INSERT statements in a z/os environment failing on duplicate key when CURRENT TIMESTAMP was used to populate a column defined as unique.
Once upon a time CURRENT TIMESTAMP had a fine enough "granularity" that the probability of a collision was extremely small. This lead to quite a few applications treating them as unique identifiers. Processors are faster and parallelism has increased tremendously over the years. Any process that expects unique values from CURRENT TIMESTAMP today is likely to crash and burn on a very regular basis.
Your colleague is running a bit behind the times (on a couple of levels).