Can Oracle golden gate adapter read trail file which generated by Extract or Pump process? - ogg

I am researching OGG, Can Oracle golden gate adapter read trail file which generated by Extract or Pump process? Most example I can find is OGG adapter works for replication process on delivery server, I just wanna apply the adapter whatever via plat file, jms or java api on extract or pump process directly.

The trail file is just an intermediate internal Oracle GoldenGate format which is used to store the data. The source is processed by the Extract process.
The Extract process can read the data from:
a database transaction log,
a JMS Queue or a flat file (through GoldenGate for Application Adapters)
The Extract process produces as output:
trail files (to be processed by Replicat processes)
XML files
SQL files
If you set a Trail file as the output, then the trail can be processed by:
another Extract (called Data Pump) to transfer the trail files to another location
a replicat Process - to apply the transactions to a destination
Later on the Replicat process can read the trails (produced by Extract) and apply the transactions to:
a database,
a JMS Queue or flat file (through GoldenGate for Application Adapters),
Hadoop, target (through GoldenGate for BigData)
The trail files have internal Oracle proprietary format and can not be produced by non-Oracle programs. You can read their content using the logdump program.

Related

Instant logging and file generation in Db2

I am currently using Community version on linux server, have configured db2audit process
Which generated audit files at respective location. Then user have to manually execute db2audit archive command to achieved logs file and have to execute thedb2 extract command to extract the archived files into flat ascIII files and then we have to load the files into respective tables.
There only we can analyze the logs by query the tables. In this whole process lots of manual intervention is required.
Question:-Do we have any config settings or utility
with the help of which we can generate logs files which include SQL statement event, host, session id,Timestamp and all instantly and automatically.
need to set instant level logging mechanism to generate flat files for logs of any SQL execution happened or any event triggered in database level in DB2 on linux server

Move file after transaction commit

I just started using Spring Batch and I don't know how I can implement my business need.
The behavior is quite simple : I have a directory where files are saved. My batch should detect those files, import them into my database and move the file to a backup directory (or an error directory if the data can't be saved).
So I create chunks of 1 file. The reader retrieve them and the processor imports the data.
I read Spring Batch create a global transaction for the whole chunk, and only the ChunkListener is called out of the transaction. It seems to be OK, but the input parameter is a ChunkContext. How can I retrieve the file managed in the chunk ? I don't see where it's stored in the ChunkContext.
I need to be sure the DB accepts the insertions before choosing where the file must be moved. That's why I need to do that after the commit.
Here is how you can proceed:
Create a service (based on a file system watching API or something like Spring Integration directory polling) that launches a batch job for the new file
The batch job can use a chunk-oriented step to read data and write it to the database. In that job, you can use a job/step execution listener or a separate step to move files to the backup/error directory according to the success or failure of the previous step.

Visual Studio Data Load from flat file to Postgres Database

I am using Visual Studio to transfer data from a CSV file to the Postgres database. My database is installed on windows server 2012 and I'm using my local machine to transfer data. My process runs successfully without throwing any errors but somehow it's not loading all the rows of my CSV file. The file contains 382,363 rows but after I check my database only 26000 rows have been loaded.
I directly loaded the CSV from the import wizard of Postgres and it successfully loaded all the 382,363 rows but when I'm loading data through visual studio it just loads 26000 rows without throwing any error. I just get two warning msgs. Has anyone faced any issue if so how do I solve it?
Pasting below the entire output of my process
SSIS package "C:\Users\Shivam SARIN\source\repos\Integration Services Project3\Integration Services Project3\Package.dtsx" starting.
Information: 0x4004300A at Data Flow Task, SSIS.Pipeline: Validation phase is beginning.
Information: 0x4004300A at Data Flow Task, SSIS.Pipeline: Validation phase is beginning.
Warning: 0x80049304 at Data Flow Task, SSIS.Pipeline: Warning: Could not open global shared memory to communicate with performance DLL; data flow performance counters are not available. To resolve, run this package as an administrator, or on the system's console.
Warning: 0x80047076 at Data Flow Task, SSIS.Pipeline: The output column "T_CTRY_DESTINATION" (115) on output "Flat File Source Output" (6) and component "Flat File Source" (2) is not subsequently used in the Data Flow task. Removing this unused output column can increase Data Flow task performance.
Information: 0x40043006 at Data Flow Task, SSIS.Pipeline: Prepare for Execute phase is beginning.
Information: 0x40043007 at Data Flow Task, SSIS.Pipeline: Pre-Execute phase is beginning.
Information: 0x402090DC at Data Flow Task, Flat File Source [2]: The processing of file "C:\Users\Shivam SARIN\Documents\Excel-csv\MS 2018 Q3.csv" has started.
Information: 0x4004300C at Data Flow Task, SSIS.Pipeline: Execute phase is beginning.
Information: 0x402090DE at Data Flow Task, Flat File Source [2]: The total number of data rows processed for file "C:\Users\Shivam SARIN\Documents\Excel-csv\MS 2018 Q3.csv" is 382364.
Information: 0x402090DF at Data Flow Task, OLE DB Destination [275]: The final commit for the data insertion in "OLE DB Destination" has started.
Information: 0x402090E0 at Data Flow Task, OLE DB Destination [275]: The final commit for the data insertion in "OLE DB Destination" has ended.
Information: 0x40043008 at Data Flow Task, SSIS.Pipeline: Post Execute phase is beginning.
Information: 0x402090DD at Data Flow Task, Flat File Source [2]: The processing of file "C:\Users\Shivam SARIN\Documents\Excel-csv\MS 2018 Q3.csv" has ended.
Information: 0x4004300B at Data Flow Task, SSIS.Pipeline: "OLE DB Destination" wrote 382363 rows.
Information: 0x40043009 at Data Flow Task, SSIS.Pipeline: Cleanup phase is beginning.
SSIS package "C:\Users\Shivam SARIN\source\repos\Integration Services Project3\Integration Services Project3\Package.dtsx" finished: Success.
The program '[21956] DtsDebugHost.exe: DTS' has exited with code 0 (0x0).
In my experience with postgres, I found that using the oledb destination to insert rows was incredibly slow and weird things as you noted above seemed to happen. It's slow because the provider does not support bulk insert operations.
My suggestion is to call psql with an Execute Process Task. This will be much faster and you will be using tooling that is native to postgres.
More on that here: https://www.postgresql.org/docs/9.6/app-psql.html

How to Schedule Task using Marklogic

These are following areas where Scheduling Task Using Marklogic can be used
1.Loading content. For example, periodically checking for new content from an external data source, such as a web site, web service, etc.
2.Synchronizing content. For example, when MarkLogic is used as a metadata repository, you might want to periodically check for changed data.
3.Delivering batches of content: For example, initiate an RSS feed, hourly or daily.
4.Delivering aggregated alerts, either hourly or daily.
5.Delivering reports, either daily, weekly, or monthly.
6.Polling for the completion of an asynchronous process, such as the creation of a PDF file
My requirement is to schedule a task for bulk loading data from local file system to Marklogic DB using any data loading option available in Marklogic such as
1.MLCP
2.Xquery
3.Rest API
4.Java API
5.WebDAV.
So is there any option to execute this programatically. I prefer MLCP since I need to perform bulk load of data from local file system
Similar to your question at Execute MLCP Content Load Command as a schedule task in Marklogic , I would start with a tool like Apache Camel. There are other options - Mule, Spring Integration, and plenty of commercial/graphical ETL tools - but I've found Camel to be very easy to get started with, and you can utilize the mlcp Camel component at https://github.com/rjrudin/ml-camel-mlcp .

Crystal Reports Server 2013 - RPT file location

I have my Crystal Reports server configured against an Oracle data source, but now I need to switch it to be configured against a SQL Server data source, on a different server. This server has no way to communicate with the original server to use the import tool.
I was looking into how to import the data myself. Looking at the tables, I see that it stores binary fields in the database. Are the reports stored as binary data inside the data source, or does each RPT file has a copy somewhere on the disk? If it is on the disk where? If it's inside the data source binary field, does importing the data to a new server will still work?
The CCM (Central Configuration Manager) – not to be confused with the CMS – has the ability to migrate your database repository from one database to another, even if the database vendor differs (i.e. Oracle vs. Microsoft).
Have a look at the administrator guide, the procedure is described in chapter 11, Managing Central Management Server (CMS) Databases, section 4 Copying data from one CMS system database to another.
A small excerpt:
You can use the Central Configuration Manager (CCM) or cmsdbsetup.sh
to copy system data from one database server into another database
server. For example, if you want to replace the database with another
database because you are upgrading the database or are moving from one
database type to another, you can copy the contents of the existing
database into the new database before decommissioning the existing
database.
Regarding the reports themselves, they are not stored in the database, but in the file repository. However, you need to view the database and file repository as one: the first contains the metadata while the latter contains the actual files. However, they can not function separately.
In other words, you cannot just copy the RPT files from the file repository to another server and expect them to work, as the information that is contained in the metadata (such as authorisations) will be missing. On the other hand, if you just copy the database repository but omit the file repository, you'll end up with all the information regarding your reports, but you won't be able to open or run them.