I am having one requirement for SQL-like interface to query S3 object storage(It's not actually AWS S3, but S3 based protocol storage. We are having server name, bucket name, access_key and secret_access_key available with us).
I need to know whether there exists any PostgreSQL extension which can leverage foreign data wrapper(FDW) feature to run query against S3 data.
Here, we don't want to copy the data(csv files) from S3 to postgreSQL server. We have external PostgreSQL server, using any extension, we directly want to query that S3 data using that PostgreSQL engine and get the result on PostgreSQL server.
A very similar requirement link was found as below:
https://www.cdata.com/kb/tech/amazons3-jdbc-postgresql-fdw.rst
But it is using it's own cdata JDBC driver. We don't want anything proprietary.
Is there anything opensource available? If yes, how can we achieve that?
We are using 12.2 PostgreSQL version.
Related
I'm trying to use the data copy activity to extract information from azure databricks delta lake, but I've noticed that it doesn't pass the information directly from the delta lake to the SQL server I need, but must pass it to an azure blob storage, when running it, it throws the following error
ErrorCode=AzureDatabricksCommandError,Hit an error when running the command in Azure Databricks. Error details: Failure to initialize configurationInvalid configuration value detected for fs.azure.account.key Caused by: Invalid configuration value detected for fs.azure.account.key
Looking for information I found a possible solution but it didn't work.
Invalid configuration value detected for fs.azure.account.key copy activity fails
Does anyone have any idea how the hell to pass information from an azure databricks delta lake table to a table in Sql Server??
These are some images of the structure that I have in ADF:
In the image I get a message that tells me that I must have a Storage Account to continue
These are the configuration images, and execution failed:
Conf:
Fail:
Thank you very much
The solution for this problem was the following:
Correct the way the Storage Access Key configuration was being defined:
in the instruction: spark.hadoop.fs.azure.account.key..blob.core.windows.net
The following change must be made:
spark.hadoop.fs.azure.account.key.
storageaccountname.dfs.core.windows.net
Does anyone have any idea how the hell to pass information from an azure databricks delta lake table to a table in Sql Server??
To achieve Above scenario, follow below steps:
First go to your Databricks cluster Edit it and under Advance options >> spark >> spark config Add below code if you are using blob storage.
spark.hadoop.fs.azure.account.key.<storageaccountname>.blob.core.windows.net <Accesskey>
spark.databricks.delta.optimizeWrite.enabled true
spark.databricks.delta.autoCompact.enabled true
After that as you are using SQL Database as a sink.
Enable staging and give same blob storage account linked service as Staging account linked service give storage path from your blob storage.
And then debug it. make sure you complete Prerequisites from official document.
My sample Input:
Output in SQL:
I am very new to Apache Nifi. I am trying to Migrate data from Oracle to Mongo DB as per the screenshot in Apache NiFi. I am failing with the reported error. Pls help.
Till PutFile i think its working fine, as i can see the below Json format file in my local directory.
Simple setup direct from Oracle Database to MongoDb without SSL or username and password (not recommended for Production)
Just keep tinkering on PutMongoRecord Processor until you resolve all outstanding issues and exclamation mark is cleared
I am first using an ExecuteSQL processor which is resulting the dataset in Avro, I need the final data in JSON. In DBconnection pooling Service, you need to create a controller with the credentials of your Orcale database. Post that I am using Split Avro and then Transform XML to convert it into JSON. In Transform XML, you need to use XSLT file. After that, I use PutMongo Processor for ingestion in Json which gets automatically converted in BSON
I have a need to dynamically create databases in code via the OrientJS driver. I see an example of how to create an encrypted DB using the Java API but nothing in the JS api/OrientJS.
Sorry, this is not available through remote OrientDB binary protocol, so you cannot do that via JS but also Java API and Console if you use remote protocol.
I have created an SQLDB service instance and bound it to my application. I have created some tables and need to load data into them. If I write an INSERT statement into RUN DDL, I receive a SQL -104 error. How can I INSERT SQL into my SQLDB service instance.
If you're needing to run your SQL from an application then there are several examples (sample code included) of how to accomplish this at the site listed below:
http://www.ng.bluemix.net/docs/services/SQLDB/index.html#run-a-query-in-java
Additionally, you can execute SQL in the SQL Database Console by navigating to Manage -> Work with Database Objects. More information can be found here:
http://www.ng.bluemix.net/docs/services/SQLDB/index.html#sqldb_005
s.executeUpdate("CREATE TABLE MYLIBRARY.MYTABLE (NAME VARCHAR(20), ID INTEGER)");
s.executeUpdate("INSERT INTO MYLIBRARY.MYTABLE (NAME, ID) VALUES ('BlueMix', 123)");
Full Code
Most people do initial database population or migrations when they deploy their application. Often these database commands are programming language specific. The poster didn't include the programming language. You can accomplish this two ways.
Append a bash script that would call your database scripts that you uploaded. This project shows how you can call that bash script from within your manifest file as part of doing a CF Push.
Some languages like offer a file type or service that will automatically get used to populate the database on initial deploy or when your migrate/synch the db. For example Python Django offers a "fixtures" file that will automatically take a JSON file and populate your database tables
How do I add new Database support (MongoDB) in 2.6.3 version of WSO2 Data Service Server.
You can use DSS (2.6.3) with any database type if the database connectivity is exposed via JDBC. In other words, if your preferred database type exposes a JDBC driver/adapter for the users to connect to it via JDBC then you can use DSS to expose your data stored in your data store as a web service. Similarly, if MongoDB too has a JDBC adapter you wouldn't have any (or too many :) ) issues integrating that with DSS. However, there are some exceptions when it comes to exposing flat files such as google spreadsheeets, excel sheets, csv files as DSS uses the relevant client APIs such as Google gdate client API, Apache POI, etc to connect to those datasources and extract data. However, if we consider the general case is you need to have an adapter or a similar mechanism to connect to your datasource via JDBC.
But in the upcoming version of DSS (v3.0.0), it is planned to introduce custom datasource support so you can easily write an adapter to any datasource and use it with DSS.
Regards,
Prabath
I am not sure about this, but I suppose that if is not supported by default you can always download the jar library for MongoDB and put it in CARBON_HOME\repository\components\lib and restart. For example for mysql I have the mysql-connector-java-5.1.7-bin.jar in that folder.
Hope this help