I am trying to create an external table in DB2 warehouse in the cloud with the following command.
CREATE EXTERNAL TABLE EXT_5 (ID INT, AGE INT)
USING (DATAOBJECT ('abc.csv')
DELIMITER ','
MAXERRORS 10
SOCKETBUFSIZE 30000
REMOTESOURCE 'LOCAL')
The response to this query is a success. But getting an error as unsupported client when trying to insert/select rows to/from this external table.
EXTERNAL TABLEs require version 11.1.2.2 or above of the Db2 Client.
E.g. see the line item "External table support by ODBC/CLI" here https://www.ibm.com/support/knowledgecenter/SSEPGG_11.1.0/com.ibm.db2.luw.wn.doc/doc/c0060895.html
You can download the latest version of your Client from here http://www-01.ibm.com/support/docview.wss?uid=swg27016878
Related
I'm new to Db2. I'm trying to send data from remote Db2 server A to remote Db2 server B using a Java based application. I was able to fetch the data from server A and get it stored in the control/data files; but when I try to send the data to server B, I get following exception.
com.ibm.db2.jcc.am.SqlSyntaxErrorException: DB2 SQL Error: SQLCODE=-104, SQLSTATE=42601, SQLERRMC=EXTERNAL;T_DATA SELECT * FROM;<table_expr>, DRIVER=4.26.14
The control file has the command:
INSERT INTO <TABLE_NAME> SELECT * FROM EXTERNAL '<PATH_TO_DATAFILE>'
USING (DELIMITER '\t' FORMAT TEXT SOCKETBUFSIZE 100 REMOTESOURCE 'JDBC')
The data file contains records where each value separated by tab per record.
Both server A and B are using Db2 v9.5
The failure was caused by the target server-B being an out of support version of Db2 (v9.5) that does not have any ability to understand external tables. Hence it reported (correctly) sqlcode -104 on the token EXTERNAL which it did not understand.
So the design is incorrect for the available Db2-versions at your site. You can only use external tables in Db2-LUW versions that are recent (v11.5).
Depending on the tools available, you can use commands (external tools, not SQL) to export data from the source, and load it into the target. Additionally, if there is network connectivity directly between server-A and server-B then an administrator can arrange federation between them allowing direct inserts.
Db2 v9.5 also supported load from cursor, and load from remote cursor (although there were problems, long since fixed in newer versions).
I am trying to import a JOBFILE.CSV from my hard drive into the table JOB using the RUN SQL script in IBM DB2 cloud.
CALL SYSPROC.ADMIN_CMD('IMPORT FROM "C:/DATAFILE/JOBFILE.CSV"
OF DEL INSERT INTO JOB');
I am getting this error:
An I/O error (reason = "sqlofopn -2029060079") occurred while opening
the input file.. SQLCODE=-3030, SQLSTATE= , DRIVER=4.25.1301
It seems the path that I have set is not working. As I have researched JOBFILE.CSV must be loaded first into the DB2 server before the IMPORT script could run.
With a file located on a local client there are two options "basic" options (excluding e.g. Db2 Cloud REST API to import the data)
LOAD with a CLIENT keyword (work also for all Db2 LUW on-premise releases)
Insert from an EXTERNAL TABLE (available in Db2 Cloud, Warehouse and 11.5 release)
The latter is typically the fastest. See an example with an input like this:
db2 "create table import_test(c1 int, c2 varchar(10))"
echo "1,'test1'" > data.del
echo "2,'test2'" >> data.del
To insert the data from the client we need we can run:
db2 "INSERT INTO import_test SELECT * FROM EXTERNAL '/home/db2v111/data.del' USING (DELIMITER ',' REMOTESOURCE YES)"
DB20000I The SQL command completed successfully.
db2 "select * from import_test"
C1 C2
----------- ----------
2 'test2'
1 'test1'
2 record(s) selected.
More examples including importing data from S3 can be found in Loading data to IBM Cloud chapter of the documentation.
I am currently trying to use a dashDB database with the db2cli utility and ODBC (values are from Connect/Connection Information on the dashDB web console). At this moment I can perfectly do SELECT or INSERT statements and fetch data from custom tables which I have created, thanks to the command:
db2cli execsql -connstring "DRIVER={IBM DB2 ODBC DRIVER - IBMDBCL1}; DATABASE=BLUDB; HOSTNAME=yp-dashdb-small-01-lon02.services.eu-gb.bluemix.net; PORT=50000; PROTOCOL=TCPIP; UID=xxxxxx; PWD=xxxxxx" -inputsql /tmp/input.sql
Now I am trying to do a DB2 LOAD operation through the db2cli utility, but I don't know how to proceed or even if it is possible to do so.
The aim is to import data from a file without cataloging the DB2 dashDB database on my side, but only through ODBC. Does someone know if this kind of operation is possible (with db2cli or another utility)?
The latest API version referenced from the DB2 on Cloud (ex DashDB) dashboard is available here. It requires first to call the /auth/tokens endpoint to generate an auth token based on your Bluemix credentials to be used to authorize the API calls.
I've published recently a npm module - db2-rest-client - to simplify the usage of these operations. For example, to load data from a .csv file you can use the following commands:
# install the module globally
npm i db2-rest-client -g
# call the load job
export DB_USERID='<USERID>'
export DB_PASSWORD='<PASSWORD>'
export DB_URI='https://<HOSTNAME>/dbapi/v3'
export DEBUG=db2-rest-client:cli
db2-rest-client load --file=mydata.csv --table='MY_TABLE' --schema='MY_SCHEMA'
For the load job, a test on Bluemix dedicated with a 70MB source file and about 4 million rows took about 4 minutes to load. There are also other CLI options as executing export statement, comma separated statements and uploading files.
This is not possible. LOAD is not an SQL statement, therefore it cannot be executed via an SQL interface such as ODBC, only using the the DB2 CLP, which in turn requires a cataloged database.
ADMIN_CMD() can be invoked via an SQL interface, however, it requires that the input file be on the server -- it won't work with a file stored on your workstation.
If JDBC is an option, you could use the CLPPlus IMPORT command.
You can try loading data using REST API.
Example:
curl --user dashXXX:XXXXXX -H "Content-Type: multipart/form-data" -X POST -F loadFile1=#"/home/yogesh/Downloads/datasets/order_details_0.csv" "https://yp-dashdb-small-01-lon02.services.eu-gb.bluemix.net:8443/dashdb-api/load/local/del/dashXXX.ORDER_DETAILS?hasHeaderRow=true×tampFormat=YYYY-MM-DD%20HH:MM:SS.U"
I have used the REST API and have not seen any size limitations. In ver 1.11 of dashDB local (warehouse db) external tables have been included. As long as file is on the container it can be loaded. Also the DB2 Load locks the table until load is finished where a external table load won't
There are a number of ways to get data into Db2 Warehouse on Cloud. From a command line you can use Lift CLI https://lift.ng.bluemix.net/ which provides the best performance for large data sets
You can also use EXTERNAL TABLEs https://www.ibm.com/support/knowledgecenter/ean/SS6NHC/com.ibm.swg.im.dashdb.sql.ref.doc/doc/r_create_ext_table.html which are also high performance and have lots of options
This is a quick example using a local file (not on the server) hence the REMOTESOURCE YES option
db2 "create table foo(i int)"
echo "1" > /tmp/foo.csv
db2 "insert into foo select * from external '/tmp/foo.csv' using (REMOTESOURCE YES)"
db2 "select * from foo"
I
-----------
1
1 record(s) selected.
for large files, you can use gzip, either on the fly
db2 "insert into foo select * from external '/tmp/foo.csv' using (REMOTESOURCE GZIP)"
or from gzip'ed files
gzip /tmp/foo.csv
db2 "insert into foo select * from external '/tmp/foo2.csv.gz' using (REMOTESOURCE YES)"
I'm having trouble making loads of tables that have CLOBS and BLOBS columns in a 'SQL Database' database in Bluemix.
The error returned is:
SQL3229W The field value in row "617" and column "3" is invalid. The row was
rejected. Reason code: "1".
SQL3185W The previous error occurred while processing data from row "617" of
the input file.
The same procedures performed in a local environment functioned normally.
under the command you use to load:
load client from /home/db2inst1/ODONTO/tmp/ODONTO.ANAMNESE.IXF OF IXF LOBS FROM /home/db2inst1/ODONTO/tmp MODIFIED BY IDENTITYOVERRIDE replace into USER12135.TESTE NONRECOVERABLE
The only manner currently you can upload lob files to a SQLDB or dashDB is to load the data and lobs from the cloud. The option is to get data from a Swift object storage in Softlayer or a Amazon S3 storage. You should have an account on one of those services.
After that, you can use the following syntax:
db2 "call sysproc.admin_cmd('load from Softlayer::softlayer_end_point::softlayer_username::softlayer_api_key::softlayer_container_name::mylobs/blob.del of del LOBS FROM Softlayer::softlayer_end_point::softlayer_username::softlayer_api_key::softlayer_container_name::mylobs/ messages on server insert into LOBLOAD')"
Where:
mylobs/ is the directory inside the Softlayer swift object storage container, defined in
LOBLOAD is the table name to be loaded in
Example:
db2 "call sysproc.admin_cmd('load from Softlayer::https://lon02.objectstorage.softlayer.net/auth/v1.0::SLOS424907-2:SL523907::0ac631wewqewre8af20c576ad5214ec70f163d600d247bd5d4dfef5453f72ff6::TestContainer::mylobs/blob.del of del LOBS FROM Softlayer::https://lon02.objectstorage.softlayer.net/auth/v1.0::SLOS424907-2:SL523907::0ac631wewqewre8af20c576ad5214ec70f163d600d247bd5d4dfef5453f72ff6::TestContainer::mylobs/ messages on server insert into LOBLOAD')"
I was used pentaho data integration 4.4.0 and using mongo hadoop connector using this I was created successfully hadoop and mongo connections. Then I installed hive 0.11.0 and using above link I was created hive and mongo connections successfully. In my mongo contains one database name as pentaho and I created database in hive name as demo and using following command I was created new table name as pentho
CREATE TABLE pentaho
(
id INT,
region STRING,
year INT,
q1 INT,
q2 INT,
q3 INT,
q4 INT
)
STORED BY 'com.mongodb.hadoop.hive.MongoStorageHandler'
WITH SERDEPROPERTIES('mongo.columns.mapping'='{"id":"_id"}')
TBLPROPERTIES('mongo.uri'='mongodb://localhost:27017/pentaho.sales');
then I was write select * from pentaho; command on hive shell it shows all records presents in sales collections.
Then I was created model in pentaho using hadoop hive datasource set host as localhost database name as demo and port as 10000 and click on test then it shows popup connections made success. But when I was click on ok then new coming popup windows options like Schemas, Tables, Views, Synonyms but in table not contains any table which I was created in hive. So how can access hive tables in pentaho data source?
After time I found where I was missing I set in my local hadoop conf/mapred-site.xml file I set port as localhost:9001 and when I start bi-server same port used for hsql. Then after changing mapred-site.xml port other than 9001 it work fine. :)