Loading /Inserting Multiple XML files into Oracle table at once - oracle12c

I have a table xml_table_date. Below is the structure and sample data in the table.
But I want to insert multiple xml files (here it is 9) into the table in one go. These files resides in a DB directory
My code to insert xml file into the table:
CREATE TABLE xml_table_data (
File_name varchar2(100),
Insert_date timestamp
xml_data XMLTYPE
);
INSERT INTO xml_tab VALUES ( 'DataTransfer_HH_TWWholesale_001_004_12142020113003.xml',
XMLTYPE (BFILENAME ('TESTING', 'DataTransfer_HH_TWWholesale_001_004_12142020113003.xml'),NLS_CHARSET_ID ('AL32UTF8')));
Please help me on this.. Thanks for reading my query.

You can use an external table with preproxessing to read the filenames from the directory.
ALTER SESSION SET CONTAINER=pdb1;
CREATE DIRECTORY data_dir AS '/u02/data';
CREATE DIRECTORY script_dir AS '/u02/scripts';
CREATE DIRECTORY log_dir AS '/u02/logs';
GRANT READ, WRITE ON DIRECTORY data_dir TO demo1;
GRANT READ, EXECUTE ON DIRECTORY script_dir TO demo1;
GRANT READ, WRITE ON DIRECTORY log_dir TO demo1;
Create a list_files.sh file in the scripts-directory. Make sure oracle is the owner and the privileges are 755 on the file.
The preprocessing script file don't inherit the $PATH enviroment variable. So you have to prepend /usr/bin before all commands.
/usr/bin/ls -1 /u02/data/test*.xml | /usr/bin/xargs -n1 /usr/bin/basename
You also need a source file for the external table, but this can be an empty dummy file.
CREATE TABLE data_files
( file_name VARCHAR2(255))
ORGANIZATION EXTERNAL
(
TYPE ORACLE_LOADER
DEFAULT DIRECTORY data_dir
ACCESS PARAMETERS
( RECORDS DELIMITED BY NEWLINE CHARACTERSET AL32UTF8
PREPROCESSOR script_dir: 'list_files.sh'
BADFILE log_dir:'list_files_%a_%p.bad'
LOGFILE log_dir:'list_files_%a_%p.log'
FIELDS TERMINATED BY WHITESPACE
)
LOCATION ('dummy.txt')
)
REJECT LIMIT UNLIMITED;
Now you can insert the xml data into your table.
INSERT INTO xml_table_data
( file_name,
insert_date,
xml_data
)
SELECT file_name,
SYSTIMESTAMP,
XMLTYPE (BFILENAME ('DATA_DIR', file_name), NLS_CHARSET_ID ('AL32UTF8'))
FROM data_files;
You still need to adapt the example to your environment, please.

Related

Script does not create tables but pgAdmin does

I am running a script that creates a database, some tables with foreign keys and insert some data, but somehow creating the tables is not working, although it doesn't throw any error: I go to pgAdmin, look for the tables created and there's no one...
When I copy the text of my script and execute it into the Query Tool, it works fine and creates the tables.
Can you please explain me what I am doing wrong?
Script:
DROP DATABASE IF EXISTS test01 WITH (FORCE); --drops even if in use
CREATE DATABASE test01
WITH
OWNER = postgres
ENCODING = 'UTF8'
LC_COLLATE = 'German_Germany.1252'
LC_CTYPE = 'German_Germany.1252'
TABLESPACE = pg_default
CONNECTION LIMIT = -1
IS_TEMPLATE = False
;
CREATE TABLE customers
(
customer_id INT GENERATED ALWAYS AS IDENTITY,
customer_name VARCHAR(255) NOT NULL,
PRIMARY KEY(customer_id)
);
CREATE TABLE contacts
(
contact_id INT GENERATED ALWAYS AS IDENTITY,
customer_id INT,
contact_name VARCHAR(255) NOT NULL,
phone VARCHAR(15),
email VARCHAR(100),
PRIMARY KEY(contact_id),
CONSTRAINT fk_customer
FOREIGN KEY(customer_id)
REFERENCES customers(customer_id)
ON DELETE CASCADE
);
INSERT INTO customers(customer_name)
VALUES('BlueBird Inc'),
('Dolphin LLC');
INSERT INTO contacts(customer_id, contact_name, phone, email)
VALUES(1,'John Doe','(408)-111-1234','john.doe#bluebird.dev'),
(1,'Jane Doe','(408)-111-1235','jane.doe#bluebird.dev'),
(2,'David Wright','(408)-222-1234','david.wright#dolphin.dev');
I am calling the script from a Windows console like this:
"C:\Program Files\PostgreSQL\15\bin\psql.exe" -U postgres -f "C:\Users\my user name\Desktop\db_create.sql" postgres
My script is edited in Notepad++ and saved with Encoding set to UTF-8 without BOM, as per a suggestion found here
I see you are using -U postgres command line parameter, and also using database name as last parameter (postgres).
So all your SQL commands was executed while you are connected to postgres database. Of course, CREATE DATABASE command did creation of test01 database, but CREATE TABLE and INSERT INTO did executed not for test01 database, but for postgres database, and all your tables are in postgres database, but not in test01.
You need to split your SQL script into 2 scripts (files): first for 'CREATE DATABASE', second for the rest of.
You need to execute first script as before, like
psql.exe -U postgres -f "db_create_1.sql" postgres
And for second one need to choose the database which was created at 1st step, like
psql.exe -U postgres -f "db_create_2.sql" test01

db2 how to configure external tables using extbl_location, extbl_strict_io

db2 how to configure external tables using extbl_location, extbl_strict_io. Could you please give insert example for system table how to set up this parameters. I need to create external table and upload data to external table.
I need to know how to configure parameters extbl_location, extbl_strict_io.
I created table like this.
CREATE EXTERNAL TABLE textteacher(ID int, Name char(50), email varchar(255)) USING ( DATAOBJECT 'teacher.csv' FORMAT TEXT CCSID 1208 DELIMITER '|' REMOTESOURCE 'LOCAL' SOCKETBUFSIZE 30000 LOGDIR '/tmp/logs' );
and tried to upload data to it.
insert into textteacher (ID,Name,email) select id,name,email from teacher;
and get exception [428IB][-20569] The external table operation failed due to a problem with the corresponding data file or diagnostic files. File name: "teacher.csv". Reason code: "1".. SQLCODE=-20569, SQLSTATE=428IB, DRIVER=4.26.14
If I correct understand documentation parameter extbl_location should pointed directory where data will save. I suppose full directory will showed like
$extbl_location+'/'+teacher.csv
I found some documentation about error
https://www.ibm.com/support/pages/how-resolve-sql20569n-error-external-table-operation
I tried to run command in docker command line.
/opt/ibm/db2/V11.5/bin/db2 get db cfg | grep -i external
but does not information about external any tables.
CREATE EXTERNAL TABLE statement:
file-name
...
When both the REMOTESOURCE option is set to LOCAL (this is its default value) and the extbl_strict_io configuration parameter is set
to NO, the path to the external table file is an absolute path and
must be one of the paths specified by the extbl_location configuration
parameter. Otherwise, the path to the external table file is relative
to the path that is specified by the extbl_location configuration
parameter followed by the authorization ID of the table definer. For
example, if extbl_location is set to /home/xyz and the authorization
ID of the table definer is user1, the path to the external table file
is relative to /home/xyz/user1/.
So, If you use relative path to a file as teacher.csv, you must set extbl_strict_io to YES.
For an unload operation, the following conditions apply:
If the file exists, it is overwritten.
Required permissions:
If the external table is a named external table, the owner must have read and write permission for the directory of this file.
If the external table is transient, the authorization ID of the statement must have read and write permission for the directory of this file.
Moreover you must create a sub-directory equal to your username (in lowercase) which is owner of this table in the directory specified in extbl_location and ensure, that this user (not the instance owner) has rw permission to this sub-directory.
Update:
To setup presuming, that user1 runs this INSERT statement.
sudo mkdir -p /home/xyz/user1
# user1 must have an ability to cd to this directory
sudo chown user1:$(id -gn user1) /home/xyz/user1
db2 connect to mydb
db2 update db cfg using extbl_location /home/xyz extbl_strict_io YES

postgres: import csv file into table

I want to import the csv file into database table .but it was not working..
I run the bash shell in the linux env .
CREATE TABLE test.targetDB (
no int4 NOT NULL GENERATED ALWAYS AS IDENTITY,
year varchar(15) NOT NULL,
name bpchar(12) NOT NULL,
city varchar(15) NOT NULL,
ts_load timestamp NOT NULL DEFAULT CURRENT_TIMESTAMP,
PRIMARY KEY (seq_no)
)
test.csv file
"2019","112 ","1123",2019-07-26-05.33.43.000000
Linux Cmd
psql -d $database -c " COPY test.targetDB from 'test.csv' delimiter ',' csv "
Error
ERROR: invalid input syntax for integer: "2019"
CONTEXT: COPY targetDB, line 1, column no: "2019"
How can I resolve this issue
You need to tell copy, that the no column is not part of the CSV file by specifying the columns that should be populated:
COPY test.targetDB(year, name, city, ts_load) from 'test.csv' delimiter ',' csv
I would recommend, using datagrip - a postgresql client tool. You can use a evaluation version if you don't wish to purchase. It's pretty simple from the UI to import a file rather using a command line.

latest .bak file in windows folder using sql script

I need help to write the sql script to find the latest backup file from the windows folder to restore the database. filename is like:-
dbnm_2019_4_5_11_30_613.bak
dbnm_2019_4_18_11_32_234.bak
dbnm_2019_4_11_11_37_34.bak
... name is made up using dbnm_year_month_date_hr_min_sec format.
used below script:-
CREATE TABLE #File
( FileName SYSNAME,
Depth TINYINT,
IsFile TINYINT
);
INSERT INTO #File
(FileName, Depth, IsFile)
EXEC xp_DirTree '[file location]',1,1;
is there anyway that I can insert date filed from the network folder to show when the backup file created and do the order by on that field to find the latest file.
when I am using top 1 in select statement, it is showing me 2019_4_5_11_30_613.bak as latest file which is incorrect.
Is there anyway that I can insert date filed from the network folder
to show when the backup file created and do the order by on that field
to find the latest file.
SQL Server recovery databases from files only from computer unitys (c: d: ).
To get the backup sets I use this statment:
SELECT
database_name as DataBaseName,
physical_device_name as PhysicalDeviceName,
backup_start_date as BackupStartDate,
backup_finish_date as BackupFinishDate,
cast(backup_size/1024.0 as decimal(19,2)) AS BackupSizeKB,
cast(backup_size/1024.0/1024.0 as decimal(19,2)) AS BackupSizeMB,
cast(backup_size/1024.0/1024/1024.0 as decimal(19,2)) AS BackupSizeGB
FROM msdb.dbo.backupset b
JOIN msdb.dbo.backupmediafamily m ON b.media_set_id = m.media_set_id
where cast(b.backup_finish_date as date)= (cast(getdate() -1 as date))
ORDER BY backup_finish_date
Pay attention to the clauses: WHERE and ORDER BY.
There is another way to get from windows folders the last file using powershell.
Take a look into Keith Hill answer here: Finding modified date of a file/folder

PSQL Copy CSV data to postgres DB

I have an empty table in postgreSQL :
CREATE TABLE public.tbltesting
(
"ID" integer,
"testValue" numeric
)
WITH (
OIDS = FALSE
)
TABLESPACE pg_default;
I have a CSV file with the following data :
ID,testValue
1,2.0
2,3.33
3,4
The file is huge and requires a bulk copy and so I am trying to run the following command from PSQL :
\copy tblfoodnutrients FROM 'C:\temp\tbltestingbulk.csv' with CSV HEADER
ERROR: relation "tblfoodnutrients" does not exist
I have also tried the following :
\copy public.tblfoodnutrients FROM 'C:\temp\tbltestingbulk.csv' with CSV HEADER
ERROR: relation "public.tblfoodnutrients" does not exist
DUH! My connection was missing the database name to begin with.