Its possibile give input at XMLTable xml file store in my desktop pc?
How ?
select id, name, T.*
INSERT INTO abc(name)
SELECT x.name
from XMLTABLE('$i/product' PASSING CAST(? AS XML) as "i"
COLUMNS
name VARCHAR(10) PATH 'name',
) as x;
How pass my file store on the desktop in my pc?
thank you
SQL statements can't access files outside of the database manager. To load data in an XML file and write it to a table, you'd either need to use a database utility (like LOAD or IMPORT), or write your own program to read the data from files on the client machine (i.e., your PC) and perform the inserts.
Related
How to Use Copy data activity to check against sink values
My Data Sources:
SourceDataset : Source_SQL_DB
DestinationDataset : Destination_SQL_DB
SourceTable : SourceTableName
Column : Name,Age,Gender,Location
DestinationTable : DestinationTableName
Column : Name,Age,Gender,Location
Below is my scenario :
I have to validate Source before moving to sinkTable by checking Destination should not have the values
On Copy data, i can directly load data,
How to pass the Location in Source Query since my source will be connecting to source dataset only
select * from SourceTableName where Location in (select distinct Location from DestinationTableName)
How to check is the name present in the destination dataset table, If name is present, i should not insert data.
select * from SourceTableName where name not in (select distinct name from DestinationTableName )
assuming both your source and sink are sql, you can use a lookup activity to get the list of names and location as comma seperated and either save them in a variable or use it to directly in source query.
Another way would be to load the source dara as is in staging table and then leveraging a stored procedure activity.
The final way would be to use dataflows
I have an Azure function that retrieves a zip file that contains multiple BCP files which unzips them and adds them as blobs.
I now want to import the BCP files into my SQL database but not sure how to go about it. I know I can use following script and run an SqlCommand:
BULK INSERT RegPlusExtract.dbo.extract_class
FROM 'D:\local\data\extract_class.bsp'
WITH ( FIELDTERMINATOR = '#**#',ROWTERMINATOR = '*##*')
But this obviously does not work as the SQL server doesn't have access to the local function's D: drive.
How should I go about loading the data? Is it possible to load the BCP file into memory and then pass the SQLCommand? Or can I pass the file direct to SQL server?
I've found out that for backup/restore I can do FROM URL = ''. If I could use this for bulk insert then I can just reference the blob url, but doesn't look like I can?
You will need to use BLOB storage..below are the steps and these are documented here Microsoft/sql-server-samples
--create an external data source
CREATE EXTERNAL DATA SOURCE MyAzureBlobStorage
WITH ( TYPE = BLOB_STORAGE,
LOCATION = 'https://sqlchoice.blob.core.windows.net/sqlchoice/samples/load-from-azure-blob-storage',
-- CREDENTIAL= MyAzureBlobStorageCredential --> CREDENTIAL is not required if a blob storage is public!
);
You also can upload files to a container and reference it like below.Here week3 is a container
CREATE EXTERNAL DATA SOURCE MyAzureInvoicesContainer
WITH (
TYPE = BLOB_STORAGE,
LOCATION = 'https://newinvoices.blob.core.windows.net/week3',
CREDENTIAL = UploadInvoices
);
Now you can use OpenRowset and BulkInsert like below
-- 2.1. INSERT CSV file into Product table
BULK INSERT Product
FROM 'product.csv'
WITH ( DATA_SOURCE = 'MyAzureBlobStorage',
FORMAT='CSV', CODEPAGE = 65001, --UTF-8 encoding
FIRSTROW=2,
TABLOCK);
-- 2.2. INSERT file exported using bcp.exe into Product table
BULK INSERT Product
FROM 'product.bcp'
WITH ( DATA_SOURCE = 'MyAzureBlobStorage',
FORMATFILE='product.fmt',
FORMATFILE_DATA_SOURCE = 'MyAzureBlobStorage',
TABLOCK);
-- 2.3. Read rows from product.dat file using format file and insert it into Product table
INSERT INTO Product WITH (TABLOCK) (Name, Color, Price, Size, Quantity, Data, Tags)
SELECT Name, Color, Price, Size, Quantity, Data, Tags
FROM OPENROWSET(BULK 'product.bcp',
DATA_SOURCE = 'MyAzureBlobStorage',
FORMATFILE='product.fmt',
FORMATFILE_DATA_SOURCE = 'MyAzureBlobStorage') as products;
-- 2.4. Query remote file
SELECT Color, count(*)
FROM OPENROWSET(BULK 'product.bcp',
DATA_SOURCE = 'MyAzureBlobStorage',
FORMATFILE='data/product.fmt',
FORMATFILE_DATA_SOURCE = 'MyAzureBlobStorage') as data
GROUP BY Color;
I already know how to unload a file from redshift into s3 as one file. I need to know how to unload with the column headers. Can anyone please help or give me a clue?
I don't want to manually have to do it in shell or python.
As of cluster version 1.0.3945, Redshift now supports unloading data to S3 with header rows in each file i.e.
UNLOAD('select column1, column2 from mytable;')
TO 's3://bucket/prefix/'
IAM_ROLE '<role arn>'
HEADER;
Note: you can't use the HEADER option in conjunction with FIXEDWIDTH.
https://docs.aws.amazon.com/redshift/latest/dg/r_UNLOAD.html
If any of your columns are non-character, then you need to explicitly cast them as char or varchar because the UNION forces a cast.
Here is an example of the full statement that will create a file in S3 with the headers in the first row.
The output file will be a single CSV file with quotes.
This example assumes numeric values in column_1. You will need to adjust the ORDER BY clause to a numeric column to ensure the header row is in row 1 of the S3 file.
******************************************
/* Redshift export to S3 CSV single file with headers - limit 6.2GB */
UNLOAD ('
SELECT \'column_1\',\'column_2\'
UNION
SELECT
CAST(column_1 AS varchar(255)) AS column_1,
CAST(column_2 AS varchar(255)) AS column_2
FROM source_table_for_export_to_s3
ORDER BY 1 DESC
;
')
TO 's3://bucket/path/file_name_for_table_export_in_s3_' credentials
'aws_access_key_id=<key_with_no_<>_brackets>;aws_secret_access_key=<secret_access_key_with_no_<>_brackets>'
PARALLEL OFF
ESCAPE
ADDQUOTES
DELIMITER ','
ALLOWOVERWRITE
GZIP
;
****************************************
There is no direct option provided by redshift unload .
But we can tweak queries to generate files with rows having headers added.
First we will try with parallel off option so that it will create only on file.
"By default, UNLOAD writes data in parallel to multiple files, according to the number of slices in the cluster. The default option is ON or TRUE. If PARALLEL is OFF or FALSE, UNLOAD writes to one or more data files serially, sorted absolutely according to the ORDER BY clause, if one is used. The maximum size for a data file is 6.2 GB. So, for example, if you unload 13.4 GB of data, UNLOAD creates the following three files."
To have headers in unload files we will do as below.
Suppose you have table as below
create table mutable
(
name varchar(64) default NULL,
address varchar(512) default NULL
)
Then try to use select command from you unload as below to add headers as well
( select 'name','address') union ( select name,address from mytable )
this will add headers name and address as first line in your output.
Just to complement the answer, to ensure the header row comes first, you don't have to order by a specific column of data. You can enclose the UNIONed selects inside another select, add a ordinal column to them and then in the outer select order by that column without including it in the list of selected columns.
UNLOAD ('
SELECT column_1, column_2 FROM (
SELECT 1 AS i,\'column_1\' AS column_, \'column_2\' AS column_2
UNION ALL
SELECT 2 AS i, column_1::varchar(255), column_2::varchar(255)
FROM source_table_for_export_to_s3
) t ORDER BY i
')
TO 's3://bucket/path/file_name_for_table_export_in_s3_'
CREDENTIALS
'aws_access_key_id=...;aws_secret_access_key=...'
DELIMITER ','
PARALLEL OFF
ESCAPE
ADDQUOTES;
Redshift now supports unload with headers. September 19–October 10, 2018 release.
The syntax for unloading with headers is -
UNLOAD ('select-statement')
TO 's3://object-path/name-prefix'
authorization
HEADER
Unfortunately, the UNLOAD command doesn't natively support this feature (see other answers for how to do it with workarounds).
I've posted a feature request on the AWS forums, so hopefully it gets added someday.
Edit: The feature has now been implemented natively in Redshift! 🎉
Try like this:
Unload VENUE with a Header:
unload ('select * from venue where venueseats > 75000')
to 's3://mybucket/unload/'
iam_role 'arn:aws:iam::0123456789012:role/MyRedshiftRole'
header
parallel off;
The following shows the contents of the output file with a header row:
venueid|venuename|venuecity|venuestate|venueseats
6|New York Giants Stadium|East Rutherford|NJ|80242
78|INVESCO Field|Denver|CO|76125
83|FedExField|Landover|MD|91704
79|Arrowhead Stadium|Kansas City|MO|79451
To make the process easier you can use a pre-built docker image to extract and include the header row.
https://github.com/openbridge/ob_redshift_unload
It will also do a few other things, but it seemed to make sense to package this in an easy to use format.
To unload a table as csv to s3 including the headers, you will simply have to do it this way
UNLOAD ('SELECT * FROM {schema}.{table}')
TO 's3://{s3_bucket}/{s3_key}/{table}/'
with credentials
'aws_access_key_id={access_key};aws_secret_access_key={secret_key}'
CSV HEADER ALLOWOVERWRITE PARALLEL OFF;
I am using PostgreSQL 9.0.3. I have an Excel spreadsheet with lots of data to load into couple of tables in Windows OS.
I have written the script to get the data from input file and Insert into some 15 tables. This can't be done with COPY or Import. I named the input file as DATALD.
I find out the psql command -d to point the db and -f for the script sql. But I need to know the commands how to feed the input file along with the script so that the data gets inserted into the tables..
For example this is what I have done:
begin
for emp in (select distinct w_name from DATALD where w_name <> 'w_name')
--insert in a loop
INSERT INTO tblemployer( id_employer, employer_name,date_created, created_by)
VALUES (employer_id,emp.w_name,now(),'SYSTEM1');
Can someone please help?
For an SQL script you must ..
either have the data inlined in your script (in the same file).
or you need to utilize COPY to import the data into Postgres.
I suppose you use a temporary staging table, since the format doesn't seem to fit the target tables. Code example:
How to bulk insert only new rows in PostreSQL
There are other options like pg_read_file(). But:
Use of these functions is restricted to superusers.
Intended for special purposes.
I'm using SQL Query Analyzer to build a report from the database on one machine (A), and I'd like to create a temp table on a database server on another machine(B) and load it with the data from machine A.
To be more specific, I have a report that runs on machine A (machine.a.com), pulling from schema tst. Using SQL Query Analyzer, I log into the server at machine.a.com and then have access to the tst schema:
USE tst;
SELECT *
FROM prospect;
I would like to create a temp table from this query window, only I'd like it built on another machine (call it machine.b.com). What syntax would I use for this? My guess is something like:
CREATE TABLE machine.b.com.#temp_prospect_list(name varchar(45) Not Null, id decimal(10) Not Null);
And then I'd like to load this new table with something like:
INSERT INTO machine.b.com.#temp_prospect_list VALUES (
USE tst;
SELECT *
FROM prospect; );
The syntax to access a remote server in T-SQL is to fully qualify any table name with the following (brackets included when necessary):
[LinkedServer].[RemoteDatabase].[User].[Table]
So, for example, to run a SELECT statement on one server that accesses a table on another server:
SELECT * FROM [machine.b.com].tst.dbo.table7;