I had a look to the IP2 Location database for SQL server 2019, BCP import or TSQL openrowset, Import Data wizard, all fail
Had no luck with the FMT file as it's the wrong version, no problem I guess I will generate it using
bcp [ip2location].[dbo].[IP2LOCATION-LITE-DB5] format nul -T -N -f D:\IP2LOCATION-LITE-DB5.CSV\DB5.fmt
The issue I have is an error:
Cannot bulk load CSV file. Invalid field parameters are specified for source column number 1 in the format file "D:\IP2LOCATION-LITE-DB5.CSV\DB5.FMT". All data fields must be either character or Unicode characters with terminator when CSV format is specified.
The SQL I use to test:
select top(10) *
from openrowset(BULK N'D:\IP2LOCATION-LITE-DB5.CSV\IP2LOCATION-LITE-DB5.CSV'
,FORMATFILE = N'D:\IP2LOCATION-LITE-DB5.CSV\DB5.FMT'
, FORMAT='CSV') AS DATA
I can't seem to be able to import IP2LOCATION-LITE-DB5.csv
Based on the FAQ page https://www.ip2location.com/faqs/db5-ip-country-region-city-latitude-longitude#database, you can create the table and import as below:
CREATE DATABASE ip2location
GO
USE ip2location
GO
CREATE TABLE [ip2location].[dbo].[ip2location_db5](
[ip_from] bigint NOT NULL,
[ip_to] bigint NOT NULL,
[country_code] nvarchar(2) NOT NULL,
[country_name] nvarchar(64) NOT NULL,
[region_name] nvarchar(128) NOT NULL,
[city_name] nvarchar(128) NOT NULL,
[latitude] float NOT NULL,
[longitude] float NOT NULL
) ON [PRIMARY]
GO
CREATE CLUSTERED INDEX [ip_to] ON [ip2location].[dbo].[ip2location_db5]([ip_to]) ON [PRIMARY]
GO
BULK INSERT [ip2location].[dbo].[ip2location_db5]
FROM 'D:\IP2LOCATION-LITE-DB5.CSV\IP2LOCATION-LITE-DB5.CSV'
WITH
(
FORMAT = 'CSV',
FIELDQUOTE = '"',
FIELDTERMINATOR = ',',
ROWTERMINATOR = '0x0D0A',
TABLOCK
)
GO
Related
I'm trying to import from CSV to Postgres and generate a uuid, uuid_generate_v4(), during the import to populate a table.
Postgres version: 14
Postgres Table
Country
id (primary key)
country_name
country_ISO_Code
Psql import
\copy "Country" (select uuid_generate_v4() as "id", "country_name", "country_ISO_code") FROM 'C:\Users\.......\data.csv' (format csv, header, delimiter ',')
However, that throws an error \copy: parse error at "as"
How do I properly instruct psql to use uuid_generate_v4() as id column?
You can't copy "from" a select statement. You need to define a default value for your primary key:
create table country
(
id uuid primary key default gen_random_uuid(),
country_name text not null,
country_iso_code text not null
);
Then tell the \copy command that your input file only contains two columns:
\copy country (country_name, country_iso_code) from 'data.csv' (format csv, header, delimiter ',')
SQL server db has a time column datatype of
[start_time] time NULL,
[end_time] time NULL,
but dataflows doesn't have a function for this.
The only way I can think of doing this is a post query (if you fully recreate the table)
alter table dbo.[testTable]
alter column [start_time] time(0)
alter column [end_time] time(0)
I tried using timestamp but again it's not a matching datatype
toTimestamp(substring(start_date,12,9),'HH:mm:ss')
so this doesn't work.
Any help on understanding this would be great
** updating with screens shots
So this issue is for parquet or csv to Sql db tables.
if you have a column that looks like DateTime you need to keep it as a string as there is no toDateTime function only toTimestamp. Neither string or Timestamp can be converted to DateTime datatype in SQLdb sink. You will end up with nulls in your column
Sample befor using expression to change the start_date to yyyy-mm-dd THH:mm:ss
You can simply map the DATETIME column to the target TIME column in the sink activity.
Make sure the option "Allow schema drift" in sink activity is unchecked.
My test schema:
-- source table
DROP TABLE IF EXISTS [dbo].[tempSourceTable]
CREATE TABLE [dbo].[tempSourceTable](
[id] int IDENTITY(1,1) NOT NULL,
[key] nvarchar(max) NULL,
[start_date] datetime NULL
)
INSERT INTO [dbo].[tempSourceTable] VALUES ('key1', '2021-10-14 12:34:56')
SELECT * FROM [dbo].[tempSourceTable]
-- target table
DROP TABLE IF EXISTS [dbo].[tempTargetTable]
CREATE TABLE [dbo].[tempTargetTable](
[id] int IDENTITY(1,1) NOT NULL,
[key] nvarchar(max) NULL,
[start_time] time NULL
)
result after execute the dataflow in a pipeline:
Here is my testing CSV input:
start_date,end_date,start_date_time,end_date_time,start_time,end_time
09/01/2020,09/01/2020,09/01/2020 11:01,09/01/2020 11:01,11:01:46,11:01:52
09/01/2020,,09/01/2020 11:01,,11:01:47,
09/01/2020,09/01/2020,09/01/2020 11:01,09/01/2020 11:50,11:01:49,11:50:41
09/01/2020,09/01/2020,09/01/2020 11:01,09/01/2020 11:01,11:01:51,11:01:55
09/01/2020,09/01/2020,09/01/2020 11:01,09/01/2020 11:01,11:01:52,11:01:56
You may specify the data/time/datetime format for CSV source data:
You can see the correct parsing result in data preview:
After that, a simple sink activity should achieve what OP wants to do:
The sink table schema I used for testing:
CREATE TABLE [dbo].[tempTargetTable](
[start_date] date NULL,
[end_date] date NULL,
[start_date_time] datetime NULL,
[end_date_time] datetime NULL,
[start_time] time NULL,
[end_time] time NULL
)
Result in DB:
i write some SQL commands in batch file (.bteq files), the bteq file is as follows:
.logon localhost:1025/dbc,dbc
SELECT 1 FROM dbc.tables WHERE databasename = 'customerservice' AND TABLENAME
= 'accounts';
.if activitycount <>0 then drop table customerservice.accounts;
CREATE SET TABLE CustomerService.accounts ,FALLBACK ,
NO BEFORE JOURNAL,
NO AFTER JOURNAL,
CHECKSUM = DEFAULT,
DEFAULT MERGEBLOCKRATIO
(
ACCOUNT_NUMBER INTEGER NOT NULL,
STREET_NUMBER INTEGER,
STREET CHAR(25) CHARACTER SET LATIN NOT CASESPECIFIC,
CITY CHAR(20) CHARACTER SET LATIN NOT CASESPECIFIC,
STATE CHAR(2) CHARACTER SET LATIN NOT CASESPECIFIC,
ZIP_CODE INTEGER,
BALANCE_FORWARD DECIMAL(10,2),
BALANCE_CURRENT DECIMAL(10,2))
UNIQUE PRIMARY INDEX ( ACCOUNT_NUMBER );
insert into customerservice.accounts
values(20031303,559,'LaSalleAve','SantaMonica','CA',94123,4550.00,4550.00);
insert into customerservice.accounts values(20033587,506,'SomersetWay',
'HermosaBeach','CA',90365,7770.00,7770.00);insert into
customerservice.accounts
values(20033118,900,'9thStreet','HermosaBeach','CA',90365,3080.00,3080.00);
insert into customerservice.accounts values
(20032649,924,'OregonTrail','HermosaBeach','CA',90365,8200.00,8200.00);
insert into customerservice.accounts values
(20030691,987,'ComptonPlace','SantaMonica','CA',94123,8240.00,8240.00);
then i run this bteq file like this:
bteq < test.bteq 2>error.txt
what i want to is redirect error to error.txt, but seems it do not work, maybe it is because 2 is just for standard error, but the SQL error is not standard system error ?
if so, then is there any ways we can redirect SQL error to a file ?
I have a database about weather that updates every second.
It contains temperature and wind speed.
This is my database:
CREATE TABLE `new_table`.`test` (
`id` INT(10) NOT NULL,
`date` DATETIME() NOT NULL,
`temperature` VARCHAR(25) NOT NULL,
`wind_speed` INT(10) NOT NULL,
`humidity` FLOAT NOT NULL,
PRIMARY KEY (`id`))
ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8
COLLATE = utf8_bin;
I need to find the average temperature every hour.
This is my code:
Select SELECT AVG( temperature ), date
FROM new_table
GROUP BY HOUR ( date )
My coding is working but the problem is that I want to move the value and date of the average to another table.
This is the table:
CREATE TABLE `new_table.`table1` (
`idsea_state` INT(10) NOT NULL,
`dateavg` DATETIME() NOT NULL,
`avg_temperature` VARCHAR(25) NOT NULL,
PRIMARY KEY (`idsea_state`))
ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8
COLLATE = utf8_bin;
Is it possible? Can you give me the coding?
In order to insert new rows into a database based on data you have obtained from another table, you can do this by setting up an INSERT query targeting the destination table, then run a sub-query which will pull the data from the source table and then the result set returned from the sub-query will be used to provide the VALUES used for the INSERT command
Here is the basic structure, note that the VALUES keyword is not used:
INSERT INTO `table1`
(`dateavg`, `avg_temperature`)
SELECT `date` , avg(`temperature`)
FROM `test`;
Its also important to note that the position of the columns returned by result set will be sequentially matched to its respective position in the INSERT fields of the outer query
e.g. if you had a query
INSERT INTO table1 (`foo`, `bar`, `baz`)
SELECT (`a`, `y`, `g`) FROM table2
a would be inserted into foo
y would go into bar
g would go into baz
due to their respective positions
I have made a working demo - http://www.sqlfiddle.com/#!9/ff740/4
I made the below changes to simplify the example and just demonstrate the concept involved.
Here is the DDL changes I made to your original code
CREATE TABLE `test` (
`id` INT(10) NOT NULL AUTO_INCREMENT,
`date` DATETIME NOT NULL,
`temperature` FLOAT NOT NULL,
`wind_speed` INT(10),
`humidity` FLOAT ,
PRIMARY KEY (`id`))
ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8
COLLATE = utf8_bin;
CREATE TABLE `table1` (
`idsea_state` INT(10) NOT NULL AUTO_INCREMENT,
`dateavg` VARCHAR(55),
`avg_temperature` VARCHAR(25),
PRIMARY KEY (`idsea_state`))
ENGINE = InnoDB
DEFAULT CHARACTER SET = utf8
COLLATE = utf8_bin;
INSERT INTO `test`
(`date`, `temperature`) VALUES
('2013-05-03', 7.5),
('2013-06-12', 17.5),
('2013-10-12', 37.5);
INSERT INTO `table1`
(`dateavg`, `avg_temperature`)
SELECT `date` , avg(`temperature`)
FROM `test`;
I have data something like this:
Akhoond,1,Akhoond,"{""Akhund"", ""Akhwan""}",0
pgAdmin's import is rejecting this. What format does the text[] need to be in the CSV?
I also tried this:
Akhoond,1,Akhoond,"{Akhund, Akhwan}",0
Here's the table create:
CREATE TABLE private."Titles"
(
"Abbrev" text NOT NULL,
"LangID" smallint NOT NULL REFERENCES private."Languages" ("LangID"),
"Full" text NOT NULL,
"Alt" text[],
"Affix" bit
)
WITH (
OIDS=FALSE
);
ALTER TABLE private."Titles" ADD PRIMARY KEY ("Abbrev", "LangID");
CREATE INDEX ix_titles_alt ON private."Titles" USING GIN ("Alt");
ALTER TABLE private."Titles"
OWNER TO postgres;
The best way to find out is to create a table with the desired values and COPY ... TO STDOUT to see:
craig=> CREATE TABLE copyarray(a text, b integer, c text[], d integer);
CREATE TABLE
craig=> insert into copyarray(a,b,c,d) values ('Akhoond',1,ARRAY['Akhund','Akhwan'],0);
INSERT 0 1
craig=> insert into copyarray(a,b,c,d) values ('Akhoond',1,ARRAY['blah with spaces','blah,with,commas''and"quotes'],0);
INSERT 0 1
craig=> \copy copyarray TO stdout WITH (FORMAT CSV)
Akhoond,1,"{Akhund,Akhwan}",0
Akhoond,1,"{""blah with spaces"",""blah,with,commas'and\""quotes""}",0
So it looks like "{Akhund,Akhwan}" is fine. Note the second example I added showing how to handle commas, quotes spaces in the array text.
This works with the psql \copy command; if it doesn't work with PgAdmin-III then I'd suggest using psql and \copy.