In our server we have a table of almost 2 GB, but problem is this table has no unique / primary key and no TimeStamp either. I am going to migration but without a unique value or / primary key or timestamp script is not working .
SQL Server 2008 R2 Enterprise
Related
Short description of a problem
I need to build a partitioned table "users" with 2 partitions located on separate servers (Moscow and Hamburg), each partition is table with columns:
id - integer primary key with auto increment
region - smallint partition key, which equals either 100 for Hamburg or 200 for Moscow
login - unique character varying with length of 100.
I intended to make sequences for id as n*1000 + 100 for Hamburg, and n*1000 + 200 for Moscow, so just looking on primary key I will know which partition it belongs to.
region is intended to be read only and never change after creation, so no records will move between partitions.
SELECT queries must be able to return records from all partitions and UPDATE queries must be able to modify records on all partitions, INSERT/DELETE queries must be able to add/delete records only to local partition, so data stored in them is not completely isolated.
What was done
Using pgAdmin4
I created a "test" table on Hamburg server, added all column info, marked it as partitioned table with partition key region and partition type List.
I created a "hamburg" partition in this table, adding primary key constraint as id,region and unique key constraint as login,region.
I created a "moscow" table on Moscow server with the same column info as "test"
I added postgres_fdw extension to Hamburg server, created Foreign server pointing to DB on Moscow server and User mapping.
I added "moscow" foreign table to Hamburg server pointing to "moscow" table on Moscow server.
What is my problem
I couldn't figure out how to attach this foreign table as second partition to "test" table.
When I tried to attach partition through pgAdmin dialog in "test" table partitions properties it shows me an error: cannot unpack non-iterable Response object
When I tried to add partition with query as follows:
ALTER TABLE public.test ATTACH PARTITION public.moscow FOR VALUES IN (200);
It shows me an error:
ERROR: cannot attach foreign table "moscow" as partition of partitioned table "test"
DETAIL: Table "test" contains unique indexes.
SQL state: 42809
I removed unique constraint from login column but it shows the same error.
When I make partitioned table with the same properties and both partitions initially located on the same server all works well, except for postgres watch for login uniqueness per-partition rather than in whole table, but I suggest this is its limitation.
So, how can I attach a table located on the second server as partition to partitioned table located on the first one?
The error message is pretty clear: Since you cannot create an index on a partitioned table, PostgreSQL cannot create a partition of the unique index. But the unique index is required to implement the constraint.
See this source comment:
/*
* If we're attaching a foreign table, we must fail if any of the indexes
* is a constraint index; otherwise, there's nothing to do here. Do this
* before starting work, to avoid wasting the effort of building a few
* non-unique indexes before coming across a unique one.
*/
Either drop the unique constraint or don't use foreign tables as partitions.
Ok, I was finally able to add a foreign table as partition to partitioned table.
It was necessary to drop primary key property on id and unique property on login columns for partitioned table
After that I was able to attach foreign table as partition to partitioned table
Later I have added primary key property on id and unique property on login columns for each local partition.
So in the end I have unique global id as it is generated by sequences for each DB with never intersected values. For login uniqueness I have to manually check global table if there is any record with it before inserting.
P.S. Hopefully, this partitioning mechanism in postgres is suitable for geographically distant regions.
I got a table of the following form in Postgres:
CREATE TABLE contract (
id serial NOT NULL,
start_date date NOT NULL,
end_date date NOT NULL,
price float8 NOT NULL,
CONSTRAINT contract_pkey PRIMARY KEY (id)
);
In Microsoft Powerapps, I create a EditForm to update the table above. For other databases, like MS SQL, I didn't need to supply the id, since it's auto increment. But for some reason, PowerApps keeps demanding to fill in the id for this table, even though it's auto increment and shouldn't be supplied to Postgres.
Anyone with the same experience with Powerapps in combination with Postgres? Struggling with it for hours...
I am trying the partitioning and FDW example explained in https://pgdash.io/blog/postgres-11-sharding.html. After creating the FDW extension and foreign partition table (temperatures_2016), when I try to insert 2016 data into the master table temperatures it throws
ERROR: cannot route inserted tuples to a foreign table.
How do I resolve this ?
-- on other server box2
CREATE TABLE temperatures_2016 (
at date,
city text,
mintemp integer,
maxtemp integer
);
--on my server
CREATE FOREIGN TABLE temperatures_2016
PARTITION OF temperatures
FOR VALUES FROM ('2016-01-01') TO ('2017-01-01')
SERVER box2;
INSERT INTO temperatures (at, city, mintemp, maxtemp)
VALUES ('2016-08-23', 'HongKong', 29, 40);
I expect the row to be inserted in the foreign partition table instead I get
ERROR: cannot route inserted tuples to a foreign table SQL state:
0A000
You probably have PostgreSQL version 10.
Inserting into a table with partitions that are foreign tables has been introduced in v11:
Allow INSERT, UPDATE, and COPY on partitioned tables to properly route rows to foreign partitions (Etsuro Fujita, Amit Langote)
This is supported by postgres_fdw foreign tables.
At any rate, that error message does not appear in the v11 code base.
Perhaps you are connecting to a v10 server with a v11 client?
I have a set of files on S3 that I am trying to load into redshift.
I am using the amazon data pipeline to do it. the wizard took the cluster, db and file format info but I get errors that a primary key is needed to keep existing fields in th table (KEEP_EXISTING) on the table
My table schema is:
create table public.Bens_Analytics_IP_To_FileName(
Day date not null encode delta32k,
IP varchar(30) not null encode text255,
FileName varchar(300) not null encode text32k,
Count integer not null)
distkey(Day)
sortkey(Day,IP);
so then I added a composite primary key on the table to see if it will work, but I get the same error.
create table public.Bens_Analytics_IP_To_FileName(
Day date not null encode delta32k,
IP varchar(30) not null encode text255,
FileName varchar(300) not null encode text32k,
Count integer not null,
primary key(Day,IP,FileName))
distkey(Day)
sortkey(Day,IP);
So I decided to add an identity column as the last column and made it the primary key but then the COPY operation wants a value in the input files for that identity column which did not make much sense
ideally I want it to work without a primary key or a composite primary key
any ideas?
Thanks
Documentation is not in a great condition. They have added a 'mergeKey' concept that can be any arbitrary key (announcement, docs). You should not have to define a primary key on table with this.
But you would still need to supply a key to perform join between your new data coming in and the existing data in redshift table.
In Edit Pipeline, under Parameters, there is a field named: myPrimaryKeys (optional). Enter you Pk there, instead of adding it to your table definition.
I have a database on azure where clustered indices are required. I would like to use Entity-Framework 6 Alpha 2, because I would like to use the new async features. When I test it on my local machine with SQL Express 2012 everything is fine, but when I try it with my azure database I get the following error:
Tables without a clustered index are not supported in this version of SQL Server. Please create a clustered index and try again.
I have no idea what to do, because when I test it with an empty database every primary key is a clustered index.
Any ideas?
Would you add a bit clarification on your situation with "Existing Database on Windows Azure" and using with EF 6? First of all - are you using EF CodeFirst, ModelFirst, DatabaseFirst?
Then if you really have existing database, how did you create it? DB + Schema manually, using some wizard (SSMS, SQL Azure Migration Wizard, EF CodeFirst created it, etc?). How this existing DB ended being in Azure?
Then trace down the full error message, check for which table it happens and manually add the clustered index on that table. It is true that every primary key you create in SQL Server, by default is also a CLUSTERED. But if the table was created first, then the Primary Key was added as separate DDL statement (ALTER TABLE ....) it might not have been created as CLUSTERED.
So, the message is pretty clear - please create clustered index first. Find out which table is SQL Azure complaining about and create a clustered index on it. If it hasn't Primary Key, just add one as CLUSTERD:
ALTER TABLE [dbo].[Individual]
ADD CONSTRAINT [PK_Individual_CustomerID]
PRIMARY KEY CLUSTERED
(
[CustomerID] ASC
)
If it has a primary key - check which columns are included, drop it, and recreate it as clustered.