generating nil id having id sequence defined, Why? - postgresql

I have a table named async_data, which has id column also auto increment defined. But in production I am seeing some insert queries are failing saying
PG::NotNullViolation: ERROR: null value in column "id" violates not-null constraint
Rails Migration file
class CreateAsyncData < ActiveRecord::Migration[5.0]
def change
create_table :async_data do |t|
t.integer :request_id
t.integer :sourced_contact_id
t.integer :data_source_id
t.boolean :is_enriched
t.column :requested_params, :json
t.text :q
t.datetime :fetched_at
t.timestamps
end
end
end
CREATE TABLE public.async_data (
id integer NOT NULL,
request_id integer,
sourced_contact_id integer,
data_source_id integer,
is_enriched boolean DEFAULT false,
requested_params json,
fetched_at timestamp without time zone,
created_at timestamp without time zone NOT NULL,
updated_at timestamp without time zone NOT NULL,
q text,
is_processed boolean DEFAULT false NOT NULL,
is_data_pushed boolean DEFAULT false NOT NULL
);
\d async_data;
Table "public.async_data"
Column | Type | Collation | Nullable | Default
-------------------+-----------------------------+-----------+----------+----------------------------------------------------
id | integer | | not null | nextval('async_data_id_seq'::regclass)
request_id | integer | | |
source_company_id | integer | | |
data_source_id | integer | | |
is_enriched | boolean | | |
requested_params | json | | |
q | text | | |
fetched_at | timestamp without time zone | | |
created_at | timestamp without time zone | | not null |
updated_at | timestamp without time zone | | not null |
Indexes:
"async_data_pkey" PRIMARY KEY, btree (id)
--
-- Name: async_data_id_seq; Type: SEQUENCE; Schema: public; Owner: -
--
CREATE SEQUENCE public.async_data_id_seq
AS integer
START WITH 1
INCREMENT BY 1
NO MINVALUE
NO MAXVALUE
CACHE 1;
I want to re-produce the same in dev environment and want to know why id nil was generated.

Related

unique constraint values in postgres

I applied over a postgres USER table and unique contraint over email. The problem that I am facing now is that the constraint seems to register each value I insert (or try to insert) no matter if a record with that value exists or not.
I.e
Table:
id
user
1
mail#gmail.com
2
mail2#gmail.com
if i insert mail3#gmail.com, delete the value and try to insert mail3#gmail.com again it says:
sqlalchemy.exc.IntegrityError: (psycopg2.errors.UniqueViolation) duplicate key value violates unique constraint "email"
my doubt is: the unique constraint guarantees that the value is newer always or that there is only one record of that value in the column?
documentations says it the second but the experience shows is the first one
more details:
| Column | Type | Nullable |
|------------------|-----------------------------|----------|
| id | integer | not null |
| email | character varying(100) | |
| password | character varying(100) | |
| name | character varying(1000) | |
| lastname | character varying(1000) | |
| dni | character varying(20) | |
| cellphone | character varying(20) | |
| accepted_terms | boolean | |
| investor_test | boolean | |
| validated_email | boolean | |
| validated_cel | boolean | |
| last_login_at | timestamp without time zone | |
| current_login_at | timestamp without time zone | |
| last_login_ip | character varying(100) | |
| current_login_ip | character varying(100) | |
| login_count | integer | |
| active | boolean | |
| fs_uniquifier | character varying(255) | not null |
| confirmed_at | timestamp without time zone | |
Indexes:
"bondusers_pkey" PRIMARY KEY, btree (id)
"bondusers_email_key" UNIQUE CONSTRAINT, btree (email)
"bondusers_fs_uniquifier_key" UNIQUE CONSTRAINT, btree (fs_uniquifier)
Insert Statement:
INSERT INTO bondusers (email, password, name, lastname, dni, cellphone, accepted_terms, investor_test, validated_email, validated_cel, last_login_at, current_login_at, last_login_ip, current_login_ip, login_count, active, fs_uniquifier, confirmed_at) VALUES ('mail3#gmail.com', '$pbkdf2-sha256$29000$XyvlfI8x5vwfYwyhtBYi5A$Hhfrzvqs94MjTCmDOVmmnbUyf7ho4kLEY8UYUCdHPgM', 'mail', 'mail3', '123123123', '1139199196', false, false, false, false, NULL, NULL, NULL, NULL, NULL, true, '1c4e60b34a5641f4b560f8fd1d45872c', NULL);
ERROR: duplicate key value violates unique constraint "bondusers_fs_uniquifier_key"
DETAIL: Key (fs_uniquifier)=(1c4e60b34a5641f4b560f8fd1d45872c) already exists.
but when:
select * from bondusers where fs_uniquifier = '1c4e60b34a5641f4b560f8fd1d45872c';
result is 0 rows
I assume that if you run the INSERT, DELETE, INSERT directly within Postgres command line it works OK?
I noticed your error references SQLAlchemy (sqlalchemy.exc.IntegrityError), so I think it may be that and not PostgreSQL. Within a transaction SQLAlchemy's Unit of Work pattern can re-order SQL statements for performance reasons.
The only ref I could find was here https://github.com/sqlalchemy/sqlalchemy/issues/5735#issuecomment-735939061 :
if there are no dependency cycles between the target tables, the flush proceeds as follows:
<snip/>
a. within a particular table, INSERT operations are processed in the order in which objects were add()'ed
b. within a particular table, UPDATE and DELETE operations are processed in primary key order
So if you have the following within a single transaction:
INSERT x
DELETE x
INSERT x
when you commit it, it's probably getting reordered as:
INSERT x
INSERT x
DELETE x
I have more experience with this problem in Java/hibernate. The SQLAlchemy docs do claim it's unit of work pattern is "Modeled after Fowler's "Unit of Work" pattern as well as Hibernate, Java's leading object-relational mapper." so probably relevant here too
To supplement Ed Brook's insightful answer, you can work around the problem by flushing the session after deleting the record:
with Session() as s, s.begin():
u = s.scalars(sa.select(User).where(User.user == 'a')).first()
s.delete(u)
s.flush()
s.add(User(user='a'))
Another solution would be to use a deferred constraint, so that the state of the index is not evaluated until the end of the transaction:
class User(Base):
...
__table_args__ = (
sa.UniqueConstraint('user', deferrable=True, initially='deferred'),
)
but note, from the PostgreSQL documentation:
deferrable constraints cannot be used as conflict arbitrators in an INSERT statement that includes an ON CONFLICT DO UPDATE clause.

Select rows where column matches any IP address in inet[] array

I'm trying to display rows that have at least one value in the inet[] type column.
I really don't know any better, so it seems it would be easiest to use something like this, but it returns results with {} which I guess is null according to the inet[] type, but not from the perspective of the is not null query?
peering_manager=# select asn,name,potential_internet_exchange_peering_sessions from peering_autonomoussystem where potential_internet_exchange_peering_sessions is not null order by potential_internet_exchange_peering_sessions limit 1;
asn | name | potential_internet_exchange_peering_sessions
------+---------------------------------+----------------------------------------------
6128 | Cablevision Systems Corporation | {}
(1 row)
peering_manager=#
So trying to dig a little more into it, I think maybe if I can try to match the existence of any valid IP address in the inet[] column, that would work, however I'm getting an error and I don't understand what it's referring to or how to resolve it to achieve the desired results:
peering_manager=# select asn,name,potential_internet_exchange_peering_sessions from peering_autonomoussystem where potential_internet_exchange_peering_sessions << inet '0.0.0.0/0';
ERROR: operator does not exist: inet[] << inet
LINE 1: ...here potential_internet_exchange_peering_sessions << inet '0...
^
HINT: No operator matches the given name and argument type(s). You might need to add explicit type casts.
peering_manager=#
Maybe it's saying that the << operator is invalid for the inet[] type or that the << operator is an invalid operation when trying to query an inet type from a value stored as an inet[] type? Or something else?
In any event, I'm kind of lost. Maybe there's a better way to do this?
Here's the table, and a sample of the data set I'm working with.
peering_manager=# \d peering_autonomoussystem;
Table "public.peering_autonomoussystem"
Column | Type | Modifiers
----------------------------------------------+--------------------------+-----------------------------------------------------------------------
id | integer | not null default nextval('peering_autonomoussystem_id_seq'::regclass)
asn | bigint | not null
name | character varying(128) | not null
comment | text | not null
ipv6_max_prefixes | integer | not null
ipv4_max_prefixes | integer | not null
updated | timestamp with time zone |
irr_as_set | character varying(255) |
ipv4_max_prefixes_peeringdb_sync | boolean | not null
ipv6_max_prefixes_peeringdb_sync | boolean | not null
irr_as_set_peeringdb_sync | boolean | not null
created | timestamp with time zone |
potential_internet_exchange_peering_sessions | inet[] | not null
contact_email | character varying(254) | not null
contact_name | character varying(50) | not null
contact_phone | character varying(20) | not null
Indexes:
"peering_autonomoussystem_pkey" PRIMARY KEY, btree (id)
"peering_autonomoussystem_asn_ec0373c4_uniq" UNIQUE CONSTRAINT, btree (asn)
Check constraints:
"peering_autonomoussystem_ipv4_max_prefixes_check" CHECK (ipv4_max_prefixes >= 0)
"peering_autonomoussystem_ipv6_max_prefixes_check" CHECK (ipv6_max_prefixes >= 0)
Referenced by:
TABLE "peering_directpeeringsession" CONSTRAINT "peering_directpeerin_autonomous_system_id_691dbc97_fk_peering_a" FOREIGN KEY (autonomous_system_id) REFERENCES peering_autonomoussystem(id) DEFERRABLE INITIALLY DEFERRED
TABLE "peering_internetexchangepeeringsession" CONSTRAINT "peering_peeringsessi_autonomous_system_id_9ffc404f_fk_peering_a" FOREIGN KEY (autonomous_system_id) REFERENCES peering_autonomoussystem(id) DEFERRABLE INITIALLY DEFERRED
peering_manager=#
peering_manager=# select asn,name,potential_internet_exchange_peering_sessions from peering_autonomoussystem limit 7;
asn | name | potential_internet_exchange_peering_sessions
-------+---------------------------------+-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
37662 | WIOCC | {2001:504:1::a503:7662:1,198.32.160.70}
38001 | NewMedia Express Pte Ltd | {2001:504:16::9471,206.81.81.204}
46562 | Total Server Solutions | {2001:504:1::a504:6562:1,198.32.160.12,2001:504:16::b5e2,206.81.81.81,2001:504:1a::35:21,206.108.35.21,2001:504:2d::18:80,198.179.18.80,2001:504:36::b5e2:0:1,206.82.104.156}
55081 | 24Shells Inc | {2001:504:1::a505:5081:1,198.32.160.135}
62887 | Whitesky Communications | {2001:504:16::f5a7,206.81.81.209}
2603 | NORDUnet | {2001:504:1::a500:2603:1,198.32.160.21}
6128 | Cablevision Systems Corporation | {}
(7 rows)
You can use array_length(). On empty arrays or nulls this returns NULL.
...
WHERE array_length(potential_internet_exchange_peering_sessions, 1) IS NOT NULL
...
better to compare array length with integer number
...
WHERE array_length(potential_internet_exchange_peering_sessions, 1) > 0
...

Cannot Save Grafana Datasource - not-null Postgres Error

I no longer seem to be able to save new Datasources in Grafana.
In particular I am trying to add new InfluxDB database as a datasource. When hitting the Add button it pop up an error of Problem! Failed to add datasource in the UI.
The Grafana logs show the following:
t=2018-07-17T09:59:32+0000 lvl=eror msg="Failed to add datasource" logger=context userId=0 orgId=1 uname= error="pq: null value in column \"id\" violates not-null constraint"
Checking the Database logs (PostgreSQL) there is a related error:
2018-07-19 07:12:46 UTC:10.204.145.134(36768):admin#grafana:[477]:DETAIL: Failing row contains (null, 1, 0, influxdb, jenkins, proxy, http://localhost:8086, root, root, jenkins, f, , , f, {}, 2018-07-19 07:12:46, 2018-07-19 07:12:46, f, {}).
2018-07-19 07:12:46 UTC:10.204.145.134(36768):admin#grafana:[477]:STATEMENT: INSERT INTO "data_source"
As you can see the UI seems to be trying to insert null as the index which produces the error.
Although we recently migrated databases (from one PG to another, same version) the config did not change and there don't appear to have been any other ill effects.
EDIT: Seems this actually affects any database operation Grafana tries to perform when it comes to adding new resources. I just had a dev try to import a new dashboard and the PostgreSQL logs show a similar error:
2018-07-19 08:05:07 UTC:10.204.25.220(34412):sharedadmin#grafana:[14263]:DETAIL: Failing row contains (null, 1, pcs-again, PCS-AGAIN, {"__requires":[{"id":"grafana","name":"Grafana","type":"grafana"..., 1, 2018-07-19 08:05:07, 2018-07-19 08:05:07, -1, -1, 0, ).
After much delving we managed to find the answer. The issues lies within the AWS Database Migration Service (DMS) we used to migrate from one RDS instance to another. It would seems that DMS does not handle PostgreSQL to PostgreSQL well, some caveats can be found in the docs here.
In the case of Grafana the streaming replication did not pick up the column modifiers. One of the migrated tables:
grafana-> \d data_source
Table "public.data_source"
Column | Type | Modifiers
---------------------+--------------------------------+-----------
id | integer | not null
org_id | bigint | not null
version | integer | not null
type | character varying(255) | not null
name | character varying(190) | not null
access | character varying(255) | not null
url | character varying(255) | not null
password | character varying(255) |
user | character varying(255) |
database | character varying(255) |
basic_auth | boolean | not null
basic_auth_user | character varying(255) |
basic_auth_password | character varying(255) |
is_default | boolean | not null
json_data | text |
created | timestamp(6) without time zone | not null
updated | timestamp(6) without time zone | not null
with_credentials | boolean | not null
secure_json_data | text |
Indexes:
"data_source_pkey" PRIMARY KEY, btree (id)
and the corresponding table from a non-migrated instance:
grafana=> \d data_source
Table "public.data_source"
Column | Type | Modifiers
---------------------+-----------------------------+-----------------------------------------------------------
id | integer | not null default nextval('data_source_id_seq1'::regclass)
org_id | bigint | not null
version | integer | not null
type | character varying(255) | not null
name | character varying(190) | not null
access | character varying(255) | not null
url | character varying(255) | not null
password | character varying(255) |
user | character varying(255) |
database | character varying(255) |
basic_auth | boolean | not null
basic_auth_user | character varying(255) |
basic_auth_password | character varying(255) |
is_default | boolean | not null
json_data | text |
created | timestamp without time zone | not null
updated | timestamp without time zone | not null
with_credentials | boolean | not null default false
secure_json_data | text |
Indexes:
"data_source_pkey1" PRIMARY KEY, btree (id)
"UQE_data_source_org_id_name" UNIQUE, btree (org_id, name)
"IDX_data_source_org_id" btree (org_id)
The moral of the story is that DMS is not suitable to all databases, read the documentation thoroughly and in some cases using the native PostgreSQL tools is better.
In order to fix this particular issue, we dropped the database (after making sure we had exports of all the dashboards), re-created it then restarted Grafana.
Please check you PSQL Tables data_source and user_auth.
grafana=# \d data_source
Table "public.data_source"
Column | Type | Collation | Nullable | Default
---------------------+-----------------------------+-----------+----------+-----------------------------------------
id | integer | | not null | nextval('data_source_id_seq'::regclass) <=== (Probably its ok Constraint)
grafana=# \d user_auth
Table "public.user_auth"
Column | Type | Collation | Nullable | Default
----------------------+-----------------------------+-----------+----------+---------------------------------------
id | bigint | | not null | <=== (Probably it is blank)
Logon as grafana user in grafana PSQL DB and run:
Alter table user_auth:
alter table user_auth
alter column id TYPE integer,
alter column id SET not null;
CREATE SEQUENCE user_auth_id_seq
START WITH 1
MINVALUE 1
NO MAXVALUE
CACHE 1;
CMD:
grafana=# CREATE SEQUENCE user_auth_id_seq
grafana-# START WITH 1
grafana-# MINVALUE 1
grafana-# NO MAXVALUE
grafana-# CACHE 1;
CREATE SEQUENCE
grafana=# ALTER TABLE user_auth ALTER COLUMN id SET DEFAULT nextval('user_auth_id_seq');
ALTER TABLE
grafana=# \d user_auth
Table "public.user_auth"
Column | Type | Collation | Nullable | Default
----------------------+-----------------------------+-----------+----------+---------------------------------------
id | integer | | not null | nextval('user_auth_id_seq'::regclass) <=== (check it is ok now)
Test nexval:
grafana=# SELECT nextval('user_auth_id_seq');
nextval
---------
1
(1 row)
grafana=# SELECT nextval('user_auth_id_seq');
nextval
---------
2
(1 row)
grafana=# SELECT nextval('user_auth_id_seq');
nextval
---------
3
(1 row)
Try to connect with ldap.
Regards

cannot update row value for integer column in postgresql table

This is the dummy function I wrote to update the counter.
def updateTable(tableName, visitorId, dtWithZone):
db_uri = app.config["SQLALCHEMY_DATABASE_URI"]
engine = create_engine(db_uri, connect_args={"options": "-c timezone={}".format(dtWithZone.timetz().tzinfo.zone)})
# create session
Session = sessionmaker()
Session.configure(bind=engine)
session = Session()
meta = MetaData(engine, reflect=True)
table = meta.tables[tableName]
print dir(table)
# update row to database
row = session.query(table).filter(
table.c.visitorId == visitorId).first()
print 'original:', row.count
row.count = row.count + 1
print "updated {}".format(row.count)
session.commit()
conn.close()
but when it reaches the line row.count = row.count + 1 it throws error:
AttributeError: can't set attribute
this is the table
\d visitorinfo;
Table "public.visitorinfo"
Column | Type | Modifiers
--------------+--------------------------+-----------
platform | character varying(15) |
browser | character varying(10) |
visitorId | character varying(10) | not null
language | character varying(10) |
version | character varying(20) |
cl_lat | double precision |
cl_lng | double precision |
count | integer |
ip | character varying(20) |
visitor_time | timestamp with time zone |
Indexes:
"visitorinfo_pkey" PRIMARY KEY, btree ("visitorId")
what am I doing wrong ?why is it saying cannot set attribute?
part of updated code:
# update row to database
row = session.query(table).filter(
table.c.visitorId == visitorId).first()
print 'original:', row.count
val = row.count
row.count = val + 1
print "updated {}".format(row.count)
use an update query:
UPDATE public.visitorinfo SET counter = counter + 1 WHERE visitorId = 'VisitorID'
Make sure that last 'VisitorID' is filled from your application

Sequelize many-to-many self reference

I'm trying to get one of the association examples from sequelize working properly and it doesn't seem to be setting up the join table correctly. In the example we have one model called Person and then a many-to-many self-reference for a Person's children. Code:
var Sequelize = require('sequelize');
var sequelize = new Sequelize('postgres://root#localhost/database_bug');
var Person = sequelize.define('Person', {
id: {
type: Sequelize.INTEGER,
primaryKey: true,
autoIncrement: true
},
name: {
type: Sequelize.STRING,
allowNull: false
}
});
Person.belongsToMany(Person, { as: 'Children', through: 'PersonChildren' });
Person.sequelize.sync({force:true}).then(function() {
Person.build({ name: 'John Doe' }).save().then(function(father) {
Person.build({ name: 'Jane Doe' }).save().then(function(daughter) {
father.addChild(daughter);
});
});
});
But when I look at my tables in postgres I feel like a column is missing on the auto-generated join table.
List of relations
Schema | Name | Type | Owner
--------+----------------+----------+-------
public | People | table | root
public | People_id_seq | sequence | root
public | PersonChildren | table | root
Table "public.People"
Column | Type | Modifiers
-----------+--------------------------+-------------------------------------------------------
id | integer | not null default nextval('"People_id_seq"'::regclass)
name | character varying(255) | not null
createdAt | timestamp with time zone | not null
updatedAt | timestamp with time zone | not null
Indexes:
"People_pkey" PRIMARY KEY, btree (id)
Referenced by:
TABLE ""PersonChildren"" CONSTRAINT "PersonChildren_PersonId_fkey" FOREIGN KEY ("PersonId") REFERENCES "People"(id)
Table "public.PersonChildren"
Column | Type | Modifiers
-----------+--------------------------+-----------
createdAt | timestamp with time zone | not null
updatedAt | timestamp with time zone | not null
PersonId | integer | not null
Indexes:
"PersonChildren_pkey" PRIMARY KEY, btree ("PersonId")
Foreign-key constraints:
"PersonChildren_PersonId_fkey" FOREIGN KEY ("PersonId") REFERENCES "People"(id)
PersonChildren needs a column called ChildId or something along those lines to link a Person to its Child.
People table:
database_bug=# SELECT * FROM "People";
id | name | createdAt | updatedAt
----+----------+----------------------------+----------------------------
1 | John Doe | 2015-02-06 09:36:44.975-08 | 2015-02-06 09:36:44.975-08
2 | Jane Doe | 2015-02-06 09:36:44.985-08 | 2015-02-06 09:36:44.985-08
Weirder still, I select to make sure daughter was added as a child to father:
database_bug=# SELECT * from "PersonChildren";
createdAt | updatedAt | PersonId
----------------------------+----------------------------+----------
2015-02-06 09:36:44.997-08 | 2015-02-06 09:36:44.997-08 | 2
But PersonId is 2, not 1. The father was supposed to add the daughter, not the other way around.
Any ideas how to get this association working?
Ok, looks like the example in the documentation is wrong. To be fair they did say must use hasMany but then showed an example using belongsToMany.
I changed belongsToMany to hasMany and it looks like we are good to go:
Table "public.PersonChildren"
Column | Type | Modifiers
-----------+--------------------------+-----------
createdAt | timestamp with time zone | not null
updatedAt | timestamp with time zone | not null
PersonId | integer | not null
ChildId | integer | not null
database_bug=# select * from "PersonChildren";
createdAt | updatedAt | PersonId | ChildId
----------------------------+----------------------------+----------+---------
2015-02-06 10:04:21.624-08 | 2015-02-06 10:04:21.624-08 | 1 | 2
Now I can do father.getChildren() and the promise will return a list of children.