How to create records concurrently in Postgres with ActiveRecord and Sidekiq - postgresql

I have a Sidekiq worker, which makes an API call, parses the returned json and creates ActiveRecord objects (products). Since the products belong to a brand and the product json also contains the data for the brand, the worker does the following before actually saving the product to the database:
check if the brand exists in the database by checking its unique api_id (the id, with which the brand comes from the api, in the db, there is a unique index on this column);
if it does - fetch its primary id;
if is doesn't - create it and get its primary id
I have implemented this like so:
def brand_id(brand_json)
Brand.where(api_id: brand_json[:api_id]).pluck(:id).first.presence ||
Brand.create!(name: brand_json[:name], api_id: brand_json[:api_id]).id
end
After that the worker creates the product with the brand_id set to the fetched id.
Now I am thinking of the following scenario:
two workers simultaneously fetch data for two products that belong to the same brand that doesn't yes exist in the database;
worker 1 one checks for the brand and doesn't find it;
shortly after that worker 2 checks for the brand and doesn't find it;
worker 1 creates the brand;
Now what happens with worker 2? My assumption - it tries to create the brand, but an error at the database level occurs, as there is already a record with the same api_id? (probably ActiveRecord::RecordNotUnique error is raised?)
Also, how do I handle this case and this type of errors in the context of Sidekiq and ActiveRecord? Should I somehow implement a table-wide lock on the brands table to prevent such things? If yes - than I will not be able to concurrently create products, as at any given time only one worker will have access to the brands table, which is required for creating a product.
Or maybe I should wrap my brand_id(brand_json) method in transaction like so:
def brand_id(brand_json)
ActiveRecord::Base.transaction do
Brand.where(api_id: brand_json[:api_id]).pluck(:id).first.presence ||
Brand.create!(name: brand_json[:name], api_id: brand_json[:api_id]).id
end
end
Please, advise.

Put the unique index constraints (possibly in the form of a multi column index) in the DB.
Just try to create the object. The database will prevent you from making more than one.
Only the thread that succeeded in creating the initial object (no exception occurred) is allowed to proceed with extra processing.

Related

Nest TypeORM Postgres update user's column('number of posts') based on the userId in the Posts Table

I'm wondering if it's possible to auto update the User's column('number of posts') if the Posts table updates. The Post entity has a ManyToOne relation with User('userId'). Is there a way to make the User Table "listen" to the Post Table and automatically updates the number of post column, or i need to write it in the post service create function to do so. I'm new to sql so i'm just trying new stuff. I'm using NestJS,typeORM, Postgres and Graphql
#Kendle's answer does work and has the advantage of pushing the computation and complexity down onto your DB server. Alternatively, you can keep that logic in the application by leveraging TypeORM's Subscribers functionality. Documentation can be found here.
In your specific use case, you could register a subscriber for your Post entity implementing afterInsert and afterRemove (or afterSoftRemove if you soft delete posts) to increment and the decrement the counter respectively.
You don't want to duplicate that data. That's the whole idea of a relational database that different data is kept in different tables.
You can create a view if you want to avoid typing a query with a JOIN each time.
For example you might create the view below:
CREATE VIEW userPosts AS
SELECT
user.id,
user.name,
COUNT(posts.id)
FROM users
LEFT JOIN posts ON user.id = posts.user_id
ORDER BY user.id;
Once you have created the view your can query it as if it were a table.
SELECT * FROM userDate WHERE id = '0001';
Of course I don't have your table definitions and data so you will need to adapt this code to your tables.

Entity Framework - Last ID of Entree

Currently I am programming an application which saves and deletes entries in a database (code first with entity framework). My question is how can I get the next database id (configured with auto increment - so it added automatically +1).
I tried something like:
var a = databaseContext.MyObject.LastOrDefault().Id;
var myNextDatabaseId = a+1;
This pseudo code is working for most of the cases. But if i had 5 entrees in my database and delete all the five entree's my next database-counter would be 6. When i'm using the code above it will return "null" becouse there is really no entree. But i must get the next database auto increment id.
Is there a possibility which doesn't create a new database entry? Think this shouldn't be necessary.
For example following data construct:
Inserted entry one (internal id = 1)
Inserted entry two (internal id = 2)
Delete entry one
Delete entry two
Read last database entry (entry = null). All data was deleted but I am trying to receive the next auto-increment id 3
Use an explicit transaction scope and identity column. Trying to maintain IDs at the application level isn't scale-able and will be prone to concurrency errors.
Ideally the transaction should span the web service call to roll back automatically if the call fails. With an explicit transaction you can call your context.SaveChanges() which will expose the next ID assigned to the entity, and then roll back if the web service call fails. This can be a transaction managed by the context or TransactionScope.
see: https://msdn.microsoft.com/en-us/library/dn456843(v=vs.113).aspx

Spring data and MongoDB - bidirectional connection of documents

I have two Documents in my Spring data - MongoDB application:
The first one is Contact and looks like this:
public class Contact {
...
private List<Account> accounts;
and the second one is Account and looks like this:
public class Account {
...
private Contact contact;
My question now is, whether there is a better way of:
1. create contact object
2. save contact object into database
3. create account object
4. set contact object into account object
5. save account object into database
6. set created account object into contact object
7. update contact object
These are many steps and I will avoid to do such a long list to get Contact and Account connected bidirectional.
Try this approach
MongoDB is a NOSQL DB and hence there is no need of an order to be preserved, such as create and store contact object and then do so more in a sequential way.
Maintain a sequence for Contact and Account object. Before storing these two records get the next number in the sequence and insert the Contact and Account documents.
References for autoincrement sequence
https://docs.mongodb.com/v3.0/tutorial/create-an-auto-incrementing-field/
https://www.tutorialspoint.com/mongodb/mongodb_autoincrement_sequence.htm
Pseudo Code:
Get the next Sequence of Contact and Account Id
Add the id's to respective documents
Insert the Documents in Mongodb
While retrieving the records you can use $lookup which is a left outer join.
Please note that chance of loss of integrity in data can happen if one insert is happened successfully and other insert did not happen for some reason.
We dont have transaction support in Mongodb across collections, more info.

Autogenerated ids create gaps in sequence after em.remove() followed by an em.merge or em.persist

I was creating an EJB application which as a CRUD feature for Hotel Rooms.
I have 4 rows in my Room database.
Now if i remove the 4th row (having room id 104) by using the em.remove() method and then add a new row the table looks like this
A gap is created in the autogenerated sequence Ids. Is there any way to avoid these gaps while adding new rows ?
By default JPA don't have this feature, also this issue is more related how to database manage the sequences and as far as I know sequence number are not reused.
Persist operation will create a new id, when the entity does not have one, if you try to persist a detached object probably you will find a EntityExistsException.
• If the Object is a removed entity instance, an IllegalArgumentException will
be thrown by the merge operation (or the transaction commit will
fail).

Entity Framework SET IDENTITY_INSERT

Is there a way to force the ID value for a new entity in EF when we have an auto-incrementing ID column, i.e. use SET IDENTITY_INSERT behaviour through EF?
Our requirement is that our create form must always show a new, unique ID for the object we're creating on the empty form before it is filled in or saved. The idea is that this ID can be out read to someone over the phone and then the user can complete and save the form after the call is complete. We could reserve an ID by inserting an empty row into the database there and then, but we have unique columns and FKs; instead I've made a 'next ID' table that we increment with locks for safety, and I test this against the top ID in the object table too to be careful. The idea was to then force the use of this new ID when we write back the entity - but I can't see how to get EF to do it.
Is that possible - is it just something I've missed? I don't think the ID even makes it down to the insert so I don't think manually calling SET IDENTITY_INSERT around the SaveChanges would help.
Or do I have to do something else? I can see alternatives:
Change our ID column to not be an identity and take manual control of it all: there's a table ID inheritance here so this is potentially tricky too.
Separate DB ID and user-visible ID into a separate column, and record our unique ID there.
Empty row to reserve the ID, as above; might need some nullability changes, and amending our data read code to ignore these records.
Thanks! This is EF4 (using an EDMX and generated classes not POCOs), and against SQL Server 2008 in case that matters.
Why not use a Guid as primary key. Nothing to do with auto-increment, no concurrency pitfalls etc. You just create the Guid at the moment you create the form. Hand it over to a caller and fill in the form afterwards. When the form is cancelled, no problem. When the form is finished create the entity with the created Guid set the other values of the entity object, apply it to the (a) context and SaveChanges()...
Alternatives that wont alter your schema
Use EF Transaction
You can call context.SaveChanges() and get the autoincremented primary key. Once the process is completed you can commit the transaction. If the transaction is cancelled or there is an error/exception, you can always rollback so you wont have holes/dirty-data in your rows. I suggest you use the singleton pattern and pass the same transaction/context to whatever methods or screens to complete the process.
Just add an additional status: Draft
Save empty form as draft with saved ID, then proceed to edit the form with the information. Once complete save the form as final/ready. If you wont proceed to save the form, you can always recycle the draft.