What would be the best solution for building a number sequence generator. I've to apply several different text formats to a sequence of numbers. I would like to save the current number values in a own table. When I need to draw a new number, I would like to increase the number in my table and save the new number to my table. But I have to make it independent from other (bigger) transactions, which may be running at the same time.
For better understanding my issue: I've to import a batch of sales orders. During the import I've to generate a new number sequence value, but it will happen quite in the middle of a transaction. After processing some header information I can now generate my number from the sequence. At this point I've to save this in the database, so that the next user has to draw a new number. After the number generation I've to process the order lines. Only if this is finished successful, then I'm allow to save the whole sales order in database. If errors occur, the whole sales order will be retried to be imported manually later. The used number of the sequence must not be used again.
So I need a possibility to save single records from the ChangeTracker instead of writing all modifications at once. Any idea how to deal with it?
Related
I have three JSON files with the same timestamp but different values to upload to the Druid. I want to upload them separately with the same segment granularity. However, it drops the existing segment and uploads the new one.
I don't want to use appendToExisting: True bc it sums the values of the same rows. This is the situation that I don't want to happen (I may be adding the same file in the future).
Is there a way to add new data to a specific segment without dropping or summing it?
Use case
I need to store texts assigned to some entity. It's important to note that I always only care about the most current texts that have been assigned to that entity. In case new texts are inserted, older ones might even be deleted. And that "might" is the problem, because I can't rely that really only the most current texts are available.
The only thing I'm unsure about how to design is the case that some INSERT can provide either 1 or N texts for some entity. In the latter case, I need to know which N texts belong to the most current INSERT done for one and the same entity. Additionally, inserting N instead of 1 text will be pretty rare.
I know that things could be implemented using two different tables: One calculating some main-ID and the other mapping individual texts with their own IDs to that main-ID. Because multiple texts should happen rarely and a one table design already provides columns which could easily be reused for grouping multiple texts together, I prefer using one only.
Additionally, I thought of which concept would make a good grouping key in general as well. I somewhat doubt that others really always implement the two table-approach only and therefore created this question to get a better understanding. Of course I simply might be wrong and everybody avoids such "hacks" at all costs. :-)
Possible keys
Transaction-local timestamp
Postgres supports the concept of a transaction-local timestamp using current_timestamp. I need to have one of those to store when the texts have been stored anyway, so they might be used for grouping as well?
While there's in theory the probability of collisions, timestamps have a resolution of 1 microsecond, which is in practice enough for my needs. Texts are uploaded by human users and it is very unlikely that multiple humans upload texts for the same entity at the same time at all.
That timestamp won't be used as a primary key of course, only to group multiple texts if necessary.
Transaction-ID
Postgres supports txid_current to get the ID of the current transaction, which should be ever increasing over the lifetime of the current installation. The good thing is that this value is always available and the app doesn't need to do anything on it's own. But things can easily break in case of restores, can't they? Can TXIDs e.g. occur again with the restored cluster?
People knowing things better than me write the following:
Do not use the transaction ID for anything at the application level. It is an internal system level field. Whatever you are trying to do, it's likely that the transaction ID is not the right way to do it.
https://stackoverflow.com/a/32644144/2055163
You shouldn't be using the transaction ID for anything except an identifier for a transaction. You can't even assume that a lower transaction ID is an older transaction, due to transaction ID wrap-around.
https://stackoverflow.com/a/20602796/2055163
Isn't my grouping a valid use case for wanting to know the ID of the current transaction?
Custom sequence
Grouping simply needs a unique key per transaction, which can be achieved using a custom sequence for that purpose only. I don't see any downsides, its values consume less storage than e.g. UUIDs and can easily be queried.
Reusing first unique ID
The table to store the texts contains a serial-column, so each inserted text gets an individual ID already. Therefore, the ID of the first inserted text could simply always be additionally reused as the group-key for all later added texts.
In case of only inserting one text, one should easily be able to use currval and doesn't even need to explicitly query the ID of the inserted row. In case of multiple texts this doesn't work anymore, though, because currval would provide updated IDs instead of the first one per transaction only. So some special handling would be necessary.
APP-generated random UUID
Each request to store multiple texts could simply generate some UUID and group by that. The mainly used database Postgres even supports a corresponding data type.
I mainly see too downsides with this: It feels really hacky already and consumes more space than necessary. OTOH, compared to the texts to store, the latter might simply be negligible.
I'm working on a tool to fetch about 3 years of historic data from a site, in order to perform some data analysis & machine learning.
The dimensions of the report I am requesting are:
[ ga:cityId, ga:dateHour, ga:userType, ga:deviceCategory ]
And my starting point is to import to a postgres db (the data may live elsewhere eventually but we have Good Reasons for starting with a relational database).
I've defined a unique index on the [ ga:cityId, ga:dateHour, ga:userType, ga:deviceCategory ] tuple for the postgres table, and my import job currently routinely fails every 30000-50000 rows due to a duplicate of that tuple.
What would cause google to return duplicate rows?
I'm batching the inserts by 1000 rows / statement because row-at-a-time would be very time consuming, so I think my best workaround is to disable the unique index for the duration of the initial import, de-dupe, and then re-enable it and do daily imports of fresh data row-at-a-time. Other strategies?
There shouldn't be duplicate reports coming back from google if the time ranges are unique.
Are you using absolute or relative (to present) dates? If the latter, you should ensure that changes in the time period cause by the progression of the relative time (ie the present) don't cause an overlap.
Using relative time period could also cause gaps in your data.
I'm building a database to track usage/generate reports of particular products on a daily basis. There is a limited number of products (about 20) where they're removed from inventory, used in production, and then remaining product is returned to inventory but not put back into the system. The idea is to have production record how much they receive and then have storage record how much they get back from production.
The tables are pretty straightforward - I have one table with product properties that I completely control for the foreseeable future and another that will house usage data. The issue I'm having is how to design the form for operations for when they receive product. I don't want to have a huge list of product #'s with entries for each one (usually 5 to 10 are used on a daily basis). I also don't want to populate the data table with a bunch of blank records. I want them to add a line, select the product code from a drop down, record the amount received and then repeat. Preferably the drop down would update to exclude any previously filled in codes on the form. I want to do this all at once to limit duplicate records. So they fill out all the product #'s they received and how many of each they received and then click save to have it populate the data table.
Is there a way to have an "add line" option for a form of this design? I know this isn't a terribly extensive database but I want to design and test it prior to integration into our plant's larger scale product tracking system.
I want to keep track of an individual's credit card transactions(the amount, in essence). Instead of having a new entry made in the database for each transaction, is there any way I can save all the transactions of one person on a single row?
i.e. , if a person A makes purchases of Rs.1500, Rs. 2600 and Rs. 3200 at different instances, I want the table entry to look something like this:
A : 1500, 2600, 3200
Also, is there any way I can keep only certain number of entries? Meaning, for one new entry added, one oldest entry should get deleted. I'm using Oracle 10g.
Please help me out.
Thank you.
Can you? Well, you certainly could define the column as a VARCHAR2 or a CLOB and write a comma-separated list of values to that column. That would mean that you would then have to write code to parse the data every time you selected it. And write code to do things like removing or modifying one element in the list. And you'd lose the benefits of proper data typing. And of proper normalization. And you'd really, really annoy whoever has to support your code in the future. Particularly when someone inadvertently stores a transaction value of 1,000 rather than 1000 in your comma-separated string column.
So you can, yes. But I cannot envision any situation where it would really make sense to do so.
Create a table to store transactions. Create a new row for every transaction. Link each transaction to a specific person. You'll be grateful you did.