I am running into the following which is kind of odd
OS X 10.11 Orient 2.1.11
I am grabbing data from a memory only graph (thread a) to feed graph g (thread b)
The logic in thread b has worked reliably for long time. It can handle super nodes of several million vertices.
The problem I am running into now is: some commits changes are not reflected on disk, let alone in subsequents read. All the reads and writes to g are executed by thread b.
The commits succeed.
I am embedding orient with with following
new OrientGraphFactory("plocal:" + f.getAbsolutePath()).setupPool(1, 2));
then setting autoStartTx to false.
If I let the logic run, it is the last commit that is missing. If I step through code it can be several commits that are missing.
Any suggestions?
Thanks a lot!
Related
I would like to write a unit test which ensures that a race condition can't happen if Thread A wants to write a row out and Thread B wants to delete it. (I already know that at the database level, the race condition will be resolved, but the way my application is written currently, if such a race condition happens, it will get settled by one of the threads dying and needing to get restarted... it's something the application can recover from, but it's a situation that is good to avoid.)
Inconveniently, Thread A polls something that doesn't change once it reaches a final state, and then when it sees that the final state has been reached, Thread A writes stuff out in the database to signal to Thread B to do new stuff. Thread B meanwhile looks at the row, does things, then deletes it when it's done. Since Thread A enables Thread B to do work on something which doesn't change in between Thread A being done with it and Thread B starting to deal with it, the only way I can have this race condition which I'm testing for is actually just writing the same data out again. That is, I can't just make an assertion that the data in the table is the same as it was before and say that's good enough - what I want to do is to assert that no updates on a given row of the table were committed at all by Thread A. (If I were to add a column like "last written at this time" that would be silly because that would be adding a column just to make my unit tests more convenient to write, not for the benefit of the application, so that's also not an option.)
I am using SQLAlchemy + postgres and my test is in a pytest suite. What's a nice way to write code which will fail the test if the code the test is exercising writes out a row to the database?
One thing I am thinking of is adding a postgres-level constraint just for the duration of the test but I think this feels really messy to me, and if I could figure out a way to do it by inspecting the SQLAlchemy Engine that was in use by Thread A, or something like that, that would be way better. But it is not obvious to me how to tell this from the SQLAlchemy docs - maybe it is obvious to someone else? :)
I don't have a code example at this time because I don't know where to get started.
I had results of 2 CFD simulations, which I visualized using Paraview. Let me name the 2 results as case A and case B. In principle I had to analyze same parameters for both the cases. Hence I named the states of case A and case B with the same name, but had saved them in entirely different folders. There were about 4 states for each of the case (with the same name). I was able to load the states without any problems until yesterday. Today when I tried to load the very same state that I was able to load yesterday, Paraview crashed. What could be the reason for this?
I thought the problem occurred as I had used the same names for the states of both the cases. Hence I even tried load them after renaming. Still I couldn't load. I also reinstalled Paraview from scratch. Yet it crashed when I tried to load the state. The version I am using is 5.7.0
I am running some streaming query jobs on a databricks cluster, and when i look at the cluster/job logs, I see a lot of
first at Snapshot.scala:1
and
withNewExecutionId at TransactionalWriteEdge.scala:130
A quick search yielded this scala script https://github.com/delta-io/delta/blob/master/src/main/scala/org/apache/spark/sql/delta/Snapshot.scala
Any one can explain what this do in laymans term?
Internally this class manages the replay of actions stored in checkpoint or delta file
Generally, this "snapshotting" relies on delta encoding and indirectly allows snaphot isolation as well.
Practically delta-encoding remembers every side-effectful operation like INSERT DELETE UPDATE that you did since the last checkpoint. In case of delta lake it would be SingleAction (source): AddFile (insert) RemoveFile (delete). Conceptually this approach is close to event-sourcing - without it you'd have to literally store/broadcast whole state (database or directory) on every update. It also employed by many classic ACID databases with replication.
Overall it gives you:
ability to continuously replicate file-system/directory/database state (see SnapshotManagement.update). Basically that's why you see a lot of first at Snapshot.scala:1 - it's called in order to catch up with the log every time you start transaction, see DeltaLog.startTransaction. I couldn't find TransactionalWriteEdge sources, but I guess it's called around the same time.
ability to restore state by replaying every action since the last snapshot.
ability to isolate (and store) transactions by keeping their snapshots apart until commit (every SingleAction has txn in order to isolate). Delta-lake uses optimistic locking for that: transaction commits will fail if their logs are not mergeable, while readers don't see uncommitted actions.
P.S. You can see that the log is accessed in line val deltaData = load(files) and actions are stacked on top of previousSnapshot (val checkpointData = previousSnapshot.getOrElse(emptyActions); val allActions = checkpointData.union(deltaData))
I have a python app ran on heroku that utilizes a standard postgresql heroku db ($50 version). There are 4 tables within the db. My app queries for one primary key within the main table based off input from the users of my app.
The querying worked great at first however now I'm finding it becoming too slow after about 40-50 minutes without restart my dyno. The queries will take 2,000ms after a while and take several seconds to load in front of the users. I'm newer to programming and this is my second app. I'm wondering what would make queries slower with time instead of constant. They are so fast at first. What are best practices for psycopg2 within an app to ensure the db doesn't get hung up? Here is an example of one of the queries (all others have similar syntax throughout the script):
if eventText=="Mc3 my champs":
user=event.source.user_id
profile= line_bot_api.get_profile(user)
name=str((profile.display_name))
cur=None
try:
cur = conn.cursor(cursor_factory=psycopg2.extras.DictCursor)
# get the user's information if it exists
cur.execute("""SELECT lineid, summoner_name, champ_data FROM prestige_data WHERE lineid = %(lineid)s LIMIT 1""", {"lineid": user})
rows = cur.fetchall()
for row in rows:
champs = row[2]
prestige=(calculate_prestige(champs))
champs = json.loads(champs)
champsdict=dict.items(champs)
champs_sorted=sorted(champsdict, key=lambda student: student[1], reverse=True)
l=('\n'.join(map(str,champs_sorted)))
hello=str(l).replace('(', '').replace(')', '')
yay=str(hello).replace("'", "").replace("'", "")
msg=(yay+'\n'+"---------------------------"+'\n'+name+'\n'+"Prestige:"+(str(prestige)))
line_bot_api.reply_message(
event.reply_token,
TextSendMessage(text=msg))
break # we should only have one result, but we'll stop just in case
# The user does not exist in the database already
else:
msg = "Oops! You need to add some champs first. Try 'Mc3 inputchamp'."
line_bot_api.reply_message(
event.reply_token,
TextSendMessage(text=msg))
except BaseException:
if cur is not None:
conn.rollback()
finally:
if cur is not None:
cur.close()
While I know I did not frame that question well (only been programming for a month), I have found one issue potentially causing this that warrants documentation on here.
I had a suspicion the concurrency issue was caused when incorrect data was not found in the queries. In this situation, my conn would not rollback, commit, nor close in the above example.
Per psycopg2s documentation, even select queries need to be committed or rolled back or the transaction will stand. This in turn will keep your heroku dyno worker focused on the transaction for 30 seconds causing h12. So make sure you commit or rollback each query, regardless of the outcome, with in an application to ensure you do not get an idle transaction.
My queries are spiffy now but the issue persists. I'm not sure what slowly but surely idles my waitress workers. I think some ancillary process is somehow started in one of the class modules I've created which goes indefinitely taking hold of each worker until they are all focused on the transaction which leads to h12.
Would love someones input if they've had a similar experience. I don't want to have a cron job reboot the app every 10 minutes to make this self functioning.
The below code i have is working as intended, but is there a better way to do it?
I am consuming a db like a queue and process in batches of a max number. I'm thinking on how i can refactor it to use page.hasNext() and page.nextPageable()
However I can't find any good tutorial/documentation on what happens if the DB is manipulated between getting a page and getting the next page.
List<Customer> toBeProcessedList = customerToBeProcessedRepo
.findFirstXAsCustomer(new PageRequest(0, MAX_NR_TO_PROCESS));
while (!toBeProcessedList.isEmpty()) {
//do something with each customer and
//remove customer, and it's duplicates from the customersToBeProcessed
toBeProcessedList = customerToBeProcessedRepo
.findFirstXAsCustomer(new PageRequest(0, MAX_NR_TO_PROCESS));
}
If you use the paging support for each page requested a new sql statement gets executed, and if you don't do something fancy (and probably stupid) they get executed in different transactions. This can lead to getting elements multiple times or not seeing them at all, when the user moves from page to page.
Example: Page size 3; Elements to start: A, B, C, D, E, F
User opens the first page and sees
A, B, C (total number of pages is 2)
element X gets inserted after B; User moves to the next page and sees
C, D, E (total number of pages is now 3)
if instead of adding X, C gets deleted, the page 2 will show
E, F
since D moves to the first page.
In theory one could have a long running transaction with read stability (if supported by the underlying database) so one gets consistent pages, BUT this opens up questions like:
When does this transaction end, so the user gets to see new/changed data
When does this transaction end, when the user moves away?
This approach would have some rather high resource costs, while the actual benefit is not at all clear
So in 99 of 100 cases the default approach is pretty reasonable.
Footnote: I kind of assumed relational databases, but other stores should behave in basically the same way.