I would like to force overwrite logdata to a CSV file. It might well be that another user is reading that file at the moment.
What give me the possibility take no care about this kind of locking and write in the file? Of course, the appended data should displayed after writing when the user close and open file again.
Maybe a useful information:
Writing data would occur many times for a very short term.
Related
I'm working on a MATLAB program in which I read a text file using textscan and then store the data in corresponding arrays. It does that every run, and it takes about two minutes each time. I would like to know if there is a way to save the arrays after I loaded the data, and have the program remember them so I can read the data only once and save time while running. I looked into the load function, but I'm not sure if that's what I need.
I have 2 independent Matlab workers, with FIRST getting/saving data and SECOND reading it (and doing some calculations etc).
FIRST saves data as .mat file on the hard-disk while SECOND reads it from there. It takes ~20 seconds to SAVE this data as .mat and 8millisec to DELETE it. Before SAVING data, FIRST deletes the old file and then saves a newer version.
How can the SECOND verify that data exists and is not corrupt? I can use exists but that doesn't tell me if the data is corrupt or not. For eg, if SECOND tries to read data exactly when FIRST is saving it, exists passes but LOAD gives you an error saying - Data Corrupt etc.
Thanks.
You can't, without some synchronization mechanism - by the time SECOND completes its check and starts to read the file, FIRST might have started writing it again. You need some sort of lock or mutex.
Two options for base Matlab.
If this is on a local filesystem, you could use a separate lock file sitting next to the data file to manage concurrent access to the data file. Use Java's NIO FileChannel and FileLock objects from within Matlab to lock the first byte of the lock file and use that as a semaphore to control access to the data file, so the reader waits until the writer is finished and vice versa. (If this is on a network filesystem, don't try this - file locking may seem to work but usually is not officially supported and in my experience is unreliable.)
Or you could just put a try/catch around your load() call and have it pause a few seconds and retry if you get a corrupt file error. The .mat file format is such that you won't get a partial read if the writer is still writing it; you'll get that corrupt file error. So you could use this as a lazy sort of collision detection and backoff. This is what I usually do.
To reduce the window of contention, consider having FIRST write to a temporary file in the same directory, and then use a rename to move it to its final destination. That way the file is only unavailable during a quick filesystem move operation, not the 20 seconds of data writing. If you have multiple writers, stick the PID and hostname in the temp file name to avoid collisions.
Sounds like a classic resource sharing problem between 2 threads (R-W)
In short, you should find a method of inter-workers safe communication. Check this out.
Also, try to type
showdemo('paralleldemo_communic_prof')
in Matlab
I have an application that updates a CSV file (single one), the CSV is being updated randomly from several processes, and I guess if two processes try to update it (add a row...) on the same time, some data will be lost I guess, or overwritten(?).
what is the best way to avoid this?
thanks,
Use Perl's DBI with the DBD::CSV driver to access your data; that'll take care of the flocking for you. (Unless you're using Windows 95 or the old Mac OS.) If you decide to switch to an RDBMS later on, you'll be well prepared.
Simple flocking as suggested by #Fluff should also be fine, of course.
If you want to have a simple and manual way to take care of file locking.
1) As soon as a process opens the csv, it creates a lock.
(Lock can be in the form of creating a dummy file. The process has to delete
the file(lock) as soon as it is done reading/updating the csv)
2) Have each process check for file lock before trying to update the csv.
(If dummy file is present, some process is accessing the csv,
else it can update the csv)
Does anyone know how to export a CSV file from a SQLite3 database in an iPhone app?
There are several iOS CSV libraries that can be used to export the data from the phone. This is a trivial data transformation task - you read the information line-by-line from the SQLite result set, and send it out into the CSV file. If you do not need to process the file after the complete read-in, then you should be able to stream the data as quickly as it is read.
Writing CSV files is mostly trivial - and can be implemented by someone without much effort. I'm certain there are libraries for iOS specifically, but I've used the python CSV export routines regularly, and they have been read by excel without much effort. You just have to be careful, as excel has the habit of interpreting the results once they are in there, making some accuracy calculations impossible.
the .csv file format is trivial, and I've implemented it several times in several languages. if you have numeric requirements, though, you will be spending a lot of time making sure that the program that you feed it into is accepting the numbers properly - and you will have to deal with 'I imported it into excel, saved it and the data is wrong now' bugs...
I need a suggest how to operate with large amount of data on iPhone. Let say I have xml file with ~120k text records. I need to perform search on this data. The solution i have tried is to use Core Data to store information in sorted order in caches. And then use binary search which works fast. But the problem is to build this caches. On first launch application takes about 15-25 seconds to build this caches. Maybe I need to use different approach to search the data?
Thanks in advance.
If you're using an XML file with the requirement that you can't cache, then you're not going to succeed unless you somehow carefully format your XML file to have useful data traversal properties -- but then you may as well use a binary file that's more useful unless you have some very esoteric requirements.
Really what you want is one of the typical indexing algorithms (on disk hash, B-tree, etc) from the get-go.
However...
If you have to read in and parse your XML text file, then you can skirt using a typical big and slow generic XML parser and write a fast hackish version since most of the data records you'll need to recognize are probably formatted the same way over and over. Nothing special, just find where the relevant data fields start, grab the data until it ends, move on to the next data field.
Honestly, 120k of text isn't very much-- sounds like whatever XML parser you're using is just slow. (I use this trick all the time for autogenerated XML data that just represents things like tables or simple data records -- my own parser is faster than any generic XML parser.)
This is probably the solution you actually want since you sound fairly attached to the XML file format. It won't be as error-proof as a generic XML parser if you're not careful, however it will eat that 120KB file up like nobody's business. And it's entry level CS work -- read in a file with certain specific formatting and grab the data values from it. Regexps are your friend if you have access to them.
Try storing and doing your searches in the cloud. (using a database stored on a server somewhere)
Unless you specifically need ALL of the information on the device..