I want to know if the following approach is correct to reverse a queue:
-Dequeue all elements of queue and store them in a array a from index 0 to a.length-1
-Enqueue each element array back to queue but starting at index a.length-1 to 0
is there a better solution? we cud use a stack but its basically the same as using an array like my solution above.
If you know the queue size in advance, your method will work and is slightly more efficient than using a stack because array access is slightly faster than updating the state of a stack.
If you do not know the queue size for any reason, a stack will work fine.
Ensure that your queue is not changing during the reversal operation. If items can be added to the queue while you are dequeueing elements, the size originally measured for the array may no longer be valid. Depending on how you write your code, that could either lead to an exception as you index past the end of the array, or to you leaving some items in the original queue and not reversing them.
Related
I am writing a scala script to download all items from the hacker news API. There are ~12M items, each being a JSON of ~200 bytes.
I identified the following issues:
Storing the data: I tried to save each item as a single JSON file, but it became very hard just to barely list them (using Linux, ext4 file system). So I changed it to just append JSON items to multiple (100) files (by taking the item's id module 100).
Keeping track of what has been downloaded, because I want to be able to stop/continue the application. First I tried writing the downloaded ids to a textfile, but it turned out a little bit buggy. So now I just read all the items and collect the ids. (It works.)
All this is done with 1 Master actor and an arbitrary number of Worker actors (tens). The Master has a Queue[Int] and pops it and Workers ask for work.
The problem I am having is fairly simple but I haven't been able to solve it in a nice way.
I can collect the ids from items already downloaded in a list. But what I really need is the complement to that set; I need all the items I have not downloaded, up to the highest item id.
I tried using a range (1 to maxItemId) and subtracting the set of done jobs but it is really slow. reaaaaaaally slow.
Now I am using a Stream, and when a worker asks for a job, I check if the stream's (the next job) has already been done. If so, I give it to the Worker. Otherwise I check the next one.
The problem with this approach is that I can not put jobs back at the stream if they fail. That would be easy with the Queue; but then again I am having trouble just setting up the queue with millions of items.
What could be a better approach to this? I don't think the issues here are trivial, this is a very large number of tasks to perform and keep track of, but it shouldn't be so hard as well.
Thanks!
As far as I understood your question, I think you don't need a very complicated data structure here.
Assuming your ids are sequential from 1 to maxItemId, you can use an array of Boolean with maxItemId size to keep track of processed items. You initialize this array by reading the processed ids. And you find the next job by searching for the next false entry.
Assuming that your maxItemId is around 12M, iterating over all items is pretty much instantaneous.
Supposed I have a key-value database, and I need to build a queue on top of it. How could I achieve this without getting a bad performance?
One idea might be to store the queue inside an array, and simply store the array using a fixed key. This is a quite simple implementation, but is very slow, as for every read or write access the complete array must be loaded / saved.
I could also implement a linked list, with random keys, and there is one fixed key which acts as starting point to element 1. Depending on if I prefer a fast read or a fast write access, I could let point the fixed element to the first or the last entry in the queue (so I have to travel it forward / backward).
Or, to proceed with that - I could also have two fixed pointers: One for the first, on for the last item.
Any other suggestions on how to do this effectively?
Initially, key-value structure is extremely similar to the original memory storage where the physical address in computer memory plays as the key. So any type of data structure could be modeled upon key-value storage surely, including linked list.
Originally, a linked list is a list of nodes including the index information of previous node or following node. Then the node it self should also be viewed as a sub key-value structure. With additional prefix to the key, the information in the node could be separately stored in a flat table of key-value pairs.
To proceed with that, special suffix to the key could also make it possible to get rid of redundant pointer information. This pretend list might look something like this:
pilot-last-index: 5
pilot-0: Rei Ayanami
pilot-1: Shinji Ikari
pilot-2: Soryu Asuka Langley
pilot-3: Touji Suzuhara
pilot-5: Makinami Mari
The corresponding algrithm is also imaginable, I think. If you could have a daemon thread for manipulation these keys, pilot-5 could be renamed as pilot-4 in the above example. Even though, it is not allowed to have additional thread in some special situation, the result of the queue it self is not affected. Just some overhead would exist for the break point in sequence.
However which of the two above should be applied is the problem of balance between the cost of storage space or the overhead of CPU time.
The thread safe is exactly a problem however an ancient problem. Just like the class implementing the interface of ConcurrentMap in JDK, Atomic operation on key-value data is also provided perfectly. There are similar methods featured in some key-value middleware, like memcached, as well, which could make you update key or value separately and thread safely. However these implementation is the algrithm problem rather than the key-value structure it self.
I think it depends on the kind of queue you want to implement, and no solution will be perfect because a key-value store is not the right data structure for this kind of task. There will be always some kind of hack involved.
For a simple first in first out queue you could use a few kev-value stores like the folliwing:
{
oldestIndex:5,
newestIndex:10
}
In this example there would be 6 items in the Queue (5,6,7,8,9,10). Item 0 to 4 are already done whereas there is no Item 11 or so for now. The producer worker would increment newestIndex and save his item under the key 11. The consumer takes the item under the key 5 and increments oldestIndex.
Note that this approach can lead to problems if you have multiple consumer/producers and if the queue is never empty so you cant reset the index.
But the multithreading problem is also true for linked lists etc.
Background:
I'm designing a list-like control (WinForms) that's backed by a DbSet. A chief requirement is that it doesn't load the entire list into local memory. I'm using a DataGridView in virtual mode as the underlying UI. I'm planning to implement the CellValueNeeded function as orderedQueryable.ElementAt(n).
Problem:
I need to allow the control's consumer to get/set the currently-selected value, by value rather than by index. Getting is easy--it's the same as the CellValueNeeded operation--but setting is harder: it requires me to get the index of a given element. There's not a built-in orderedQueryable.FirstIndexOf(value) operation, and although I could theoretically fake it with some sort of orderedQueryable.SkipWhile shenanigans where the expression has a side-effect, in practice the DbSet's query provider probably doesn't support doing that.
Questions:
Is there an efficient way to get the index of a particular value within an IOrderedQueryable? How?
(If this approach turns out to be untenable, I'd settle for suggestions on how I might restructure the problem to make it solvable.)
Side notes:
Elements can be inserted and removed from the list, in which case the old indices will be invalid--that's acceptable, since they're never exposed to the consumer. It's an error for the consumer to attempt to select an item that isn't actually in the list, and actually the consumer would have gotten the item from the list in the first place (although perhaps the indices have changed since then).
I have a fetchRequest which takes up to 4-5 seconds to finish. Since it is part of a search-as-you-type solution, is there any way to abort a fetchRequest?
I use a timer set to start searching my database after 600ms after the user ended typing. So there is a possibility that a new search has to start before the old one has finished.
I haven't found any methods for the NSMangedObjectContext that seem to be right. Is simply setting the old fetchRequest = nil the way to go? Or is there still something going on in the background?
Any ideas?
Thanks in advance!
PS: I'm also trying to enhance my query speed. Maybe someone has an idea for that too:
https://stackoverflow.com/questions/4695729/query-performance-with-large-database
Better way is a place limit for fetch request objects.
(void)setFetchLimit:(NSUInteger)limit
Parameters
limit
The fetch limit of the receiver. 0 specifies no fetch limit.
Discussion
Special Considerations
If you set a fetch limit, the framework makes a best effort, but does not guarantee, to improve efficiency. For every object store except the SQL store, a fetch request executed with a fetch limit in effect simply performs an unlimited fetch and throws away the unasked for rows.
Assuming you're using a UITextField for the text entry, why don't you move your fetchRequest logic to the textField:shouldChangeCharactersInRange:replacementString: (UITextField) delegate method?
This method is called every time a user enters or deletes a character from the textfield, so it is the perfect place to check for a minimum number of characters before firing off a fetch request, as well as changes to the text that would require you to set the existing fetchrequest to nil and start a new one.
I don't think you can, since it almost certainly ties up the thread doing the fetch. The last time I needed to do something like this, I spawned a background thread, with a basic condvar (NSCondition) to signal when a new input was available, and -performSelectorOnMainThread:... to signal when the output was ready. This means that the background thread will continue to work on out-of-date inputs for a while before picking up the new "most recent" input.
You can probably do a similar thing with NSOperation/NSOperationQueue by cancelling all operations on the queue (representing old inputs) before adding a new one (representing the latest input).
Since NSMO/NSMOC isn't thread-safe, you probably want to pass the set of (the first few) MOIDs instead.
As I remember and checked, the usual way for traversing a tree or crawling the web breadth first (BFS) is by using a queue. Is there actually a way to implement it not using a queue?
I know this question is old now, but I just wanted to answer. You can do this with arrays, linked lists (or any other linear container) and without recursion. Keep two containers, old and new, and swap old with new when you traverse all of the items in old. Very similar to the implementation with queue.
In Python it would look like:
def breadth_first(root):
if not root:
return
old = []
new = []
old.append(root)
while old:
for n in old:
process(n) # Do something
if n.left:
new.append(n.left)
if n.right:
new.append(n.right)
old = new
new = []
Runtime complexity would be the same as the queue implementation, O(n).
You really should be using a queue, as its easier to implement. Also, a queue allows for multiple machines to work together (one queues site while another pops sites off of the queue to traverse).
The only other way I see to do this is by using recursion (much more difficult, and uses only marginally either more or less memory).
With recursion. But the queue is in the stack...
if you care about ordering, use queue. queue preserves the insertion ordering. or you can use list's implementation, say, two array lists, to alternate. But fundamentally, list preserves ordering too.
if you don't care about ordering, you can use any set implementations. sets doesn't preserve this ordering.
For example, in BFS implementation, if you don't care the ordering of nodes, you can use two sets, old and new to alternate, rather than a queue.