I'd like to use an optical flow system to get velocities from surrounding environment. I've read papers about how optical flow works, but they don't treat details about optic sensors.
My question is: How do I determine how much computational power is required to perform optical flow analysis?
I'd like to use a low-power system (like microcontrollers), but I don't know what kind of camera I could use with such a system. I mean, could it be color or does it need to be B/W? Rolling shutter or global shutter? Which frame rate or number of pixels?
I'd like to specify the system myself but, without knowing how those camera attributes impact the processing load, I'm not sure where to start.
As Chuck already said in the comment. You first need to start with something. Opticalflow calculation really depends on what you are using it for and what you are trying to achieve. For realtime applications you might want to consider using faster processors (this is always true though).
Continuing to my answer.
Opticalflow calculation performance depends on few main things:
The optical-flow method you choose (dense or sparse), you can read more about it here and here. Of course that you should take into account not only that sparse is faster than dense, also that sparse might be less accurate in some cases. Again, this depends on what you're trying to achieve.
In addition, you will see that there are different optical-flow algorithms. Some might be faster than others. There are many algorithms such as Lucas-Kanade, Horn-Schunck, TVL1, Farneback, etc.
Most optical-flow methods from libraries such as OpenCV gives you the ability to change some parameters in order to play with the trade-off between accuracy and performance. See this and also check the OpenCV methods such as this and this for example - see the different arguments.
The resolution of your image. Smaller image usually means faster calculation.
Few things you might also want to consider:
If you are using a processor that has multiple cores, make sure that you are using all the cores in the optical-flow calculation. Some libraries may already do this for you, but in some cases you will need to do it by yourself. Take a look at my question and answer in this post, it might give you some idea and help you getting starting with such case.
If you want more accurate optical-flow results you must use global shutter camera. Rolling shutter cameras, such as most of the web-cams, will give you an extra error you don't want.
You don't need color image, if you have a grayscale camera it will be even better. If not, you will need to convert it to grayscale (not B/W) for faster performance as well.
Some libraries such as OpenCV has an option (in some cases) to run these algorithms on a GPU. If using a GPU is an option you might want to consider this as well.
From my own experience, the main thing that gave me a boost in performance was changing my resolution from 640x480 to 320x240 and even 160x120. In my case it didn't really hurt the accuracy.
I used an Odroid U3 mini-pc with OpenCV PyrLK algorithm and input frames of 320x240 resolution. After applying what's described here (splitting the image to 4 for parallel calculation) it worked pretty well (realtime).
The answer given by Sarid has some strong points, and many of them are shared by researchers around the world. My opinions are shared by anyone who has actually worked with these topics in the real-world setting.... with real world, i mean implementing optical flow in drones, on mobile phones and IP cameras that are not sitting in a protected office, and where other systems (such as humans) need to interact and be co-dependent.
First of all, depending on your problem, you may want to invest time in looking for ready-made solutions. Optical flow sensors are readily available, cheap and robust (but usually not strong in accuracy). These are the kind of sensors you find in optical mice. They are low power, and easily interfaced with micro-controllers. Some have staggering sample rates of thousands of fps. They commonly have low spatial resolution however, and (to emphasize) high robustness but low accuracy.
If instead you are looking for the kind of optical flow that can be used for shape from motion, pedestrian detection and video-encoding, for example, then you are probably better off to look for something more advanced, and thats where Sarids answer becomes relevant.
Since your question has been migrated from robotics stack exchange, I am going to assume you are interested applications close to machine control and human machine interaction. In that case, the most important aspects are the ones usually most ignored by people working in the field of optical flow estimation, namely:
Latency. If you have a human interfacing at the front-end... then the common term is "glass-to-glass latency". This is completely different from the fps of your system, which is connected to throughput. If you find that you are in a discussion with someone, and they do not understand the difference between latency and fps, then they are not the expert you are interested in. For example, almost all researchers in computer vision who do GPU implementations of optical flow add massive latency by allowing for frame delays and ineffecient memory handling (inefficient from perspective of latency, but efficient in terms of throughput and hard-ware utilization). Consider the problem of controlling a drone, say make it self-stabilizing, it is better to receive a bad optical flow estimation 10 ms earlier, then a good one with 10 ms extra delay.... especially if the optical system does not give you any upper bounds of the delay for any given time.
Algorithm stability. This is completely different from accuracy. Accuracy is what 99% of all research in optical flow has been obsessing about for the last 30 years. Stability is not at all something evaluated in the Middlebury benchmark for example. Stability deals with how small changes in your data will guarantee small changes in the estimated optical flow. While some good work has been done in the community (on robust statistics most interestingly) in the end the final evaluation of any algortihm disregards stability. Consider the optical mouse as a good example. The first generations of optical mice had higher accuracy (the average error from the true motion was smaller) but they had lower stability (especially when you ran the mice over "bad textures", with rotational motions). Later generations of optical mouse have worse accuracy, but are focusing on the stability, as that is the most important thing. You dont experience the mouse cursor jumping around as much as you did the earlier days of the devices.... but if you move the mouse on your mat, left and right repeatedly, you will see the cursor slowly drifting (i.e. low accuracy).
Heat. Any device that will estimate high accuracy optical flow, will require lots of computations. When it comes to computations per watt, GPUs are not that good. In drones, you may be able to get away with this, because it is a setting where you have active cooling as a by-product of the propulsion system. In the real-world, you most often can not assume active cooling nor unlimited power supply.
To conclude, its a fascinating area, and I hope you have a great experience coding solutions.
Related
I'm creating an evolution-artificial-life-simulation game in 2D (purely for fun purposes). It combines neural networks (for behaviour controlling) and genetic algorithm (for breeding and mutations).
On input I give them X,Y position of nearest food (normalized) and X,Y position of the "look at" vector.
Currently they fly around and when they collide with food (let's call it "eating apples") their fitness index is increased by one and the apple's position is randomed - after 2000 turns the GA interrupts and does its magic.
After about 100 generations they learn that eating apples is good and try to fly to the nearest ones.
But my question, as a neural network newbie, is - if I created a room where apples spawn way more frequent than on the rest of the map, would they learn and understand that? Would they fly to that room more often? And is it possible to tell how many generations would it take for them to learn?
What they can learn and how fast depends a lot on the information you give them access to. For instance, if they have no way of knowing that they are in the room where food generates more frequently, then there is no way for them to evolve to go there more frequently.
It's not entirely clear from your question what the "look at" vector is. If it, for instance, shows them what's directly in front of them, then it might be enough information for them to figure out that they're in the room of plenty, particularly if that room "looks" distinctive somehow. A more useful input to give them might be their current X and Y coordinates. If you did that, then I would definitely expect them to evolve to be in the good room more frequently (in proportion to how good it is, of course), because it would be possible for them to take action to go to and stay in that room.
As for how many generations it will take, that is incredibly hard to predict (especially without knowing more about your setup). If it takes them 100 generations to learn to eat food, then I would expect it to be on the order of hundreds. But the best way to find out is just to try it.
If it's all about location, they may keep a state of the map in their mind and simple statistics will let them learn where the food may be located. Neural nets is an overkill there.
If there are other features of locations (for example color, smell, height etc...) to map those features to the label (food exists or not) is good for neural nets. Especially if some of features not available or not reliable randomly at the moment.
If they need many decisions to reach the goal, you will need reinforcement learning. Forexample, they may go to a direction which is good for a time, but make them away from resources they will need later.
I believe that a recurrent neural network could learn to expect apples to spawn in a certain region.
I currently work in an experimental rock mechanics lab, and when I conduct an experiment, I record the output signals such as effective torque, normal force and motor velocity. However, the latter quantity causes significant cross-talk over the recorded channels, and I want to filter this out. Let me give an example:
Here the upper plot is the strong signal (motor velocity), and the lower is an idle signal that is affected by the cross-talk (blue is raw signal, red is median filtered). The idle channel is only recording noise. We see three effects here. When the motor voltage changes:
the amplitude of the noise increases
the idle signal's median shifts
there is a spike that lasts approximately 0.1 seconds
If we zoom in on the first spike that occurs at around 115 seconds, we get the following plot. This does not seem to be your typical delta-function type of spike, but rather some kind of electronic "echo".
I have seen much work on blind source separation through independent component analysis (ICA), but that did not prove to be effective in my situation. However, since I know the shape of the signal that is causing the cross-talk, there may be better ways to include this information. My question is this: is there a filter or a combination of filters that can tackle the effects mentioned above?
As I am a geologist and not an electrician or mathematician, I don't have a proper background for this kind of material, so please bear with me. I write Python, MATLAB and C++ quite well, so suggested algorithms written in any of those languages is preferred (but not required).
The crosstalk you encounter, results from a parasitic transmission line. Just think of your typical FM-receiver - where the wires equal the antennae. These effects include parasitic and inductive coupling, and form an oscillator (which is the reason, why you cannot see, the theoretically ideal delta spike)
I recognize two different approaches:
use a hardware filtering circuit
use a software-implemented filter
ad 1:
depending on the needed bandwith (maximum frequency/rate of change) on the idle channel, you can determine the corner frequency, as well as the required filter-order, for a given rate of suppression
ad 2:
you can implement several types of filters (IIF, FIR) which resemble these circuits.
Additionally, if you are measuring the aggressive signal anyways, you can use the measurement on the idle channel to determine system-parameters for a mathematical model of the crosstalk. With this model you'd be able, to exclude the interference by calculation
Hi I have collected some process data for 3 years and I want to mimic a EWMA prospective analysis, to see if my set smoothing parameter would have detect all the important changes (without too many false alarms).
It seems like most textbooks and literature that I have looked that use a mean and standard deviation to calculate the Control Limits. This is usually the "in-control" mean and standard deviation from some historical data, or the mean and sd of the population from which the samples are drawn. I don't have either information.
Is there another way to calculate the Control Limits?
Is there a variation of the EWMA chart that does not use mean and standard deviation?
Any creative ideas?
Thank you in advance
From a practical/operational perspective, the use of statistical analysis of historical data alone, is rare. Yes, it provides some guidance on how the process (and its control system) are performing, however the most important thing by far is to have a good understanding and knowledge of the "engineering limits".
I refer to the operational limits, which are determined by the specifications and performance characteristics of the various pieces of equipment. This allows one to develop a good understanding of how the process is supposed to behave (in terms of optimal operating point and upper/lower control limits) and where the areas of greatest deviation from optimal are. This has very little to do with statistical analysis of historical data, and a great deal to do with process engineering/metallurgy - depending on the type of process you are dealing with.
The control limits are ultimately determined from what the Process Manager / Process Engineer WANTS, which are usually (but not always) within the nameplate capacity of the equipment.
If you are working within the operational limits, and you are in the realm of process optimisation, then yes, statistical analysis is more widely used and can offer good insight. Depending upon the variability of your process, how well your control system is set up, and the homogeneity of your feed product, the upper/lower control limits that are selected will vary. A good starting point is the optimal operating point (e.g. 100 m3/hr), then use a sensible amount of historical data to calculate a standard deviation, and make your upper limit 100 + 1 standard dev, and your lower limit 100 - 1 standard dev. This is by no means a "hard and fast" rule, but it is a sensible starting point.
I am analyzing EMG data in my research lab. One of the steps is to calculate a continuous wavelet transformation of the dataset (size ~80000). Therefore, I use Matlab with the wavelet toolbox and "cwt" to plot a 3D-scalogram.
The calculation takes a lot of time and any interaction like 3D-rotation (which is very important to see different aspects of the data) is nearly impossible.
The resource-monitor shows that only one of my hexa-core processor is working. I use parallel computing for other calculations and haven't found any solution or even a similiar question like this.
Is there anything I can do to activate multicore support for plots?
I'll hazard an educated guess and plump for the answer No to your question Is there anything I can do to activate multicore support for plots?
Matlab can certainly use multiple cores for its computations. Many of its intrinsic functions are already multi-threaded and will use any available cores without the programmer (or user) having to do take any special measures. For your own computations you can use the Parallel Compute Toolbox.
However, unless you have some very special graphics hardware (and if you do why didn't you mention it ?) your display shows you why only one processor is being used when you interact with your 3D plots -- somewhere between the screen and the hardware of your computer there is a bottleneck through which the outputs of all those cores are squeezed into one stream of bits and bytes for presentation.
Your experience is consistent with that bottleneck being the Matlab visualisation routines, I think it is safe to conclude from the evidence you present, that the Mathworks haven't multi-threaded the routines which compute the new screen positions of each element in a plot as you rotate it, or any of the other processing that goes on to turn the results of your analyses into a picture or pictures. If they did parallelise those routines, that would shift the bottleneck but not remove it.
To remove the bottleneck you would have to have a way for different Matlab threads to separately address different parts of your screen; I see no evidence that Matlab has that capability. Google will find you a ton of references to parallel rendering but I see no sign that Matlab currently implements any aspect of this.
I'll just add, in response to your comment where you write unfortunately I can't resample my data that you should be mindful that Matlab's visualisation routines are resampling your data for presentation unless you are only visualising datasets with numbers of samples less than the number of pixels available. If you visualise a time series with 80000 samples on a display with 2000 pixels horizontally, something has got to give.
You might get better graphics performance and superior understanding if you take charge of that resampling yourself.
Matlab plotting performance is pretty bad, it's more focused on customizability than performance. Using MEX to run some native C++ code to plot the data with OpenGL will likely be much much faster.
Does anyone know if Core Location in the iPhone OS uses anything but simple vector math to calculate speed? I've read that the GPS system can provide speed measurements that can be accurate when position is not (I believe using the Doppler shifts of the signals).
I've tried and failed to see if the iPhone does this. The question is basically, does this data contain information or is it just convenience functions, using (filtered?) location data?
I suppose my question is if anyone have tried this in reality, or knows beyond what is in the documentation.
The Core Location documentation describes the speed reading thus:
This value reflects the instantaneous speed of the device in the direction of its current heading.
While not absolutely definitive, this strongly suggests that the reading is direct, rather than an interpolation of positions, which cannot be described as "instantaneous" by any reasonable definition.
The GPS system in itself is not able to provide speed measurements. The only way this can practically be done is by comparing to discrete position measurements and the time between those. It's just a matter of applying simple math to get the speed and direction traveled. More samples can be used to get a more accurate measurement.
It is not feasible to measure the speed directly by simple GPS receivers, e.g. by use of Doppler shift. This is due to the fact that each satellite itself is traveling at very high speed around the globe. Each satellite orbits the globe twice every day, resulting at a speed of almost 14000 km/hour. Since the direction of the satellite compared to the GPS unit varies depending on where it is on the sky, the difference in the measure Doppler shift would be huge compared to the Doppler shift resulting from moving of the GPS receiver itself.
I'm however not saying that this couldn't be done by very sophisticated hardware and algorithms, but the cost/benefit would probably not be worth even considering it.