Currently Im building a simple collaboration room for Hololens 2, I want people to be able to see each other.
So I built a system using udp socket to share the hand of user across network (client-server).
To try proving that my system is not at fault I tried a simple experiment ,
I have 1 server and 2 clients, 1 of the client is on Holo 2 and the other on windows standalone both are connected to the server via wifi.
When both are connected they can see each other avatar as well as their own (The computer simulate hand with the MicrosoftRealityToolkit)
Inside the hololens when the user move his hand, we can see the Pc client receiving it instantly with a really small delay
however the Holo recv his own move way later (0.5s delay approximatively)
Same experiment with the pc moving his avatar, It will recv his own move instantly , however the holo will once again recv it with a delay.
Also I noticed that if I increase the messages frequency from the server the hololens will start to lose all the packets
It's really strange because all the packets are received, all the movements are well restored they just have a big delay on holo. I suppose that the networks card is at fault and buffer the received msgs, but Im not knowledgeable enough on network stuff to really understand what's happening there.
Alright so here are the couple of things you can do.
Try with TCP communication so you can keep track of the packages.
Don't send the the pose values constantly, just send when the pose changes and when you receive the data on the Hololens, perform interpolation to get a smoother results. This will make it feel like you have less lag.
Late reply to this, but hopefully helps someone:
Hololens seems to absolutely hate receiving UDP broadcasts. Was having identical latency issue, even with all other devices disconnected from the network, simple packets, and sending at a far lower framerate than the hololens. Try addressing the hololens IP only, not the broadcast address, even if it means you need a second server running for all the other devices
Related
I'm researching and trying to building a RC car that can be controlled by the internet. I've started looking into how communication over the web works, but I seem to be going nowhere. My goal for the project is straight forward:
The RC car has an on-board camera and 4g wifi router that enables communication (driving commands, video streaming) over the internet. A Raspberry Pi will serve as the on-board computer.
I will be able to control the car with my PC even across the globe, as long as I'm connected.
I want to preferably do as much by myself as possible without relying too much on other people's code.
So here are my questions:
How does an application communicate over the internet? What is the interface between the application's logic (e.g pressing "w" to go forward), and transmitting/receiving that command over the internet?
How is video data stream handled?
I've looked into WebRTC and WebSockets for communication, but they are aimed at providing real time communication to web browsers and mobile, not something like a raspberry pi, and I'm still in the blind as for exactly what technology should I use, and in general the overview and architecture of real time communication.
All I've achieved so far was an app that sends text messages between devices through a server on my network, with very primitive reading/writing using java Socket.
In short, what does messenger/skype/zoom do in the background when you send a message or video call?
Any guidance would be greatly appreciated.
First things first. You cannot do real-time control over Internet, period. There is absolutely no way to guarantee the delivery latency. Your control commands can arrive with a delay from milliseconds to seconds, or never. No way around it.
Now, you can still do a number of reasonable steps to absorb that unpredictable latency as much as possible and safe-guard your remote robot from the consequences of the unreliable communication.
For example, instead of sending the drive commands directly - as in, acceleration, deceleration, turn angle, etc., you can send a projected trajectory that is calculated from your drive commands locally on a model. Your RC car must be sufficiently smart to do some form of localisation - at the very least, wheel odometry, and with a good enough time sync between the sender and the RC car you'll be able to control the behaviour remotely without nasty consequences of drive commands executed at an unpredictable delay.
You can add a heart-beat to your protocol, to monitor the quality of the communication line, and if hear-beat is delayed or missing, initiate emergency stop.
Also, don't bother with TCP, use UDP only and maintain your own sequence counter to monitor missing packets. Same applies to the telemetry stream, not just command channel.
I am trying to reverse engineer the API of an IoT device using mitmproxy. My setup is an iphone, computer running on MacOS 10.14, and an IoT device (watering pump) that can only access the wireless network after being plugged into the computer via USB connection, i.e. a not genuinely WiFi device. My phone is configured to point to the computer, which is running mitmproxy on a standard configuration.
When I send instructions from the app controlling the device on my phone to the device itself, presumably these instructions are sent to the computer, to the device cloud server, and then to the device. With these assumptions, one would think that they would see the flow of POST requests in mitmproxy before observing the results of those instructions. That is, if you send an instruction to turn on the pump, you'd think you'd see POST request containing that instruction show up in the mitmproxy flows before you see the pump turn on.
However that is not the case here. What happens is that, when I send instructions from the app, I observe the expected behavior from the IoT device, and then the flow of requests appear on the mitmproxy console seemingly at random. There seems to be no determinate relationship between the instructions I send and the requests that appear; they show up 5 seconds later, 5 minutes later, or 30 minutes later. Is this an intentional security feature? To somehow jam MITMproxies so that hackers cannot easily isolate the knowledge of which packet is performing which instruction? Or is it just something that I am doing wrong? Does anyone have any ideas as to what could be happening hear and potential solutions for making the flow of requests appear in real time? Ty
I have an Arduino based device interfaced to a 3G modem which I use to record data from several sensors in a remote environment. I would like to be able to send commands and stream some data from the device every now and then back to my standard network connected PC. If the remote device was connected to a WIFI or other local area network this would be relatively straightforward, but as the device connects over 3G this means that it is behind the 3G carriers NAT and so establishing a connection to the device becomes difficult.
The device can, of course, open a TCP connection to my host PC any time it wishes, the problem is telling the device when i want it to do so. I need some way of getting some kind of message to the device to notify it that I would like it to initiate a connection to my PC.
I've been reading up on NAT traversal techniques that app developers use to initiate P2P comms between 2 devices both behind NATs such as UDP and TCP 'hole punching' but this method seems rather too complex for my arduino system. Another general idea is to have the device polling a web server periodically looking for a signal to initiate a connection, but I'm not sure how much traffic (and data usage costs) this would generate as the device would have to poll every 10 seconds or so in order to make sure it initiates it's connection within a reasonable time frame of the request being set on the web server that it polls.
Is there any commonly used method of achieving something like this? Any general ideas or insight would be much appreciated
Thanks,
James
I think the solution will depend largely on your particular applications and requirements.
There are several ways to achieve this type of functionality and it looks like you have covered some of them already. The most common are:
have the device poll the server. This may be ok depending on the response times you need. If you need to poll as regularly as you suggest above then I imagine power may be more important to you than data rates, especially if you are battery connected. With a typical 3G data plan the polling itself will have a negligible data overhead, I would think.
have the server send a SMS which then triggers the device to contact the server. You need to make sure the SMS variable delivery time is ok for you and you also have to be aware that SMS delivery is not guaranteed so you would have to build in some sort of check for delivery at a higher layer (or into your application).
use some low cost Android based device for your 3G connectivity and leverage the Google push notifications mechanism
It is worth noting that server polling typically gets very bad press as it is seems intuitively wasteful to have the client and the server constantly checking for messages, especially when the actual messages are fairly infrequent. However, underneath most push solutions there is still a pull mechanism in the background, albeit generally a very efficient one that may, for example, piggy back on other messages between the network and the mobile device and hence have minimal power and data overhead. Personally, I would say that if you do not have major concerns with battery/power or with the load polling might generate for your servers, then it is worth exploring if the simplicity benefits of a polling solution outweigh its other disadvantages.
I'm trying to create an iOS application that sends data over UDP continuously over wifi/3G network.
I have an issue when I launch my app over 3G network after like 10 seconds I get this message :
sendto() : No buffer space available
It's not that a big deal because my app still works well BUT when I quit the app, I guess my buffer stays full because I can't use 3G anymore (I have to wait some time or reboot my phone)
Is there a way to flush this buffer before I quit my app ?
It sounds as if you're hitting the outbound bandwidth limit. If your app does this continuously while in use, isn't that going to make it very expensive for users to run? Most mobile users, I would guess, are on some kind of metered plan where they pay for transferred data.
I would guess that closing the socket normally before exiting should flush it first, since you've requested the data to be sent after all, but sometimes UDP sockets don't try very hard to deliver the data (since they are "lossy"), perhaps that's what's happening in your case too.
I'm interested in writing an app that send messages over IP (using 3g, not neccesarily on the same WiFi network as the receiving end) to a PIC microcontroller connected to a router (via ethernet or wifi)
I saw some descriptions and examples on how to send messages on the same network, not sure if just by giving a different IP it would work outside the network it self. I was wondering how can it be received by the PIC (still hasn't decided which PIC, depends on the possibility to perform this)
and in turn, depends on the msg received, the PIC will perform an action, for example, light a certain LED in a LED array.
I have the sending side (the app sending over IP), and receiveing side (the PIC which lights the LEDs)
I'm just not quite sure what to send, or if such "translation" is even possible.
I've searched the web but couldn't find any such thing except for made kit (for RC cars for example)
Thanks.
Carmel
Here's a UDP library I built for Mac/iOS wrapped in simple Cocoa that sends UDP packets to an IP address of your choice. Comes with sample server client app.
This is generic UDP packets so you just have to listen for them on the other end ( whatever that platform is ) and read the content. Let me know if you need help with that
https://github.com/Shein/Mac-UDP-Client-Server
Well, if the PIC can run a TCPserver, or maybe just UDP, then sure you can control a LED by sending an appropriate message. With an appropriate static IP address and routing, then you can use your iPhone to turn on the LED from anywhere you can get a connection.
PICs are not exactly renowned for having high performance and lots of RAM. If you do manage to get a server working, try to not use XML/SOAP
This looks more like a question for http://electronics.stackexchange.com.
If your familiar with SPI you could consider Microchip's MRF24WB0MA/RM module which
enables PIC to connect to a wireless network (either to an infrastructure wireless or
to AdHoc, or it can create its own AdHoc).
If you're using MCC compiler then you also have library available for this module
at Microchip Application Libraries.
TCP stack takes some memory space though. PIC18F would do but you'd be better of with PIC24F (they tend to be cheaper than 18F anyway).