I wanted to know which server/web framework I should go for. I'm developing an application that would run on IoT-styled cameras and/or small embedded boards(like Nvidia Jetson boards). After getting the feed from the camera, I'll do some postprocessing and then I want to transfer the data directly to the browser.
Some things to consider,
The main focus is on the local network since I'll be setting fixed local IPs for the cameras but may want to extend this in the future.
The numbers of clients would be small, typically around the 1-10 range.
Server will be low on ram and space
Camera can capture 5k images and I want to be able to transfer them with minimal fps drop. Currently, the fps for 5k images is approx 10 but it may increase in future.
The code base is in c++ so something compatible with it would be favoured.
I tried flask(even tho it is in python) and it worked great for lower resolution capture but there was a significant drop with higher resolutions.
Also, should I go with something like RTSP, RTMP or webRTC kind of thing? In an initial test with RTSP, I saw a drop in fps as well as an image quality drop.
Related
Native android supports different pixel densities like ldpi, mdpi, hdpi, xhdpi, xxhdpi and so on. This feature balances app quality and app size.
Currently I'm facing this issue in Unity mobile games (iOS & android) when I use,
high quality graphics it increases crashes and lagging in low-end devices.
low quality graphics it looks blurry and pixelated in high-end devices like iPad Pro etc.
I can use 2 different quality images but again it increases app size as the low-end devices end up downloading the HD images too.
How to solve this issue?
I suggest looking at the Unity Addressables system available in Unity 2020 LTS and beyond. This is a whole new tool that you'll have to investigate so I cannot provide a quick class or line of code to solve your problem. However, the Unity Addressables system is available in the Package Manager with docs available here.
Using this system will likely make it easier to run hi-res assets on lower-end devices. Since assets are streamed in when needed your texture memory is going to be significantly reduced as textures are unloaded as soon as you're done with them.
Addressables can also be used to load in assets remotely which would reduce your total file size. However, depending on how far you are in development this could be a big change.
You may also want to look at splitting the application binary if changing over to Addressables is too much work. If you split the binary, you can reduce the initial download of the application and have users opt-in to hi-res textures. There are a variety of other solutions provided by the Unity docs on Android builds here.
Good luck on getting your game to a shipped state!
I'm a bit lost looking through all the various Agora.io modules (and not sure what it means that only some of them have Unity-specific downloads).
I want to make a Unity app where two remote phones exchange data as follows:
Streaming voice in both directions
Streaming video in one direction (recorded from device camera)
Streaming a small amount of continuously-changing custom data in the other direction (specifically, a position + orientation in a virtual world; probably encoded as 7 floats)
The custom data needs to have low latency but does not need reliability (it's fine if some updates get lost; app only cares about the most recent update). Updates basically every frame.
Ideally I want to support both Android and iOS.
I started looking at Agora video (successfully built a test project) and it seems like it will cover the voice and video, but I'm struggling to find a good way to send the custom data (position + orientation). It's probably theoretically possible to encode it as a custom video feed but that sounds complex and inefficient. Is there some out-of-band signalling mechanism I could use to send some extra data alongside/instead of a video?
Agora real-time messaging sounds like it would probably work for this, but I can't seem to find any info about integrating it with Unity (either on Agora's web site or in a general web search). Can I roll this in somehow?
Agora interactive gaming could maybe also be relevant? The overview doesn't seem real clear about how it's different from regular Agora video. I suspect it's overkill but that might be fine if there isn't a large performance cost.
Could anyone point me in the right direction?
I would also consider alternatives to Agora if there's a better plugin for implementing this feature set in Unity.
Agora's Video SDK for Unity supports exporting projects to Android, iOS, MacOS, and Windows (non-UWP).
Regarding your data streaming needs, Agora's RTM SDK is in the process of being ported to work within Unity. At the moment the best way to send data using the Agora SDK is to use CreateDataStream to leverage Agora's ability to open a data stream that is sent along with the frames. Data stream messages are limited to 1kb per frame and 30kb/s so I would be cautious about running it on every frame if you are using a frame-rate above 30fps.
I am the IT guy for a small chain of preschools. Each school current has about 30-40 Axis IP webcams each, ranging from the old Axis 205,206 models, up to the Axis M1011. These cams allow parents to log into a website to view their kid's classrooms throughout the day. The IP cams are on a local network, streaming on port 80. Each school has one public ip address, on which is a reverse-proxy HTTP server. We directly serve the MJPEG from the Axis webcams by basically doing a URL rewrite to allow the client viewing app direct access to the cam via the Axis HTTP API.
This has worked fine for years, and even worked on iPhone, up until recently. In fact, it still works on iPhone, but only when connected Wifi. When the iPhone is connected via 3G, the MJPEG stream no longer works. The carrier is AT&T.
I have done quite a bit of research the past few days, and know that I have to move towards HTTP Live Streaming for Apple devices, so I'm just trying to get my head around the various parts of the puzzle I have to solve to make it work.
My mix of Axis cams may be a problem. The Axis 205 and 206 are MJPEG only, but the M1011 can serve h.264, but apparently only via RTSP. I'm willing to swap out the older Axis cams for the M1011 if that is required or helpful to the overall solution.
From my understanding so far, the first thing I have to sort out is getting my 30 to 40 Axis M1011 h.264 streams (re)encoded / chunked as MPEG-2 (.m3u8 and .ts files).
Question #1
Is it possible and/or feasible to setup one computer, with a "batch" file of VLC command lines, to start and encode 30 to 40 input streams, coming from Axis webcams, into MPEG-2, ready to serve up to iPhone? I'm wondering about the stability, hardware requirements, etc. to handle that many webcams. This same computer would have a HTTP server (probably IIS) and a public IP address, so no need to get the MPEG-2 files any farther than the local computer.
Question #2
For the older Axis models (205, 206) that only serve MJPEG, is is possible to use VLC to encode those to MPEG-2 as well? Am I correct in assuming that (re)encoding or re-packaging .h264 into MPEG-2 is a lot less "work" than encoding MJPEG into MPEG-2, or is it about the same amount of CPU, etc? I have installed the latest version of Videolan and it was easy enough to connect to the Axis M1011 .h264 stream via it's RTSP URL, so I know that much works.
Question #3
Can anyone share any specific VLC command lines or configuration, for either side of my given configuration: Axis M1011 .h264 and/or Axis MJPEG as inputs, and for the output, MPEG-2 (.m3u8 and .ts files), "chunked" into the size as required by Apple iPhone, especially when connected 3G.
Question #4
Finally, if anyone has another suggested approach (different encoder, Media Server that would work better, etc. ) I would like to hear those suggestions as well.
If you have enough network capability to recieve the input stream and enough CPU power and RAM to encode it is possible. There is no inherent limitation otherwise on using a single machine for multiple instances of vlc or ffmpeg.
If you have a camera feeding in 512 kbps input for 30 streams you need 30x512 = 15 Mbps network capability which should be quite fine for a modern machine.
The difficult part is getting 30 transcodes for iPhone output. The output is a MPEG2 TS container but internally it has to have mpeg4 or h.264 encoded content. H.264 encoding is expensive but requires lesser outgoing bandwidth for same quality at MPEG4 [ballpark ~30% less]
Assuming it is ok for you to view a 320x240 which for decent quality with mpeg4 should be decent quality at 256 kbps output.
For h.264 you could go to 192 kbps [baseline for 3GS and below]
Trouble now is do you have the horsepower to do the transcoding. Here is how you check
Take one video file representative of your input. Check the input frame rate. Transcode it to the output you need. Check the fps you are getting from ffmpeg for encoding. Notice the multiple of input rate it is at. Remove 5-10% for handling multiple simultaneous streams. that is the maximum you are getting on your machine.
ffmpeg can take input from a http or rtsp source and create an output which is segmented. See this for understanding how to use ffmpeg to create segments. You still need to write a small program to generate the m3u8 files I guess.
It is non trivial here but you will enjoy it. It can be done.
EDIT: Another thing is you only need transcoding to happen when someone is viewing it. So at a given time if someone is not viewing 10 cameras, you dont have to do those transcodings. So perhaps you should first find out statistically how many cameras are viewed at a given time. Look at your history. Then you need a less powerful machine.
EDIT: If you can handle some programming instead of command lines look at: this
ffmpeg might be a possible alternative for you. It's command line based, cross platform and you'll be able to control quality, formats as well as stream. The encoding/re-encoding quality basically controls the processing speed of your application, taken the throughput neccessary to grab the data out of the loop.
I'm working on getting a streaming video solution implemented for a client, with iOS devices targeted (mostly iPad).
I have diced up my video files into TS's and I have my accompanying m3u8 file. They are being hosted by a generic web host, and CDN'd by Amazon CloudFront, so on paper speed should be fine.
I am noticing that pretty much no matter what, the iPad is still having substantial buffering problems at pre-determined points (presumably where one segment ends and another begins).
My lowest bitrate for the TS files is 600kb/s which seems like that would be plenty low for typical WiFi streaming, but it still stops pretty hard.
I'm trying to figure out what is going wrong... I don't think it's the file hosting... as once it STARTS downloading, it goes fast. I feel like perhaps my m3u8 is somehow incomplete or inadequate...
As a side note, these videos are only 30-35 seconds long, and the media segmenter slices them into 3-4 pieces.
Has anyone seen anything like this?
I'd like to get real-time video from the iPhone to another device (either desktop browser or another iPhone, e.g. point-to-point).
NOTE: It's not one-to-many, just one-to-one at the moment. Audio can be part of stream or via telephone call on iphone.
There are four ways I can think of...
Capture frames on iPhone, send
frames to mediaserver, have
mediaserver publish realtime video
using host webserver.
Capture frames on iPhone, convert to
images, send to httpserver, have
javascript/AJAX in browser reload
images from server as fast as
possible.
Run httpServer on iPhone, Capture 1 second duration movies on
iPhone, create M3U8 files on iPhone, have the other
user connect directly to httpServer on iPhone for
liveStreaming.
Capture 1 second duration movies on
iPhone, create M3U8 files on iPhone,
send to httpServer, have the other
user connected to the httpServer
for liveStreaming. This is a good answer, has anyone gotten it to work?
Is there a better, more efficient option?
What's the fastest way to get data off the iPhone? Is it ASIHTTPRequest?
Thanks, everyone.
Sending raw frames or individual images will never work well enough for you (because of the amount of data and number of frames). Nor can you reasonably serve anything from the phone (WWAN networks have all sorts of firewalls). You'll need to encode the video, and stream it to a server, most likely over a standard streaming format (RTSP, RTMP). There is an H.264 encoder chip on the iPhone >= 3GS. The problem is that it is not stream oriented. That is, it outputs the metadata required to parse the video last. This leaves you with a few options.
Get the raw data and use FFmpeg to encode on the phone (will use a ton of CPU and battery).
Write your own parser for the H.264/AAC output (very hard)
Record and process in chunks (will add latency equal to the length of the chunks, and drop around 1/4 second of video between each chunk as you start and stop the sessions).
"Record and process in chunks (will add latency equal to the length of the chunks, and drop around 1/4 second of video between each chunk as you start and stop the sessions)."
I have just wrote such a code, but it is quite possible to eliminate such a gap by overlapping two AVAssetWriters. Since it uses the hardware encoder, I strongly recommend this approach.
We have similar needs; to be more specific, we want to implement streaming video & audio between an iOS device and a web UI. The goal is to enable high-quality video discussions between participants using these platforms. We did some research on how to implement this:
We decided to use OpenTok and managed to pretty quickly implement a proof-of-concept style video chat between an iPad and a website using the OpenTok getting started guide. There's also a PhoneGap plugin for OpenTok, which is handy for us as we are not doing native iOS.
Liblinphone also seemed to be a potential solution, but we didn't investigate further.
iDoubs also came up, but again, we felt OpenTok was the most promising one for our needs and thus didn't look at iDoubs in more detail.