I'm trying to set up a custom video source for a video stream in Agora for Unity, following the instructions from Agora's developer center here (and particularly, the example code at the bottom):
https://docs.agora.io/en/Video/custom_video_unity?platform=Unity
THIS CODE WORKS. I can successfully send a video stream and watch it on another device and it looks correct.
However, the Unity console is reporting an error on every single frame, saying:
D3D11 unsupported ReadPixels destination texture format (14)
Unity's documentation for Texture2D.ReadPixels says that it works on RGBA32, ARGB32 and RGB24 texture formats, but Agora's example is using a texture in BGRA32 format.
If I alter the example to set the texture to RGBA32 format instead, then the program still works, except the colors are wrong--red and blue are swapped (unsurprisingly).
I tried to adjust the expected texture on Agora's end by modifying this line of the example:
externalVideoFrame.format = ExternalVideoFrame.VIDEO_PIXEL_FORMAT.VIDEO_PIXEL_BGRA;
But...there is no corresponding define for VIDEO_PIXEL_RGBA. The available options are VIDEO_PIXEL_UNKNOWN, VIDEO_PIXEL_I420, VIDEO_PIXEL_BGRA, VIDEO_PIXEL_NV12, VIDEO_PIXEL_I422
So....my app is functioning correctly, but I'm drowning in error messages of dubious significance, which seems like it's going to cause headaches for development and debugging down the road.
What can I do?
For the inverted color issue, make sure you have the same encoding format on the receiver side. If you are using the SDK script VideoSurface.cs, change the line where it instantiates the Texture (about line 172), where it should be like:
nativeTexture = new Texture2D((int)defWidth, (int)defHeight, TextureFormat.BGRA32, false);
(It was RGBA32 in the stock SDK code).
Update: This format issue has been resolved in version 3.0.1. If it hasn't been released in Asset Store, you may come grab the beta to try out. Check with slack channel here: https://agoraiodev.slack.com/messages/unity-help-me
Related
I'm working with the raw camera image, but need to restart the preview following the capture of an image. When i call startPreview() following takePicture, the android hangs because the camera is till in use. I've waited till after the raw image has been written to disk, but the camera is still in use, so the start preview still hangs the system.
camera.takePicture(null, null, null);
(test needed here){
camera.startPreview();
}
Putting the start preview in the rawcallback hangs the android.
When calling takePicture, the jpegCallback occurs after the camera is finished, so it is safe to start the preview, but this also creates a .jpg.
The question is following the takePicture, is there any way to determine when the camera is idle? (other than the jpegCallback?).
I've found ways around the problem, such as starting the preview on a timer, but still wonder if there is any way to determine the actual status of the camera and a way to test when it is save to restart the preview.
After calling the takePicture() function, you must wait until the JPEG callback has returned for the camera to be ready to use again. There is no way to monitor its status. The documentation explicitly states the following:
After calling this method, you must not call startPreview() or take
another picture until the JPEG callback has returned.
If you need some fast way of grabbing frames (i.e. images) without stopping and restarting the camera preview, you can implement the PreviewCallback interface which will simply allow you to grab preview frames without stopping the preview at all. Just remember that you will need to convert these from YUV format (which is not difficult using Android's YUVImage class.
I need to process the video frames from a remote video in real-time and present the processed frames on screen.
I have tried using AVAssetReader but because the AVURLAsset is accessing a remote URL, calling AVAssetReader:initWithAsset will result in a crash.
AVCaptureSession seems good, but it works with the camera and not a video file (much less a remote one).
As such, I am now exploring this: Display the remote video in an AVPlayerLayer, and then use GL ES to access what is displayed.
Questions:
How do I convert AVPlayerLayer (or a CALayer in general) to a CAEAGLLayer and read in the pixels using CVOpenGLESTextureCacheCreateTextureFromImage()?
Or is there some other better way?
Note: Performance is an important consideration, otherwise a simple screen capture technique would suffice.
As far as I know, Apple does not provide direct access to the h.264 decoder and there is no way around that. One API you can use is the asset interface, where you give it a URL and then that file on disk is read as CoreVideo pixel buffers. What you could try would be to download from your URL and then write a new asset (a file in the tmp dir) one video frame at a time. Then, once the download was completed and the new h264 file was fully written, close the writing session and then open the file as an asset reader. You would not be able to do streaming with this approach, the entire file would need to be downloaded first. Otherwise, you could try the AVPlayerLayer approach to see if that supports streaming directly. Be aware that the texture cache logic is not easy to implement, you need and OpenGL view already configured properly you would be better off just looking at an existing implementation that already does the rendering instead of trying to start from scratch.
This is now possible on modern iOS. If you're able to represent your real-time processing with Core Image—and you should be able to given Core Image's extensive support for custom filters nowadays—you can make use of AVAsynchronousCIImageFilteringRequest to pass into an AVPlayerItem per the documentation.
If you'd rather process things totally manually, you can check out AVPlayerItemVideoOutput and CVMetalTextureCache. With these, you can read sample buffers directly from a video and convert them into Metal textures from a texture buffer pool. From there, you can do whatever you want with the textures. Note with this approach, you are responsible for displaying the resultant textures (inside your own Metal or SceneKit rendering pipeline).
Here's a blog post demonstrating this technique.
Alternately, if you'd rather not manage your own render pipeline, you can still use AVPlayerItemVideoOutput to grab sample buffers, process them with something like vImage and Core Image (ideally using a basic Metal-backed CIContext for maximum performance!), and send them to AVSampleBufferDisplayLayer to display directly in a layer tree. That way you can process the frames to your liking and still let AVFoundation manage the display of the layer.
I am decoding a H264 stream using FFmpeg on the iPhone. I know the H264 stream is valid and the SPS/PPS are correct as VLC, Quicktime, Flash all decode the stream properly. The issue I am having on the iPhone is best shown by this picture.
It is as if the motion vectors are being drawn. This picture was snapped while there was a lot of motion in the image. If the scene is static then there are dots in the corners. This always occurs with predictive frames. The blocky colors are also an issue.
I have tried various build settings for FFmpeg such as turning off optimizations, asm, neon, and many other combinations. Nothing seems to alter the behavior of the decoder. I have also tried the Works with HTML, Love and Peace releases, and also the latest GIT sources. Is there maybe a setting I am missing, or maybe I have inadvertently enabled some debug setting in the decoder.
Edit
I am using sws_scale to convert the image to RGBA. I have tried various different pixel formats with the same results.
sws_scale(convertCtx, (const uint8_t**)srcFrame->data, srcFrame->linesize, 0, codecCtx->height, dstFrame->data, dstFrame->linesize);
I am using PIX_FMT_YUV420P as the source format when setting up my codec context.
What you're looking at is ffmpeg's motion vector visualization. Make sure that none of the following debug flags are set:
avctx->debug & FF_DEBUG_VIS_QP
avctx->debug & FF_DEBUG_VIS_MB_TYPE
avctx->debug_mv
Also, keep in mind that decoding H264 video using the CPU will be MUCH slower and less power-efficient on iOS than using the hardware decoder.
When Video is made with the Sorenson CODEC... MPMoviePlayerController just plays Audio(and not the Video), Instead i want to show my custom error message at this point. How can i detect which CODEC is used by particular File programmatically ... ?
EDIT: I am not using Quick time in my code so that solution won't work
Thanks
Check this documentation to understand the Quicktime file format :
http://developer.apple.com/library/mac/documentation/QuickTime/QTFF/qtff.pdf
The field you are looking for is the "vfmt" code that is containing the video fourcc code (there is one for each video track in your file, so take care if your file is containing several video tracks). The fourcc codes for Sorenson codec are "SVQ1" and "SVQ3".
Now you'll have to write some code to parse the QT file to find the correct atom, extract the "vfmt" value and compare it to SVQ1/SVQ3 !
Apple is providing some classes to easily parse quicktime files, but it is only available on Mac OS, not on iOS !
I am trying to extract an image frame from a video taken from iPhone camera using ffmpeg. But it usually throws me a EXEC_BAD_ACCESS and the stacktrace is showing in another method calls that is never called (I know my code didn't call it)
I am using the ffmpeg built from the instruction of the iFrameExtractor website. If anybody do it successfully, please help me or if possible, send me some codes. I don't know why it crashes, although it works well on the simulator (which I manually import a video into the library). My guess is that ffmpeg cannot decode the iPhone video camera correctly.
I already tried to use all 3 sets of library files like arvm6, arvm7 and i386 but doesn't work. My iPhone is 3gs. My iphone sdk is 3.1.3
I think it is my fault in calling the VideoFrameExtractor. The example code doesn't work well. I have to change from videoExtractor.currentImage to [videoExtractor currentImage]
Why would you use ffmpeg? You can extract frames using the AVFoundation framework in iOS4. It's faster and easier to use.
Can you paste in your stack trace and possibly the code you are using to read the frames?