For the playback of VideoPlayer, there has always been a problem that the audio and video will not be synchronized. The audio track will be later than the video. The degree of the lagging is different on different devices. I have tried to use the VP8 compression format. Some device have improved but some others cannot playback or still lag, the biggest disadvantage is that the file will be fat when packing the AssetBundle files.
I wonder is anyone have the same (or similar) problem or have a solution?
BTW, I dont know how to do if the Audio Output Mode of VideoPlayer selects API Only? I can't find any related usage at present.
My environmental information is:
Unity version:
{Unity2018.4.6f1}
PlayerSettings:
{Platform = Android(min API Level= 16), AutoGraphicsAPI = true, MultithreadedRendering = false}
Project Settings:
{Audio.DefaultSpeakerMode = Stereo, Audio.DSPBufferSize = Best latency, Audio.MaxVirtualVoices = 512, Audio.MaxRealVoices = 32}
VideoPlayer:
{Source = VideoClip, SkipOnDrop = true, AudioOutputMode = Direct}
The format of the video file is mp4:
Format: H.264, AAC, 44100Hz, stereo (LR)
Resolution: 1920x1080
Size: 1.1 MB
FrameRate: 30.14
Data transfer rate (BitRate): 1.23 Mbit / sec
I am not sure if this will help you, but there is a similar question from 4 years back.
Unity play video on Android
Related
I can use AVCaptureDevice as background.contents of SCNScene. And it works, but there is one problem with it. I would like to use video format that has resolution 1920x1080, and 60 FPS. But I can clearly see, that different format is used, it is 30 FPS. I am configuring used device before applying it to background, but somehow SCNScene is changing it. SceneView itself works in 60 FPS, but camera preview is a different story. Can I somehow force SCNScene to use video format I choose?
I know I could just add layer with camera preview under SceneView, but I have reasons why this approach is not working correctly, so I need to use this background property od the scene.
Sample project is here: https://www.dropbox.com/s/b820wxlg8voya58/SampleApp.zip?dl=1
In terminal you can clearly see, that after starting SceneView active format for device changes:
Selected format is:
<AVCaptureDeviceFormat: 0x282d58d00 'vide'/'420v' 1280x 720, { 3- 60 fps}, HRSI:4096x2304, fov:58.632, supports vis, max zoom:120.00 (upscales #2.91), AF System:2, ISO:23.0-736.0, SS:0.000013-0.333333>
2018-10-10 14:47:35.009890+0200 SampleApp[6799:1110910] [SceneKit] Error: Could not get pixel buffer (CVPixelBufferRef)
Format after 3 seconds is:
<AVCaptureDeviceFormat: 0x282d58fb0 'vide'/'420v' 1920x1080, { 3- 30 fps}, HRSI:4096x2304, fov:58.632, supports vis, max zoom:16.00 (upscales #1.94), AF System:2, ISO:23.0-736.0, SS:0.000013-0.333333>
I want to find a method to eliminate the repeating frames from a video. If I consider a video which will repeatedly show the same frame for 5 seconds, I want to include only one frame of that in the video and make it visible for 5 seconds. In here I am looking forward to minimize the file size by eliminating duplicate frames. Is there a method to do this using Matlab?
If your movie is just a series of stills that you wish to show as a slideshow/presentation with a fixed five second delay, then you should be able to used the 'FrameRate' property for the VideoWriter class. Try something like this example:
writerObj = VideoWriter('peaks.mp4','MPEG-4');
writerObj.FrameRate = 0.2; % One frame every 5 seconds
open(writerObj);
Z = peaks;
surf(Z);
for k = 1:4:20
surf(sin(2*pi*k/20)*Z,Z);
writeVideo(writerObj,getframe);
end
close(writerObj);
However, the frame-rate property cannot be varied over the course of your movie, so the more general form of your question is fundamentally an issue of encoder support for variable frame-rate encoding. Most modern encoders (e.g., H.264 implementations) are not designed to explicitly handle this, but rather have heuristics that can detect when content is not changing and efficiently encode the data (particularly if multi-pass encoding is used). Unfortunately, Matlab (I'm assuming that you've been using the VideoWriter class) doesn't really provide great deal of fidelity in this respect. I'm not even sure what inter-frame encoding settings are used for MPEG-4 with H.264 videos.
If the MPEG-4 with H.264 videos produced by VideoWriter are unacceptable, I'd recommend exporting your video in the highest quality possible (or lossless) and then learn to use a full-fledged encoding framework/library (ffmpeg, libav, x264) or application to encode to the quality and size you desire. Apparently Handbrake has support for variable frame-rate encoding, though it's not necessarily designed for what you may want I've not tested it. Or export your individual still frames and use actually video editing software (e.g. iMovie on OS X). There are also likely dedicated applications that can create a movie from a slideshow/presentation (both PowerPoint and Keynote can do this).
Within Matlab, another alternative, is to use a codec that explicitly supports variable frame rates – QuickTime's image-based codecs: Photo JPEG (not to be confused with Motion-JPEG), Photo PNG (a.k.a. Apple PNG), and Photo TIFF (a.k.a. Apple Tiff). You can encode content directly using these codecs with my QTWriter, which is available on Github. Note however, that on OS X 10.9+, QuickTime Player converts lossless variable frame-rate Photo PNG and Photo TIFF movies to lossy fixed frame-rate H.264 (Photo JPEG movies are not converted). See my note towards the bottom of this page for further details and a workaround.
When I try to upload videos captured from my iPhone in my app, the server performs a conversion from .mov to .mp4 so that it can be played in other platforms. However the problem is that when I shoot the video (in portrait orientation) and it is converted (using ffmpeg) and then played back from the server, it appears to be rotated. Any idea?
FFMPEG changed the default behavior to auto rotate video sources with rotation metadata in 2015. This was released as v2.7.
If your ffmpeg version is v2.7 or newer, but your rotation metadata isn't respected, the problem is likely that you are using custom rotation based on metadata. This will cause the same logic to be applied twice, changing or cancelling out the rotation.
In addition to removing your custom rotation (recommended), there's an option to turn off auto rotation with -noautorotate.
ffmpeg -noautorotate -i input.mp4...
This will also work in some older releases.
For sake of completeness, the reason this is happening is that iPhones only actually capture video in one fixed orientation. The measured orientation is then recorded in Apple-specific metadata.
The effect is that Quicktime Player reads the metadata and rotates the video to the correct orientation during playback, but other software (e.g., VLC) does not and shows it as oriented in the actual codec data.
This is why rotate=90 (or vflip, or transpose, or etc.) will work for some people, but not others. Depending on how the camera is held during recording, the rotation necessary could be 90, 180, or even 270 degrees. Without reading the metadata, you're just guessing at how much rotation is necessary and the change that fixes one video will fail for another.
What you can also do is remove the QuickTime specific metadata when rotate the .mov.
This will make sure that the video is rotated the same way in VLC and QuickTime
ffmpeg -i in.mov -vf "transpose=1" -metadata:s:v:0 rotate=0 out.mov
Here's the documentation on the -metadata option (from http://ffmpeg.org/ffmpeg.html):
-metadata[:metadata_specifier] key=value (output,per-metadata)
Set a metadata key/value pair.
An optional metadata_specifier may be given to set metadata on streams or chapters. See -map_metadata documentation for details.
This option overrides metadata set with -map_metadata. It is also possible to delete metadata by using an empty value.
For example, for setting the title in the output file:
ffmpeg -i in.avi -metadata title="my title" out.flv
To set the language of the first audio stream:
ffmpeg -i INPUT -metadata:s:a:1 language=eng OUTPUT
Depending on which version of ffmpeg you have and how it's compiled, one of the following should work...
ffmpeg -vf "transpose=1" -i input.mov output.mp4
...or...
ffmpeg -vfilters "rotate=90" -i input.mov output.mp4
Use the vflip filter
ffmpeg -i input.mov -vf "vflip" output.mp4
Rotate did not work for me and transpose=1 was rotating 90 degrees
So - I too ran into this issue, and here my $0.02 on it:
1.) some videos DO have Orientation/Rotation metadata, some don't:
MTS (sony AVHCD) or the AVIs I have - DO NOT have an orientation tag.
MOVs and MP4s (ipad/iphone or samsung galaxy note2) DO HAVE it.
you can check the setting via 'exiftool -Rotation file'.
My videos often have 90 or 180 as the rotation.
2.) ffmpeg - regardless of the man-page with the metadata-tag, just doesn't EVER seem to set it in the output file. - the rotation-tag is ALWAYS '0'.
it correctly reports it in the output - but it's never set right to be reported by exiftool. - But hey - at least it's there and always 0.
3.) rotation angles:
if you want rotate +/- 90: transpose=1 for clockwise 90, 2 ccw
now if you need 180 degree - just add this filter TWICE.
remember - it's a filter-chain you specify. :-) - see further down.
4.) rotate then scale:
this is tricky - because you quickly get into MP4 output format violations.
Let's say you have a 1920x1080 MOV.
rotate by 90 gives 1080x1920
then we rescale to -1:720 -> 1080*(720/1920) = 405 horiz
And 405 horizontal is NOT divisable by 2 - ERROR. fix this manually.
FIXING THIS automatically - requires a bit of shell-script work.
5.) scale then rotate:
you could do it this way - but then you end up with 720x1280. yuck.
But the filter-example here would be:
"-vf yadif=1,scale=-1:720,transpose=1"
It's just not what I want - but could work quite OK.
Putting it all together: - NOTE - 'intentionally WRONG Rotation-tag', just to demonstrate - it won't show up AT ALL in the output !
This will take the input - and rotate it by 180 degree, THEN RESCALE IT - resetting the rotation-tag. - typically iphone/ipad2 can create 180deg rotated material.
you just can leave '-metadata Rotation=x' out the line...
/usr/bin/ffmpeg -i input-movie.mov -timestamp 2012-06-23 08:58:10 -map_metadata 0:0 -metadata Rotation=270 -sws_flags lanczos -vcodec libx264 -x264opts me=umh -b 2600k -vf yadif=1,transpose=1,transpose=1,scale=1280:720 -f mp4 -y output-movie.MP4
I have multiple devices - like a settop box, ipad2, note2, and I convert ALL my input-material (regardless whether it's mp4,mov,MTS,AVI) to 720p mp4, and till now ALL the resulting videos play correct (orientation,sound) on every dev.
Hope it helps.
For including into web pages my portrait-format videos from iPhone, I just discovered the following recipe for getting .mp4 files in portrait display.
Step 1: In QuickTime Player, Export your file to 480p (I assume that 720p or 1080p would work as well). You get a .mov file again.
Step 2: Take the new file in QT Player, and export to “iPad, iPhone…”. You get a .m4v file.
Step 3: I’m using Miro Video Converter, but probably any readily-available converter at all will work, to get your .mp4 file.
Works like a (long-winded) charm.
I've filmed the video with Ipad3 and it was oriented upside down, which I suppose is the common situation of all Apple devices at some versions. Besides of it, the 3-minutes long MOV file (1920x1090) took about 500 Mb in size, which made it not available to share easily. I had to convert it to MP4, and analyzing all threads I've found on stackoverflow, here's the final code string for ffmpeg I've used (ffmpeg ver. 2.8.4):
ffmpeg -i IN.MOV -s 960x540 -metadata:s:v rotate="0" -acodec libmp3lame OUT.mp4
I suppose you may just leave '-metadata:s:v rotate="0"' if you don't need the resize and audio codec change. Note that if you resize the video, width and height should fully divide to 4.
Although the topic is old.
Hope this will help some one:
Get ffmpeg latest version : https://www.ffmpeg.org/download.html
The command that worked for me (to flip 180 degrees):
ffmpeg -noautorotate -i input.mp4 -filter:v "rotate=PI" output.mp4
When the degrees are determined by -filter:v "PI/180*degrees"
for example
-filter:v "45*PI/180" for 45 degrees
A nice explanation is here
https://superuser.com/questions/578321/how-to-rotate-a-video-180-with-ffmpeg
Or... to simply change the tag in an existing file:
Read the current rotation
exiftool -Rotation <file>
then, for example:
exiftool -Rotation=180 <file>
to set it to 180
I've recently been working on porting my game to be cross-platform, and decided to go with OpenAL for my cross-platform audio engine.
I have 16 "channels" (OpenAL sources) for playing up to 16 sounds concurrently. To play a sound, I switch which buffer is linked to a given source, and also set the gain, source position, and so on in order to play a sound in a given "channel" (source).
The problem is, I've noticed that my "gain" settings do not seem to have immediate effect. For instance, if a loud "lightning" sound plays in a given source at 0.5 gain, then when I have a button click sound play at 0.15 gain later, this click starts off FAR too loud. Then, each subsequent time it is played, the volume decreases until around the 3rd or 4th click it sounds like it's around the proper 0.15 gain.
The first button click is barely audible, and it seems to ramp up in volume until it reaches the 0.15 gain.
So in short, a "source" seems to be remembering the former gain settings, even though I am resetting those before playing a new sound in the same source! Is this a bug? Or something I don't understand about OpenAL? How can I get it to "instantly" change to the new gain/position settings?
Relevant code to play a sound:
[Channel is a value between 0 and 15, soundID is a valid index into the gBuffer array, stereoPosition is an integer between -255 and 255, and volume is between 0 and 255. This is from a function that's a wrapper for my game that used to use values between 0-255, so it converts the values to proper OpenAL values.]
// Stop any sound currently playing in this channel ("source")
alSourceStop( gSource[channel] );
// What sound effect ("buffer") is currently linked with this channel? (Even if not currently playing)
alGetSourcei( gSource[channel], AL_BUFFER, &curBuffer );
// attach a different buffer (sound effect) to the source (channel) only if it's different than the previously-attached one.
// (Avoid error code by changing it only if it's different)
if (curBuffer != gBuffer[soundID])
alSourcei( gSource[channel], AL_BUFFER, gBuffer[soundID] );
// Loop the sound?
alSourcei( gSource[channel], AL_LOOPING, (loopType == kLoopForever) );
// For OpenAL, we do this BEFORE starting the sound effect, to avoid sudden changes a split second after it starts!
volume = (volume / 2) + 1; // Convert from 0-255 to 0-128
{
float sourcePos[3] = {(float)stereoPosition / 50, 0.0, 2.0};
// Set Source Position
alSourcefv( gSource[channelNum], AL_POSITION, sourcePos ); // fv = float vector
// Set source volume
alSourcef( gSource[channelNum], AL_GAIN, (float)newVolume / 255 );
}
// Start playing the sound!
alSourcePlay( gSource[channel] );
I can post setup code too if desired, but nothing fancy there. Just calling
alSourcef( gSource[n], AL_REFERENCE_DISTANCE, 5.0f );
for each source.
We just faced the same problem when setting the ByteOffset for seeking to a position in a sample and found now the solution to get it working:
Just always delete and recreate the source before setting any parameter on it.
So if you want to change the gain and/or other parameters:
OpenAL.AL.DeleteSource(oldSourceID);
newSourceId = OpenAL.AL.GenSource();
OpenAL.AL.Source(newSourceId , OpenTK.Audio.OpenAL.ALSourcef.Gain, yourVolume);
OpenAL.AL.Source(newSourceId , OpenTK.Audio.OpenAL.ALSourcef.xxx, yourOtherParameter);
Hope it works for you and finally have a workaround after 10 years :-)
These are the values I pass in, it's the only combination of values I have got working.
dataFormat.mSampleRate = 44100;
dataFormat.mFormatID = kAudioFormatLinearPCM;
dataFormat.mFormatFlags = kAudioFormatFlagIsSignedInteger | kLinearPCMFormatFlagIsBigEndian;
dataFormat.mBytesPerPacket = 4;
dataFormat.mFramesPerPacket = 1;
dataFormat.mBytesPerFrame = 4;
dataFormat.mChannelsPerFrame = 2;
dataFormat.mBitsPerChannel = 16;
status = AudioQueueNewInput( &dataFormat, AudioInputCallback, self, NULL, NULL, 0,
&queue);
status = AudioFileCreateWithURL(fileUrl, kAudioFileCAFType, &dataformat, kAudioFileFlags_EraseFile, &audioFile
The recording works, but it is a lot of noise during the recording, and on the playback. Can it have anything to do with this code?
I can see two possible errors. First, as #invalidname pointed out, recording in stereo probably isn't going to work on a mono device such as the iPhone. Well, it might work, but if it does, you're just going to get back dual-mono stereo streams anyways, so why bother? You might as well configure your stream to work in mono and spare yourself the CPU overhead.
The second problem is probably the source of your sound distortion. Your stream description format flags should be:
kAudioFormatFlagIsSignedInteger |
kAudioFormatFlagsNativeEndian |
kAudioFormatFlagIsPacked
Also, don't forget to set the mReserved flag to 0. While the value of this flag is probably being ignored, it doesn't hurt to explicitly set it to 0 just to make sure.
Edit: Another more general tip for debugging audio on the iPhone -- if you are getting distortion, clipping, or other weird effects, grab the data payload from your phone and look at the recording in a wave editor. Being able to zoom down and look at the individual samples will give you a lot of clues about what's going wrong.
To do this, you need to open up the "Organizer" window, click on your phone, and then expand the little arrow next to your application (in the same place where you would normally uninstall it). Now you will see a little downward pointing arrow, and if you click it, Xcode will copy the data payload from your app to somewhere on your hard drive. If you are dumping your recordings to disk, you'll find the files extracted here.
What's your input device? The mic on the provided earbuds or the phone's built-in mic or what? Or are you recording into the Simulator?
Aside from the noise, does everything else sound right: speed, pitch, etc.?
It probably isn't causing any problems, but you're specifying two-channel input, while your input device is probably mono.
One last thought: is this a first generation iPhone? I think there's a weird issue with that model where 8 KHz input gets upconverted to 44.1.