cool, thanks. worked on synchronization a bit more today. realized i might be able to ensure correct sync in the decoder by monitoring the difference between the audio and video timestamps of decoded frames. then theoretically encoding would self-generate monotonic timestamps. first problem i've run into is that sometimes there are many audio frames before the next video frame, so the audio appears to get way ahead of the video even though the difference will be made up in the next few video frames (wide interleaving i guess). not convinced this won't work yet but i'll have to change my approach. maybe after decoding one second of a/v frames, check the average a/v timestamp difference and adjust. btw the way i'm adjusting right now is to either drop audio frames or generate silent ones. think i talked earlier about how dropping or inserting video frames is usually inappropriate for sda video.
SDA Forum
What happens if you throw VFR into the mix? Or are we not doing that. 
By the way the way..
or lol I misread that. nevermind
Isn't that what AviSynth does basically (in alignedsplice)? Makes sense. And definitely don't mess with the video frames.
Quote from nate:
btw the way i'm adjusting right now is to either drop audio frames or generate silent ones.
By the way the way..
Isn't that what AviSynth does basically (in alignedsplice)? Makes sense. And definitely don't mess with the video frames.
vfr input is a bit less mysterious to me than when i first started learning about this stuff. basically you have a base framerate and then vfr is defined as any input that doesn't have monotonically rising presentation timestamps on frames. a lot of the wacky vfr files i've seen are very nearly cfr, probably generated by software that uses the wallclock time it received the frame rather than assuming cfr and dropping/inserting. in that case, under the system i'm trying to make work right now, yua would whack the audio whenever there need to be dropped or inserted frames to make it match the video according to the timestamps on the decoded input frames within a certain tolerance (in milliseconds). i guess if i nail this then vfr input would come out with correct sync but maybe the audio won't sound so good depending on how much variance there was from cfr.
of course i can theoretically preserve all the timestamp information for identical output sync but that takes control away from me for functionality like appending videos with differing framerates. definitely need to think about this more.
of course i can theoretically preserve all the timestamp information for identical output sync but that takes control away from me for functionality like appending videos with differing framerates. definitely need to think about this more.
I honestly think you are opening a big can of worms here, nate. For every legitimate encoder out there, there are probably going to be 2 broken ones.
No matter how we do, getting sync for every file is probably going to be impossible due to thousands of different combinations. It's not as if there is a spec defining exactly how it must be. Even if there were, broken encoders would just ignore it anyway.
But maybe you can try a hybrid approach:
- Automatic (your current approach),
- No sync,
- Manual (the user somehow specifies the sync information him/herself).
You would probably need to produce some test files for the user to verify sync.
Well, I suppose we could check how many "bad files" we encounter before trying a hybrid approach.
But do we often have to append videos with different framerates? Is it worth trying to spend all this extra time on a feature that will rarely be needed (if it's rare, of course)?
No matter how we do, getting sync for every file is probably going to be impossible due to thousands of different combinations. It's not as if there is a spec defining exactly how it must be. Even if there were, broken encoders would just ignore it anyway.
But maybe you can try a hybrid approach:
- Automatic (your current approach),
- No sync,
- Manual (the user somehow specifies the sync information him/herself).
You would probably need to produce some test files for the user to verify sync.
Well, I suppose we could check how many "bad files" we encounter before trying a hybrid approach.
But do we often have to append videos with different framerates? Is it worth trying to spend all this extra time on a feature that will rarely be needed (if it's rare, of course)?
yeah it's not clear to me what the "right way" is yet. it's possible there's no right way. but i think that if the user supplies one or more files all with correct sync as input then the user expects correct sync on output and there's no reason i shouldn't be able to provide that. basically right now i'm doing something i usually do when i encounter what seems to be a difficult problem which is i start with what appears to be the simplest/most naive solution and then work my way down to the "right way" as necessary. if doing it on decode doesn't end up working for some reason then i can write a full abstraction system under which the input video timestamps are converted to a representation in the least common denominator timebase of all input files.
yeah, i have thought about this. it seems like an optimization with pitfalls. some people (those with perfect pitch probably included) may prefer a single break in the audio over audio with incorrect pitch for a longer period of time. i also don't know yet how commonly yua will actually have to adjust the synchronization of the input. it seems like input that plays back with correct sync shouldn't need adjustment (because i have access to both the audio and video presentation timestamps and i am writing audio and video pts's), and this covers the vast majority of cases.
Quote from DJGrenola:
you could use a timestretching algorithm to make chunks of audio fit the time available for them without changing their pitch.
yeah, i have thought about this. it seems like an optimization with pitfalls. some people (those with perfect pitch probably included) may prefer a single break in the audio over audio with incorrect pitch for a longer period of time. i also don't know yet how commonly yua will actually have to adjust the synchronization of the input. it seems like input that plays back with correct sync shouldn't need adjustment (because i have access to both the audio and video presentation timestamps and i am writing audio and video pts's), and this covers the vast majority of cases.
D:
I'm seeing what looks like poorer resizing from yua than from anri. The text in the yua encode is kind of smeared, and I think the video in general looks blockier. Perhaps yua disapproves of my choice of game.
I'm telling yua the video is interlaced/top field first/standard/D4/F1/2D. I'm giving anri the equivalent inputs. I've attached the HQ encodes from both, as well as the original clip (which I hope was okay to upload here despite it being big-ish).
Not sure if this is expected (I'll admit to having skimmed most of the thread and not properly read it). Worst case, deinterlacing and resizing in vdub before encoding is still easier than typing stuff in anri, especially since I already use vdub to cut out the part of the recording I want.
I'm telling yua the video is interlaced/top field first/standard/D4/F1/2D. I'm giving anri the equivalent inputs. I've attached the HQ encodes from both, as well as the original clip (which I hope was okay to upload here despite it being big-ish).
Not sure if this is expected (I'll admit to having skimmed most of the thread and not properly read it). Worst case, deinterlacing and resizing in vdub before encoding is still easier than typing stuff in anri, especially since I already use vdub to cut out the part of the recording I want.
yeah, this is probably because i still need to implement a lanczos (anri-style) horizontal resize for d4. it's on my todo. now i have a test clip which is great.
spent most of the day rewriting video encoding to use ffmpeg instead of libx264 by itself so i could write to a format that supports timestamps. finally got it working just now. turns out the stream time base is format-dependent. mp4 seems to require the numerator to be 1. so after you open the file it magically changes to something else. once i figured that out i was able to finish translating input timestamps to output regardless of interlacing or f.
for audio i'm already writing to a format that supports timestamps. there's a different challenge for audio which is that the output frame size is not the same as the input frame size. right now i just roll the resampled/converted audio data into a giant buffer that i read frame-sized chunks out of. instead i probably need to save an array of frames and their timestamps and then use those as a basis for calculating new ones as necessary. luckily resampling/converting doesn't change playtime so this should be possible without too much effort assuming i can figure out how to calculate the new audio timestamps. it took me a long time to figure out how to do that for the video.
spent most of the day rewriting video encoding to use ffmpeg instead of libx264 by itself so i could write to a format that supports timestamps. finally got it working just now. turns out the stream time base is format-dependent. mp4 seems to require the numerator to be 1. so after you open the file it magically changes to something else. once i figured that out i was able to finish translating input timestamps to output regardless of interlacing or f.
for audio i'm already writing to a format that supports timestamps. there's a different challenge for audio which is that the output frame size is not the same as the input frame size. right now i just roll the resampled/converted audio data into a giant buffer that i read frame-sized chunks out of. instead i probably need to save an array of frames and their timestamps and then use those as a basis for calculating new ones as necessary. luckily resampling/converting doesn't change playtime so this should be possible without too much effort assuming i can figure out how to calculate the new audio timestamps. it took me a long time to figure out how to do that for the video.
Quote from nate:
unsupported: with a vengeance.
I use AppGeeker video converter, handles just about every format, and has all the codecs already built in.
http://www.ilikemall.com/how-to/convert-avi-to-mp4-mac.html
nearing release of alpha 4. i finished synchronization before a week-long business trip. got stuck on x264 getting incorrect bitrate information making sane encoding impossible. figured out last night that the ffmpeg interface to x264 expects the AVFrame.pts value to be in the codec's timebase rather than the stream's as is customary. so now i ensure that the input AVFrame's pts is in the codec's timebase and then, after encoding, rescale the encoded AVPacket's pts and dts values from the codec's timebase to the stream's timebase before calling av_interleaved_write_frame():
seems to work well. again, i needed to rewrite encoding to use ffmpeg rather than libx264 directly because i needed to output to mp4 in a clean way. mp4box doesn't handle timestamps in other containers very well. this should also make it easier to change the video codec if it becomes necessary. h.265 or something.
today i finished statid integration with the new synchronization code. i seem to have sync with my few test files here, but i will need to run more tests. i figure that releasing alpha 4 is a good way to get help with that.
unfortunately i discovered a major problem with yua tonight: nothing it produces is compatible with my iphone 4s. it's bizarre because i can often see a thumbnail of the video in the videos app, but then i get "the operation could not be completed" when i try to play it. it's even more baffling because everything works fine in quicktime 7. actually, while testing, i enabled b-pyramid, and even that was compatible with quicktime 7, so it seems that testing in quicktime 7 is not valuable. there is some difference between what anri produces (even hq is compatible with my iphone) and what yua produces (even mq is not compatible), and i don't know what it is. i'm out of time for today, but maybe snow or someone can help bounce ideas off the thread. some data:
note that anri hq is compatible with my iphone, so one would assume that yua hq would be as well. i'm baffled because setup of x264 is so simple for hq. here are the anri options passed to x264 on the commandline and the equivalent lines from yua:
it's not just you. looks like nnedi3 in yua doesn't like weird widths. or maybe it's the height. it's on my todo.
Code:
pkt.pts = av_rescale_q(pkt.pts, st->codec->time_base, st->time_base); pkt.dts = av_rescale_q(pkt.dts, st->codec->time_base, st->time_base);
seems to work well. again, i needed to rewrite encoding to use ffmpeg rather than libx264 directly because i needed to output to mp4 in a clean way. mp4box doesn't handle timestamps in other containers very well. this should also make it easier to change the video codec if it becomes necessary. h.265 or something.
today i finished statid integration with the new synchronization code. i seem to have sync with my few test files here, but i will need to run more tests. i figure that releasing alpha 4 is a good way to get help with that.
unfortunately i discovered a major problem with yua tonight: nothing it produces is compatible with my iphone 4s. it's bizarre because i can often see a thumbnail of the video in the videos app, but then i get "the operation could not be completed" when i try to play it. it's even more baffling because everything works fine in quicktime 7. actually, while testing, i enabled b-pyramid, and even that was compatible with quicktime 7, so it seems that testing in quicktime 7 is not valuable. there is some difference between what anri produces (even hq is compatible with my iphone) and what yua produces (even mq is not compatible), and i don't know what it is. i'm out of time for today, but maybe snow or someone can help bounce ideas off the thread. some data:
Code:
[libx264 @ 0x105213400] using SAR=1/1 [libx264 @ 0x105213400] using cpu capabilities: MMX2 SSE2Fast SSSE3 FastShuffle SSE4.2 [libx264 @ 0x105213400] profile High, level 2.2 [libx264 @ 0x105213400] 264 - core 129 r2 bc13772 - H.264/MPEG-4 AVC codec - Copyleft 2003-2013 - http://www.videolan.org/x264.html - options: cabac=1 ref=16 deblock=1:0:0 analyse=0x3:0x133 me=umh subme=10 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=24 chroma_me=1 trellis=2 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 lookahead_threads=1 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=8 b_pyramid=0 b_adapt=2 b_bias=0 direct=3 weightb=1 open_gop=0 weightp=2 keyint=250 keyint_min=25 scenecut=40 intra_refresh=0 rc_lookahead=60 rc=crf mbtree=1 crf=17.0 qcomp=0.60 qpmin=0 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00 [libx264 @ 0x105213400] frame I:3 Avg QP:18.64 size: 17417 [libx264 @ 0x105213400] frame P:221 Avg QP:20.92 size: 3196 [libx264 @ 0x105213400] frame B:494 Avg QP:27.69 size: 495 [libx264 @ 0x105213400] consecutive B-frames: 0.7% 21.4% 28.4% 15.0% 29.2% 0.0% 3.9% 0.0% 1.3% [libx264 @ 0x105213400] mb I I16..4: 17.4% 30.7% 51.9% [libx264 @ 0x105213400] mb P I16..4: 0.5% 0.2% 0.3% P16..4: 41.0% 12.9% 17.6% 1.3% 0.6% skip:25.7% [libx264 @ 0x105213400] mb B I16..4: 0.0% 0.0% 0.0% B16..8: 33.9% 7.8% 2.9% direct: 1.6% skip:53.7% L0:53.8% L1:38.7% BI: 7.5% [libx264 @ 0x105213400] 8x8 transform intra:26.4% inter:31.8% [libx264 @ 0x105213400] direct mvs spatial:99.2% temporal:0.8% [libx264 @ 0x105213400] coded y,uvDC,uvAC intra: 51.7% 77.1% 71.4% inter: 8.1% 20.4% 16.4% [libx264 @ 0x105213400] i16 v,h,dc,p: 37% 56% 3% 3% [libx264 @ 0x105213400] i8 v,h,dc,ddl,ddr,vr,hd,vl,hu: 8% 25% 20% 4% 5% 3% 10% 4% 21% [libx264 @ 0x105213400] i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 12% 15% 13% 7% 9% 7% 12% 8% 17% [libx264 @ 0x105213400] i8c dc,h,v,p: 27% 46% 21% 6% [libx264 @ 0x105213400] Weighted P-Frames: Y:0.0% UV:0.0% [libx264 @ 0x105213400] ref P L0: 43.3% 4.5% 21.2% 9.2% 4.2% 3.4% 3.2% 1.5% 1.8% 1.2% 1.3% 1.1% 1.0% 1.1% 1.2% 0.7% [libx264 @ 0x105213400] ref B L0: 64.8% 16.5% 6.5% 2.7% 1.8% 1.6% 1.7% 0.6% 0.7% 0.5% 0.6% 0.5% 0.5% 0.5% 0.6% [libx264 @ 0x105213400] kb/s:669.91
note that anri hq is compatible with my iphone, so one would assume that yua hq would be as well. i'm baffled because setup of x264 is so simple for hq. here are the anri options passed to x264 on the commandline and the equivalent lines from yua:
| anri | yua |
| --preset veryslow | av_dict_set(&opts, "preset", "veryslow", 0); |
| --crf 17 | av_dict_set(&opts, "crf", "17", 0); |
| --pass 1 | c->flags |= CODEC_FLAG_PASS1; |
| --slow-firstpass | av_dict_set(&opts, "fastfirstpass", "0", 0); |
| --no-interlaced | n/a; no equivalent? |
| --b-pyramid 0 | av_dict_set(&opts, "b-pyramid", "none", 0); |
Quote from DJS:
is it just me or does Yua (alpha 3) crash when i open this video, and set it to interlaced, top field, d1, f1?
it's not just you. looks like nnedi3 in yua doesn't like weird widths. or maybe it's the height. it's on my todo.
Edit history:
ballofsnow: 2013-03-17 10:48:12 am
ballofsnow: 2013-03-17 10:19:16 am
ballofsnow: 2013-03-17 10:19:02 am
I recall having problems with preset veryslow and its default ref 16. I suggest specifying a value of 8. 16 is a bit overkill imo.
About --no-interlaced, it was only added because later versions of x264 was interpreting input as interlaced and would separate the fields! If you don't have this problem, don't worry about this command.
About --no-interlaced, it was only added because later versions of x264 was interpreting input as interlaced and would separate the fields! If you don't have this problem, don't worry about this command.
released alpha 4. links in the first post. please try it out and let me know if you can make it encode video with desync when the original isn't desynced (encoding both with and without statid). if you can or anything else weird happens i would love to get a copy of the input video. thanks.
two minor changes this time not related to synchronization. first i moved to lanczos resizing with libswscale every place i could find in the code, so no more shitty horizontal scaling hopefully (thanks sdfg). second i started to break the dependency on a statid image with special dpi metadata so people can eventually specify their own statid image. unfortunately that means statids will look a little weird under windows right now.
two minor changes this time not related to synchronization. first i moved to lanczos resizing with libswscale every place i could find in the code, so no more shitty horizontal scaling hopefully (thanks sdfg). second i started to break the dependency on a statid image with special dpi metadata so people can eventually specify their own statid image. unfortunately that means statids will look a little weird under windows right now.
The Great Farming Empire
Just tried out yua for HQ, MQ & LQ encodes. What is immediately apparent is that there is an audio desync in all of them. Doing another test, It seems to be caused by the StatID. I have a feeling that yua isn't matching the audio with the cut video when the StatID is inserted. I guess that it's not implemented yet? :S yua in general is still rather new to me, especially in terms of discussion since I haven't been reading a lot of the earlier posts.
One other note. Encoding these were really fast for me! I think it took only 4-5 minutes to encode all three qualities. So this is definitely much faster for me at least than Anri.
Also, ignore the big fraps logo on the top...I don't have the full version. >_<
One other note. Encoding these were really fast for me! I think it took only 4-5 minutes to encode all three qualities. So this is definitely much faster for me at least than Anri.
Also, ignore the big fraps logo on the top...I don't have the full version. >_<