help sync the marathon videos

2011-01-19 04:53:44 pm

Eh, right, but they did record the games in their original resolution, I gather? So then it should be possible to access those. Last year's marathon downloads were awfully low quality. I don't want to see such things again, if it can be helped.

Aktan

Edit history:

Aktan: 2011-01-19 05:09:38 pm

2011-01-19 04:56:58 pm

Quote from nate:

from what i understand, avisource() uses vfw, which is all 32-bit, so no audio longer than four hours or whatever since it tries to serve the audio as wav. actually directshowsource() seems to have the same problem except that you actually get audio with it if you trim the output of the script down to less than four hours or whatever using trim() or similar. right now i'm having no problem whatsoever opening the transcoded avis in vdub not through avisynth (which is interesting because vdub is all vfw and the audio is all there). no problems with av sync or anything. it's the avisynth requirement that brings everything down. it seems like i'm just going to have to cut each run out of the masters in order to get the length down to where people can use avisource(). obviously from the above post and others like it, this is not going to work as-is.

The limit for the length of a WAV is actually a lot longer than 4 hours. I've calculated this before and I rememeber it being 10 hours+ at a sample rate of 44100. It is indeed limited to 2^32, aka 4 GB.

Edit: I recalculated and the amount of time for a 44100 audio with 2 channels is ~ 6.76 hours. My bad.

Quote from nate:

i'm uploading two samples right now - one commentary stream one and one game stream one. commentary stream is vp6/mp3 flv and game stream is h.264/mp3 f4v. feel free to play around with them, but i'm not sure we really stand to gain anything right now from those files (unless you can figure out how to get the video beyond the format change out of g.10). there's no way i'm writing a program to read that format when i can just ask mike to torrent me those runs from the dvd backup in vastly superior quality.

I have studied the FLV format a while back, and had some fun looking at it with an hex editor. I kind of meant that I am willing to write a program since I do understand how FLV works. I would like a sample of g.10 since it does have the problem after all.

[quote="nate"]from what i understand, avisource() uses vfw, which is all 32-bit, so no audio longer than four hours or whatever since it tries to serve the audio as wav. actually directshowsource() seems to have the same problem except that you actually get audio with it if you trim the output of the script down to less than four hours or whatever using trim() or similar. right now i'm having no problem whatsoever opening the transcoded avis in vdub not through avisynth (which is interesting because vdub is all vfw and the audio is all there). no problems with av sync or anything. it's the avisynth requirement that brings everything down. it seems like i'm just going to have to cut each run out of the masters in order to get the length down to where people can use avisource(). obviously from the above post and others like it, this is not going to work as-is.[/quote]

The limit for the length of a WAV is actually a lot longer than 4 hours.  I've calculated this before and I rememeber it being 10 hours+ at a sample rate of 44100.  It is indeed limited to 2^32, aka 4 GB.

Edit:  I recalculated and the amount of time for a 44100 audio with 2 channels is ~ 6.76 hours.  My bad.

[quote="nate"]i'm uploading two samples right now - one commentary stream one and one game stream one. commentary stream is vp6/mp3 flv and game stream is h.264/mp3 f4v. feel free to play around with them, but i'm not sure we really stand to gain anything right now from those files (unless you can figure out how to get the video beyond the format change out of g.10). there's no way i'm writing a program to read that format when i can just ask mike to torrent me those runs from the dvd backup in vastly superior quality.[/quote]

I have studied the FLV format a while back, and had some fun looking at it with an hex editor.  I kind of meant that I am willing to write a program since I do understand how FLV works.  I would like a sample of g.10 since it does have the problem after all.

Aktan

2011-01-19 04:58:28 pm

Quote from Mystery:

Eh, right, but they did record the games in their original resolution, I gather? So then it should be possible to access those. Last year's marathon downloads were awfully low quality. I don't want to see such things again, if it can be helped.

Mind you I could be completely wrong on all this, but I think the DVD Backups are indeed the native 720x480 resolutions. The Backups are hard to get though, hence why the streaming versions are used instead.

UraniumAnchor

2011-01-19 04:58:43 pm

Not a walrus

Quote from Mystery:

Eh, right, but they did record the games in their original resolution, I gather? So then it should be possible to access those. Last year's marathon downloads were awfully low quality. I don't want to see such things again, if it can be helped.

The amount of absolutely useless and/or wrong posts you continue to make never ceases to amaze me.

nate

2011-01-19 04:58:45 pm

Quote from Mystery:

Eh, right, but they did record the games in their original resolution, I gather? So then it should be possible to access those. Last year's marathon downloads were awfully low quality. I don't want to see such things again, if it can be helped.

i tried, but failed. if you want to you can ask mike and trihex (mike has the first half, trihex has the second) to burn everything and upload it somewhere but i doubt they will want to do that. better luck next year.

aktan: ok, i'm going to go ahead and send up all the original files after i send up the segmented avis tonight. that will also help soothe those who are dissatisfied with how i'm doing this.

Mystery

2011-01-19 05:04:55 pm

Quote from UraniumAnchor:

Quote from Mystery:

Eh, right, but they did record the games in their original resolution, I gather? So then it should be possible to access those. Last year's marathon downloads were awfully low quality. I don't want to see such things again, if it can be helped.

The amount of absolutely useless and/or wrong posts you continue to make never ceases to amaze me.

Wow. Your absolutely useless posts never cease to amaze me.
Perhaps you should stop being rude to other people or share your wisdom to the ignorant.

Quote from nate:

i tried, but failed. if you want to you can ask mike and trihex (mike has the first half, trihex has the second) to burn everything and upload it somewhere but i doubt they will want to do that. better luck next year.

Well, that's a shame. But I'll live with it. You did record them for free, after all. I missed 99% of the marathon, so I'm grateful for that, at least.

UraniumAnchor

2011-01-19 05:08:47 pm

Not a walrus

Quote from Mystery:

Wow. Your absolutely useless posts never cease to amaze me.
Perhaps you should stop being rude to other people or share your wisdom to the ignorant.

I spent eight hours last night trying to get g.10 into a usable state when I should have been sleeping.

All you've done is complain about 'oh why are you using avi' and 'last year's downloads were SOOOOO TERRIBLE'.

Fuck off unless you have something useful to contribute other than griping.

Mystery

Edit history:

Mystery: 2011-01-19 05:15:47 pm

2011-01-19 05:14:18 pm

You really take everything as a negative thing, don't you? Every little thing I post makes you go "BLOODY IDIOT! DIE!!!!!".
I did say last year's downloads were terrible, but I didn't complain about it. I only wished to see if we could get higher quality this year.
And I asked why you were using AVI because the container is outdated and causes problems, problems, problems. H264 is avi is just asking for trouble.

I have no idea how the streams were recorded, and it seems odd that you should get random desyncs and stuff. I've messed a lot with stuff like this.
So I did suggest something. Actually, I asked if there was a reason it wasn't done: why the videos haven't been compressed into a more friendly format in a proper container.
Perhaps that might stop the random errors. And using a more modern software tool that works with modern containers rather than the old virtualdub which only works with vfw--yet another extremely old technology that doesn't play well with modern stuff.

I have every bit of respect for the people who want to do this. I wouldn't want to do it myself. I've had my shares of desync, and it's a PITA.
So kudos to everyone who wants to attempt this.

You really take everything as a negative thing, don't you? Every little thing I post makes you go "BLOODY IDIOT! DIE!!!!!".
I did say last year's downloads were terrible, but I didn't complain about it. I only wished to see if we could get higher quality this year.
And I asked why you were using AVI because the container is outdated and causes problems, problems, problems. H264 is avi is just asking for trouble.

I have no idea how the streams were recorded, and it seems odd that you should get random desyncs and stuff. I've messed a lot with stuff like this.
So I did suggest something. Actually, I asked if there was a reason it wasn't done: why the videos haven't been compressed into a more friendly format in a proper container.
Perhaps that might stop the random errors. And using a more modern software tool that works with modern containers rather than the old virtualdub which only works with vfw--yet another extremely old technology that doesn't play well with modern stuff.

I have every bit of respect for the people who want to do this. I wouldn't want to do it myself. I've had my shares of desync, and it's a PITA.
So kudos to everyone who wants to attempt this.

nate

2011-01-19 05:14:32 pm

if you're curious, it was because i couldn't use my nice laptop to stream the game stream. that pushed everything back one machine and i ended up using mike's laptop to stream the game stream. actually that's also why the files are corrupt - we had "degrade quality" checked in fme on there thinking it was a good idea. we only had one wired internet connection and the commentary stream laptop was already using it, so his laptop was on wireless the whole time, and it seems now that every time the wireless went out, it would start varying the quality and that would be the end of the recorded game stream for that file (unless aktan or someone can pull it back).

so that's two lessons learned right there - don't use degrade quality and don't use wireless no matter what. i mean the latter sounds obvious enough but i wasn't really at my best while i was there and it involved dealing with the management and stuff so i was just like eh whatever, seems to be working well enough.

the original plan was to use what ended up being the commentary stream laptop to record both streams to two 2 tb hard drives i bought just for the marathon and ended up never using. it sort of slowly dawned on me that i couldn't record the commentary stream at all outside of fme since it's all digital and i have no idea how to split usb audio like one splits analog audio. and there was no laptop or capture device available to record the game stream. i had a whole day to set this thing up and i used almost every minute right up to when the marathon started. it was that bad. none of the capture devices we had worked in vdub with the drivers we could find. so that's one of the reasons i'm considering just buying a whole bunch of blackmagic cards or whatever and putting them in a new pc i buy for next year's. no more usb capture devices.

i guess i need to be saying these things in here because only two people in the world know most of them and lag's not posting in here atm.

if you're curious, it was because i couldn't use my nice laptop to stream the game stream. that pushed everything back one machine and i ended up using mike's laptop to stream the game stream. actually that's also why the files are corrupt - we had "degrade quality" checked in fme on there thinking it was a good idea. we only had one wired internet connection and the commentary stream laptop was already using it, so his laptop was on wireless the whole time, and it seems now that every time the wireless went out, it would start varying the quality and that would be the end of the recorded game stream for that file (unless aktan or someone can pull it back).

so that's two lessons learned right there - don't use degrade quality and don't use wireless no matter what. i mean the latter sounds obvious enough but i wasn't really at my best while i was there and it involved dealing with the management and stuff so i was just like eh whatever, seems to be working well enough.

the original plan was to use what ended up being the commentary stream laptop to record both streams to two 2 tb hard drives i bought just for the marathon and ended up never using. it sort of slowly dawned on me that i couldn't record the commentary stream at all outside of fme since it's all digital and i have no idea how to split usb audio like one splits analog audio. and there was no laptop or capture device available to record the game stream. i had a whole day to set this thing up and i used almost every minute right up to when the marathon started. it was that bad. none of the capture devices we had worked in vdub with the drivers we could find. so that's one of the reasons i'm considering just buying a whole bunch of blackmagic cards or whatever and putting them in a new pc i buy for next year's. no more usb capture devices.

i guess i need to be saying these things in here because only two people in the world know most of them and lag's not posting in here atm.

nate

2011-01-19 05:17:48 pm

more detail on why xvid avi is that i don't know of any more friendly format for avisynth. it also has to do with how the files were originally very long. really really really hard to get audio in avisynth after that 10 hour limit or whatever it is. a lot of this is psychological on all of our parts - one of my failures was to think i could get around having to segment the runs out. and some of that is from thinking well since i achieved av sync in the nmf i don't want to segment because that means more sync work. but now that people are helping me sync it doesn't really matter. sync in this case means game/commentary stream.

Mystery

2011-01-19 05:22:39 pm

Well wow. That just sucks.
All I can say is thanks for still attempting to repair this mess and let's hope it turns out better next time.
You have my sympathies.

UraniumAnchor

Edit history:

UraniumAnchor: 2011-01-19 05:28:37 pm

2011-01-19 05:26:12 pm

Not a walrus

Quote from Mystery:

You really take everything as a negative thing, don't you? Every little thing I post makes you go "BLOODY IDIOT! DIE!!!!!".
I did say last year's downloads were terrible, but I didn't complain about it. I only wished to see if we could get higher quality this year.
And I asked why you were using AVI because the container is outdated and causes problems, problems, problems. H264 is avi is just asking for trouble.

It's because you continue to second guess every last thing anybody does and you manage to do in a way that comes across as massively condescending. Of course avi isn't the final format, but it plays nice with just about every tool out there (most importantly avisynth) which makes it an excellent intermediate step. xvid at quantizer 1 is extremely close to being lossless (barring colorspace conversions) and works with any tool made in the last decade. mp4 is a wonderful format for the final encode, but many useful processing tools (again, most importantly, avisynth) still have issues dealing with it.

[quote="Mystery"]You really take everything as a negative thing, don't you? Every little thing I post makes you go "BLOODY IDIOT! DIE!!!!!".
I did say last year's downloads were terrible, but I didn't complain about it. I only wished to see if we could get higher quality this year.
And I asked why you were using AVI because the container is outdated and causes problems, problems, problems. H264 is avi is just asking for trouble.
[/quote]

It's because you continue to second guess every last thing anybody does and you manage to do in a way that comes across as massively condescending. Of course avi isn't the final format, but it plays nice with just about every tool out there (most importantly avisynth) which makes it an excellent intermediate step. xvid at quantizer 1 is extremely close to being lossless (barring colorspace conversions) and works with any tool made in the last decade. mp4 is a wonderful format for the final encode, but many useful processing tools (again, most importantly, avisynth) still have issues dealing with it.

nate

2011-01-19 05:29:56 pm

basically it just seems like you're more emotionally than rationally motivated (you see avi and go ew) and it bothers people. over the years i've sort of realized that a lot of stuff i see has some kind of rationale behind it even when it at first seems shocking. so you see me using avi and it's like is he really a philistine or does he have some reason for that? i realize i'm also at fault here for not reporting everything that led up to this but it's also helpful to ask rather than to assume that the emotional response is valid. there just isn't very good support for long files in avisynth and for a long time i thought that was something to overcome but now i've sort of realized that with a bunch of people helping me sync the streams that the proper response is to just segment the marathon initially and defeat it that way.

nate

2011-01-19 05:31:12 pm

ideally the initial format never would have been weird shit in the first place and we would never have had this conversation. better luck next year.

Mystery

Edit history:

Mystery: 2011-01-19 05:34:48 pm

2011-01-19 05:34:34 pm

Quote:

Of course avi isn't the final format, but it plays nice with just about every tool out there (most importantly avisynth) which makes it an excellent intermediate step. xvid at quantizer 1 is extremely close to being lossless (barring colorspace conversions) and works with any tool made in the last decade. mp4 is a wonderful format for the final encode, but many useful processing tools (again, most importantly, avisynth) still have issues dealing with it.

Oh? You have experience in this then?
Can you tell me exactly what "issues" avisynth has with mp4? Directshowsource is a known problem with many things. Is there anything else you can bring to my attention?

nate

2011-01-19 05:35:31 pm

do you regularly use the modern containers with avisynth? if so is that with ffmpegsource() and friends? i guess directshowsource() no one really uses any more.

UraniumAnchor

2011-01-19 05:36:56 pm

Not a walrus

Considering I put together last year's highlight/blooper/sleep cam videos from mp4 source, yes, I do know.

And let me tell you, getting the audio sync to work properly was a massive pain in the ass. ffmpegsource() would randomly tell me there was no audio stream. And by 'random' I mean 'I just refreshed the script and suddenly there's no audio'.

Aktan

2011-01-19 05:41:07 pm

it is a pain with other containers besides AVI really. In general H.264 in Avisynth is a pain. For MKV/MP4 you do have the option of also dss2 (DirectShowSource2) from haali. But that doesn't have any audio. Maybe Audio on the outside is a better idea. I'll have to see when I get my hands on these files.

Mystery

Edit history:

Mystery: 2011-01-19 05:43:03 pm

2011-01-19 05:41:43 pm

Quote from nate:

do you regularly use the modern containers with avisynth? if so is that with ffmpegsource() and friends? i guess directshowsource() no one really uses any more.

Yes, I do exclusively work with mp4 and mkv. From my I know, directshowsource is slow and doesn't support vfr, so I'd say ffmpegsource is a good thing. I've had good experiences with it.
That said, I don't really do a lot of writing avisynth scripts. It's usually for converting fps.

Quote from UraniumAnchor:

Considering I put together last year's highlight/blooper/sleep cam videos from mp4 source, yes, I do know.

And let me tell you, getting the audio sync to work properly was a massive pain in the ass. ffmpegsource() would randomly tell me there was no audio stream. And by 'random' I mean 'I just refreshed the script and suddenly there's no audio'.

That is weird. I haven't had such problems myself, though I usually just encode video, then mux original or encoded sound.
When delaying and doing other things to audio I typically do when muxing or with audacity.

And I have put together a big file of many clips with audio. That was a PITA, but I don't think I ever used avisynth to do it.

[quote="nate"]do you regularly use the modern containers with avisynth? if so is that with ffmpegsource() and friends? i guess directshowsource() no one really uses any more.[/quote]
Yes, I do exclusively work with mp4 and mkv. From my I know, directshowsource is slow and doesn't support vfr, so I'd say ffmpegsource is a good thing. I've had good experiences with it.
That said, I don't really do a lot of writing avisynth scripts. It's usually for converting fps.

[quote="UraniumAnchor"]Considering I put together last year's highlight/blooper/sleep cam videos from mp4 source, yes, I do know.

And let me tell you, getting the audio sync to work properly was a massive pain in the ass. ffmpegsource() would randomly tell me there was no audio stream. And by 'random' I mean 'I just refreshed the script and suddenly there's no audio'.[/quote]
That is weird. I haven't had such problems myself, though I usually just encode video, then mux original or encoded sound.
When delaying and doing other things to audio I typically do when muxing or with audacity.

And I have put together a big file of many clips with audio. That was a PITA, but I don't think I ever used avisynth to do it.

nate

2011-01-19 05:59:22 pm

lol ok so i just lost three and a half hours because i assumed that because i could direct stream copy parts of these xvid avis starting at the beginning that i could start in the middle and not get av desync. so i'm going to have to segment the marathon again basically, this time without direct stream copy. i'm still going to send the original flash files up tonight but only the segmented marathon files will go up that have av sync. going to check through to find those now.

nate

2011-01-19 06:01:12 pm

ok, false alarm, there's only desync via avisource(). opening the same file in vdub doesn't cause desync which means i should be able to reencode these to nmf (probably only the audio) real fast and send them up after all.

UraniumAnchor

2011-01-19 06:02:11 pm

Not a walrus

I'm attempting to recover g.38 (the NIDL race) right now. FMS is twitchy but ffmpeg is able to read rtmp streams, so it might work. For some reason despite running on my local network the connection seems very unstable when doing it this way.

nate

2011-01-19 06:03:05 pm

don't forget that we only need the video.

UraniumAnchor

2011-01-19 06:07:54 pm

Not a walrus

I tried it with mencoder but for some reason it only reads the first 6 minutes and then quits.

ffmpeg will pause for a minute or so at a time and then grab about 10 minutes of video and then do it again. I turned the timeout parameter way up, but I don't know if it'll be enough to get the entire stream.

I don't know if running FMS in a VM is what's doing it, but watching from the test page works perfectly so worst case I can bust out camtasia.

nate

2011-01-19 06:12:59 pm

there are several more runs that were lost the same way btw ... i didn't note them as i was going through even though i probably should have. it should be fairly obvious once i send the segmented files up though.