Auto decimate (no more F database?)

Edit history:

ballofsnow: 2009-09-11 09:51:58 pm

2009-09-07 12:22:12 pm

So I revisited an AviSynth filter called Dup by Neuron2. It analyzes the video, looks for duplicate frames and has the ability to create a log file which can be parsed to determine if a video is likely F1, 2, or 3. The filter is not 100% accurate - such was my expectation the last time I visited this and subsequently became discouraged and gave up - but I don't think it needs to be.

- Download Dup (2.30) and extract to AviSynth plugins directory
- Created two text files: dupcount.bat and RunNullVideoPass.vcf. Modify path to vdub.exe cli as needed.

RunNullVideoPass.vcf

Code:

VirtualDub.Open(VirtualDub.params[0]);
VirtualDub.RunNullVideoPass();

dupcount.bat

Code:

@echo off
SETLOCAL ENABLEDELAYEDEXPANSION

SET vdubdir=C:\Program Files\VirtualDub-1.9.5
SET duplog=%~dp1dup.txt
SET unique=0
SET duplicate=0

"%vdubdir%\vdub.exe" /i RunNullVideoPass.vcf %1

FOR /F "skip=2 tokens=3,6 delims=: " %%G IN ('type "%duplog%"') DO (
	IF "%%G"=="%%H" (
		IF !dupseq! LEQ 2 (
			SET /A unique= !unique! + 1
			SET /A duplicate= !duplicate! + !dupseq!
		)
		SET dupseq=0
	) ELSE (
		SET /A dupseq=!dupseq! + 1
	)
)

SET /A total = %unique% + %duplicate%
SET /A uniquepercent = (%unique% * 100) / %total%
ECHO.
ECHO Frames counted: %total%
ECHO Unique: %unique%
ECHO Percentage: %uniquepercent%
ECHO.
PAUSE

- Add following AviSynth code after separatefields

Code:

Dup(threshold=20, blksize=16, debug=true, log="dup.txt")

- Open a command prompt: dupcount.bat avisynthscript.avs

- Result of F1 video, short sample:

Obviously not 100%, but if it's above 50%.... Can play with the thresholds after doing more testing... (nate? Smiley

)

Any thoughts?

Extra links: http://www.virtualdub.org/docs/vdscript.txt

edit- Menus will affect results from the dup filter... The above screenshot was a sample of sonic 2 which had a little bit of intro. Take out the intro and the percentage rises to 97 percent. May be possible to add a bit more logic to the batch file, detecting when there's a string of duplicates higher than say.. 3-5 and ignoring them.

edit- Updated batch code with above idea, getting surprisingly accurate results: 95/49/33% on F1/F2/F3. Basically the code ignores sequences of DDDU or DDDDDDDDDU, etc. It will count only sequences of UU, DU, DDU.

Thread title:

So I revisited an AviSynth filter called Dup by Neuron2. It analyzes the video, looks for duplicate frames and has the ability to create a log file which can be parsed to determine if a video is likely F1, 2, or 3. The filter is not 100% accurate - such was my expectation the last time I visited this and subsequently became discouraged and gave up - but I don't think it needs to be.

- Download [url=http://neuron2.net/dup/dupnew.html]Dup[/url] (2.30) and extract to AviSynth plugins directory
- Created two text files: dupcount.bat and RunNullVideoPass.vcf. Modify path to vdub.exe cli as needed.

[u]RunNullVideoPass.vcf[/u]
[code]
VirtualDub.Open(VirtualDub.params[0]);
VirtualDub.RunNullVideoPass();
[/code]

[u]dupcount.bat[/u]
[code]
@echo off
SETLOCAL ENABLEDELAYEDEXPANSION

SET vdubdir=C:\Program Files\VirtualDub-1.9.5
SET duplog=%~dp1dup.txt
SET unique=0
SET duplicate=0

"%vdubdir%\vdub.exe" /i RunNullVideoPass.vcf %1

FOR /F "skip=2 tokens=3,6 delims=: " %%G IN ('type "%duplog%"') DO (
	IF "%%G"=="%%H" (
		IF !dupseq! LEQ 2 (
			SET /A unique= !unique! + 1
			SET /A duplicate= !duplicate! + !dupseq!
		)
		SET dupseq=0
	) ELSE (
		SET /A dupseq=!dupseq! + 1
	)
)

SET /A total = %unique% + %duplicate%
SET /A uniquepercent = (%unique% * 100) / %total%
ECHO.
ECHO Frames counted: %total%
ECHO Unique: %unique%
ECHO Percentage: %uniquepercent%
ECHO.
PAUSE
[/code]

- Add following AviSynth code after separatefields

[code]Dup(threshold=20, blksize=16, debug=true, log="dup.txt")[/code]

- Open a command prompt: dupcount.bat avisynthscript.avs

- Result of F1 video, short sample:

[img]http://img5.imageshack.us/img5/866/dup.png[/img]

Obviously not 100%, but if it's above 50%.... Can play with the thresholds after doing more testing... (nate? :) )

Any thoughts?

Extra links: http://www.virtualdub.org/docs/vdscript.txt

edit- Menus will affect results from the dup filter... The above screenshot was a sample of sonic 2 which had a little bit of intro. Take out the intro and the percentage rises to 97 percent. May be possible to add a bit more logic to the batch file, detecting when there's a string of duplicates higher than say.. 3-5 and ignoring them.

edit- Updated batch code with above idea, getting surprisingly accurate results: 95/49/33% on F1/F2/F3. Basically the code ignores sequences of DDDU or DDDDDDDDDU, etc. It will count only sequences of UU, DU, DDU.

[img]http://img188.imageshack.us/img188/5900/50520378.png[/img]
[img]http://img188.imageshack.us/img188/4189/40394732.png[/img]
[img]http://img181.imageshack.us/img181/2788/60997384.png[/img]

nate

2009-09-11 08:53:19 am

those results are very accurate. however i don't think it would be responsible to deploy this without testing it on a wide variety of sources and comparing the results to what we have the game listed as in the database. the best way to do that is to deploy it but ignore what it returns - simply post it to sda instead. i don't think we even need to ask for permission since it doesn't send anything that could be used to identify people. just the game they picked and the autocalculated f. in this way we could quickly develop an idea of its accuracy.

there's also the question of d and framerate decimation - since f comes along with those anyway, as far as i know, it wouldn't streamline things from the user's perspective to automatically discover f at the present time. this is not to say that this method shouldn't be deployed, but simply that it shouldn't be deployed unless we can be reasonably sure about its performance in the field. it wouldn't serve to do anything anri doesn't already do as far as i can tell; it would simply be an r&d project waiting for automatic d (and maybe fdp) to come around.

those results are very accurate. however i don't think it would be responsible to deploy this without testing it on a wide variety of sources and comparing the results to what we have the game listed as in the database. the best way to do that is to deploy it but ignore what it returns - simply post it to sda instead. i don't think we even need to ask for permission since it doesn't send anything that could be used to identify people. just the game they picked and the autocalculated f. in this way we could quickly develop an idea of its accuracy.

there's also the question of d and framerate decimation - since f comes along with those anyway, as far as i know, it wouldn't streamline things from the user's perspective to automatically discover f at the present time. this is not to say that this method shouldn't be deployed, but simply that it shouldn't be deployed unless we can be reasonably sure about its performance in the field. it wouldn't serve to do anything anri doesn't already do as far as i can tell; it would simply be an r&d project waiting for automatic d (and maybe fdp) to come around.

ballofsnow

2009-09-11 07:52:52 pm

Agreed. Just need to be patient then, or maybe this will be useful in a different way and not necessarily have to obsolete the F database.

bmn

2009-09-11 08:52:01 pm

Perhaps a dedicated routine in Anri?

1: Start a new project
2: Rip DVD
...
n: Analyse source to find F

nate

2009-09-11 08:55:01 pm

why would anyone want to do that though?