DeDup 0.17 by Loren Merritt, Copyright 2004
based on Dup 2.20 beta 1 by Donald Graft/Klaus Post

DeDup is an Avisynth 2.5 filter, intended to remove duplicate frames in the
interest of compression quality and speed. In some cases, it can also reduce
noise or compression artifacts in the source.

DeDup differs from Dup in that it completely drops the duplicate frames,
rather than keeping several copies of each in the output stream. Then, to
compensate for the dropped frames, DeDup outputs a Matroska-compatible
timecode file to put the remaining frames back in the right places.

This is a two pass filter. While my algorithm could work with one pass, it
wouldn't fit in the Avisynth API.



Usage example:

--- first pass ---

LoadPlugin("DeDup.dll")
AVISource("blah.avi")
DupMC(log="blah.dup.txt")

-------------------
--- second pass ---

LoadPlugin("DeDup.dll")
AVISource("blah.avi")
DeDup(threshold=0.3, maxcopies=10, maxdrops=3, log="blah.dup.txt", times="blah.times.txt")

-------------------




First Pass Options:

DupMC(clip c, bool chroma, string log)

chroma:
	default: true
	Use chroma planes when calculating differences.
	(This is the same as in Dup)

log:
	default: none. required.
	File to save dup metrics for the second pass.

DupMC does not modify the frames passed through it. It only generates the log.
There is no harm in seeking during the first pass: The log will be out of order,
but DeDup doesn't care.
Unlike Dup, which compares the last frame in a run to the first frame,
DupMC always compares adjacent frames. You don't decide on a threshold until
the second pass.



Second Pass Options:

DeDup(clip c, float threshold, float threshold2, int range2, float trigger2,
      bool show, bool dec, int maxcopies, int maxdrops, bool blend,
      string log, string times, string timesin, string ovr, string debug)

threshold:
	range: 0 - 100
	default: 0.3
	Consecutive frames that differ by less than or equal to this much will be
	merged into a single output frame, taken from the end of the merged sequence.
	(This uses the same units as in Dup)

threshold2:
	range: 0 - 100
	default: 0.5 (but disabled due to range2)
	Use this instead of threshold if there are nearby high-motion frames in
	both directions. (Actually, linearly interpolate between threshold and
	threshold2 based on the amount of motion.)

range2:
	range: 0 - lots (sane values: 0 - maxdrops)
	default: 0 (disabled)
	Search range for threshold2.

trigger2:
	range: 0 - 100
	default: 5.0
	What counts as "high-motion" for threshold2.

show:
	default: false
	Print diagnostic information into each frame.

dec:
	default: true
	Drop frames. Without this, DeDup won't do much. But you might want to
	disable this with show=true, to see the frames that would be dropped.

maxcopies:
	range: 1 - 20
	default: 12
	Break any string of duplicates longer than this.
	(This is the same as in Dup.)

maxdrops:
	range: 1 - maxcopies
	default: 1 (no dropped frames (like the original Dup))
	Max input to output frame ratio.
	(So maxdrops=3 means a max of 2 consecutive dropped frames.)
	Don't set this above 3-4 or so if you want to display subtitles, because many
	media players can't update the subtitles until they get a new video frame.

decwhich:
	Which frame from a run of duplicates do we keep?
	0: first frame
	1: middle, round down
	2: middle, round up (default)
	3: last

log:
	default: none.
	The file created in the first pass with DupMC().
	Required, unless you want to specify every frame with ovr.

times:
	default: none.
	File to write matroska timecodes to.
	Not strictly required, but you'll need it for muxing unless maxdrops=1.

timesin:
	default: none.
	Read timestamps for a VFR input.
	Matroska timecode formats v1 and v2 accepted.

ovr:
	default: none.
	File to override settings on a per-frame basis. See below for format.

debug:
	default: none.
	File to write the same information as with show=true.



Override File Format:
Example:

  0-350 threshold=.3
350-900 threshold=.7 threshold2=1.5 trigger2=8.0
100 k
200-220 ddk


Any line not beginning with a frame number is ignored as a comment.

There is no notation for 'last frame', aside from using it's real frame
number. However, it's OK for ranges to extend past the end of the movie.

If two lines override the same property of the same frame, the one later in
the ovr file takes precedence.

Each line is of the format:
	startframe-endframe option [more options]
or
	frame option [more options]

Option overrides accepted:

A string matching /[kd]+/:
	This forces the frame(s) to be (k) kept or (d) considered dups.
	The string of [kd]s will be repeated as necessary to fill the frame range. 
	Forcing a dup does not necessarily cause that frame to be dropped;
	it's still subject to maxdrops.
	This option modifies thresholds, so put any [kd] option after any relevant
	threshold override.

threshold=%f
threshold2=%f
trigger2=%f
	These are the same as the global options.

range2=%d
	The range2 parameter is taken from the frame being considered from dropping,
not from the frames being searched over.

maxdrops=%d
maxcopies=%d
	These values are taken from the frame at the end of the clump of dups, but
it's better to just set them the same over the whole clump.



Tips:
	If you're doing other CPU-intensive filtering, you can save time
(at the cost of space) by saving the first pass (with DupMC) to a huffyuv,
and then run the second pass (with DeDup) from that, instead of repeating
the other filters.
	Don't set threshold too high (especially if maxcopies is large):
Unlike the original Dup, I don't do sanity checking on slow zooms/pans/fades.
	The second pass options 'times' and 'debug' take effect as soon as you
open the avs. You don't have to play/encode the whole video in order to produce
those files.
