Username:
B
I
U
S
"
url
img
#
code
sup
sub
font
size
color
smiley
embarassed
thumbsup
happy
Huh?
Angry
Roll Eyes
Undecided
Lips Sealed
Kiss
Cry
Grin
Wink
Tongue
Shocked
Cheesy
Smiley
Sad
<- 1  -   of 69 ->
--
--
List results:
Search options:
Use \ before commas in usernames
That must be on newer encodes then, because I carved off at least 3x the size of FF9 with the same quality for S3K.
Perhaps I should test on some newer encodes? Any specific that could be recommended?
just record anything on gbp, put it into anri, and encode hq. it's usually around 512 kbit even though the hq bitrate cap is 2048 kbit.
Edit history:
Mystery: 2010-06-08 03:32:57 am
Well, that's going to be a problem seeing as anri only accepts avi. It simply won't accept any published runs and I don't have anything to encode files with.
We'll just see how it turns out.

I also saw that in your settings for x264, you use VBV. Why?
Especially when using bitrate encodes, that seems like a bad idea.
Edit history:
ballofsnow: 2010-06-08 10:50:19 am
Interesting idea on the crf + (maybe) 2nd pass. Need to do some R&D on that. We should compare the time it takes to the usual 2 pass. In 2 pass, the 1st pass is designed to be quick and the 2nd pass does the real job and takes longer, by how much I don't know. I think crf + 2nd pass may take longer since each pass would be assumed as the final pass. In other words, 2 pass uses relaxed settings on the first pass, and then the exact settings you want on the second pass. With crf, it's your exact settings on possibly 2 passes. If this is true it comes down to statistics: how often do we see undersized videos and is it worth it to possibly increase the encoding time on all the other 2 pass encodes?

Also, don't test on published runs... In other words don't test on videos that have already gone through compression.

The VBV settings are used for hardware compatibility (iPod, X360, PS3). It's to keep the bitrate peaks from being too high.
Edit history:
Mystery: 2010-06-08 01:46:41 pm
Quote from ballofsnow:
Also, don't test on published runs... In other words don't test on videos that have already gone through compression.

The VBV settings are used for hardware compatibility (iPod, X360, PS3). It's to keep the bitrate peaks from being too high.

Unfortunately I don't have any "raw" video game video, so I cannot test with that.
I was not aware that the hardware needed the bitrate to be kept inside constraints. Confound them all.

Oh and yes, it will probably take longer to do crf + 2-pass bitrate. Although you could use a quick first pass with crf, but it will hurt quality (in my tests, it was something like 0.1 SSIM).
But hopefully we shouldn't get too many oversized videos. I hope we get none.
And we should get a lot of undersized videos. I hope we get a lot.
i see that failure to meet target bitrate warning a hell of a lot. i don't want to put out a number but it may be more than half the time. obviously way more common in hq/iq. it will probably be 100% of the time with gbp hq.
I fail to see that happening, but I wouldn't want to guess.
But then lowering the crf factor should do the trick in such situations. I don't think the quality hit will be that noticeable.
Anyway, tests, tests, tests. We should discuss more when we have some more raw data.
Quote from Mystery:
I have a working prototype in code that enables this scheme, and it works from I'm told. I could help integrate such a system, or at the very least, donate some code for it. Although I don't know what anri is coded in.

What is your code?
Parsing code for x264's output:
Code:
void x264WriteOut(SX264Data& X264Data, char* buf)
{
	static std::string LeftOverData;
	decltype(LeftOverData) Data = LeftOverData + buf;
	auto pStartOfRow = Data.begin();
	decltype(pStartOfRow) pEndOfRow;

	// Temp variables
	decltype(Data) Temp;
	std::remove_reference<decltype(*Data.begin())>::type c;

	for (;; pStartOfRow = pEndOfRow)
	{
		while (pStartOfRow != Data.end() && (*pStartOfRow == '\n' || *pStartOfRow == '\r'))
			++pStartOfRow;
		pEndOfRow = std::find_if(pStartOfRow, Data.end(), [](decltype(*pStartOfRow) c) { return (c == '\n' || c == '\r'); });
		if (pEndOfRow == Data.end())
		{
			LeftOverData = std::string(pStartOfRow, Data.end());
			break;
		}

		decltype(Data) Row(pStartOfRow, pEndOfRow);
		boost::trim_left(Row);
		if (Row.find("avs [warning]") == 0) // Unnecessary info
			continue;
		if (Row.find("avs [info]") == 0) // Unnecessary info
			continue;
		if (Row.find("x264 [info]") == 0) // Unnecessary info
			continue;

		std::stringstream RowStream(Row);
		if (Row.find("encoded ") == 0)
		{
			// Data looks like: "encoded 32795 frames, "
			if (!(RowStream >> Temp) || Temp != "encoded" ||!(RowStream >> X264Data.TotalEncodedFrames) ||!(RowStream >> Temp) || Temp != "frames,")
				continue; // Unknown data

			// Data will look like: 154.69 fps, "
			if (!(RowStream >> X264Data.AvgFps) || !(RowStream >> Temp) || Temp != "fps,")
				continue; // Unknown data
			
			// Data will look like: "383.99 kb/s"
			if (!(RowStream >> X264Data.AvgKbPerSecond) || !(RowStream >> Temp) || Temp != "kb/s")
				continue;
		}
		else
		{
			// Data will look like: "[100.0%] "
			if (!(RowStream >> c) || (c != '[') || !(RowStream >> X264Data.Progress) || !(RowStream >> c) || c != '%' || !(RowStream >> c) || c != ']')
				continue; // Unknown info

			// Data will look like: "XX/YYYY frames, "
			if (!(RowStream >> X264Data.ProcessedFrames) || !(RowStream >> c) || c != '/' || !(RowStream >> X264Data.TotalFrames) || !(RowStream >> Temp) || Temp != "frames,")
				continue; // Unknown data

			// Data will look like: "XX.YY fps, "
			if (!(RowStream >> X264Data.Fps) || !(RowStream >> Temp) || Temp != "fps,")
				continue; // Unknown data

			// Data will look like: "271.19 kb/s, "
			if (!(RowStream >> X264Data.KbPerSecond) || !(RowStream >> Temp) || Temp != "kb/s,")
				continue; // Unknown data

			// Data will look like: "eta 0:00:00"
			if (!(RowStream >> Temp) || Temp != "eta" ||
				!(RowStream >> X264Data.EtaHours) || !(RowStream >> c) || c != ':' || 
				!(RowStream >> X264Data.EtaMinutes) || !(RowStream >> c) || c != ':' ||
				!(RowStream >> X264Data.EtaSeconds))
				continue; // Unknown data

		}
	}
}


Code that implements the logic of encoding two hard-coded movies:
Code:
	NewThread([]()
	{
		try
		{
			SX264Data X264Data = {};
			auto redir = MakeRedirect([](){}, [&X264Data](char* buf){ x264WriteOut(X264Data, buf); }, [](){}, [](){});
			redir.StartChildProcess(_T("C:\\Program Files\\MeGUI\\tools\\x264\\x264_64.exe"),
				_T("--tune film \"E:\\Encoding Test\\Sonic3Knuckles_eu_SS_Knuckles_3655_HQ.avs\" --pass 1 --crf 15 --output \"E:\\Encoding Test\\Sonic3Knuckles_eu_SS_Knuckles_3655_HQ TEST.mkv\""));
			redir.WaitForThreads();
			if (X264Data.AvgKbPerSecond > 2000)
			{
				redir.StartChildProcess(_T("C:\\Program Files\\MeGUI\\tools\\x264\\x264_64.exe"),
					_T("--tune film \"E:\\Encoding Test\\Sonic3Knuckles_eu_SS_Knuckles_3655_HQ.avs\" ")
					_T("--pass 2 --bitrate 2000 --output \"E:\\Encoding Test\\Sonic3Knuckles_eu_SS_Knuckles_3655_HQ TEST.mkv\""));
				redir.WaitForThreads();
			}
			redir.StartChildProcess(_T("C:\\Program Files\\MeGUI\\tools\\x264\\x264_64.exe"),
				_T("--tune film \"E:\\Encoding Test\\FinalFantasy9_832_HQ_part01.avs\" --pass 1 --crf 15 --output \"E:\\Encoding Test\\FinalFantasy9_832_HQ_part01 TEST.mkv\""));
			redir.WaitForThreads();
			if (X264Data.AvgKbPerSecond > 2000)
			{
				redir.StartChildProcess(_T("C:\\Program Files\\MeGUI\\tools\\x264\\x264_64.exe"),
					_T("--tune film \"E:\\Encoding Test\\FinalFantasy9_832_HQ_part01.avs\" --pass 2 --bitrate 2000 --output \"E:\\Encoding Test\\FinalFantasy9_832_HQ_part01 TEST.mkv\""));
				redir.WaitForThreads();
			}
		}
		catch(const CRedirectError& err)
		{
			std::cout << err.what();
		}
	});

Now, the x264Output function is called by my non-written class CRedirect which basically just copies the output from stdout/stderr. It then parses the data and stores it into a struct.
I'm opting not to post that class for now. I don't know if it will be helpful to you?
Cool. Obviously Anri 3 couldn't use your C code, heh, but it doesn't look as complicated as I thought it might be.
It's C++, btw, but yeah...
It's not complicated, but if there's something you don't understand, I'm willing to answer questions Smiley
I'll also try utilizing IRC.
snow, it looks like the 1-pixel bob applied in anri 3.1 by default (saying yes to the 1-pixel bob question) has the wrong "parity" for the attached input (weaved). can check the sample output to see what i mean. not sure if this is the new 1-pixel bob being applied where it should be the old or not because the 1-pixel bob i used successfully with this clip was this one:

Code:
converttorgb32
clip1=SelectEven.Crop(0,0,0,last.height).AddBorders(0,0,0,0)
clip2=SelectOdd.Crop(0,0,0,last.height-1).AddBorders(0,1,0,0)
Interleave(clip1,clip2)
nate_strobe_fix
converttoyv12


... which i believe is the old one. that confuses me a bit.

also attached the steps i took in anri to produce the output. thanks for looking into this; you know much better than me which 1-pixel bobs are in play.

edit: disregard lol.avi, that was a firefox error ...



Confirmed on my end on 3.0 and 3.1. The 1-pixel code is the same. Is this sample.mp4 a special case perhaps??

3.2 fixes this, uses the alternate pixel bob.

btw, do we need an alternate deflicker as well?
does 3.2 have both the old and the new one?

not sure about retard bob ... it seems like since that is a d1-based filter it may not but i have a sneaking suspicion that's not right since the output is d4. so you would smear down instead of smearing up. but then again maybe not. always found it hard to visualize this stuff.
3.2 has 1 pixel and its alternate, each has the strobe fix.

And I found a clip that will need an alternate retard bob. Will get that done.

3.2 should be going out soon, maybe today.
awesome.
Btw, here should be an interesting piece of information that I managed to acquire:

[BlackFlower] Hmmm. A wonder. If you encode with x264 bitrate mode and set a minimum quantizer, will x264 take this into account when making decisions of where to spend the bitrate or will it produce undersized files?
<Dark_Shikari> minimum quantizer is not really a good way to do that
<Dark_Shikari> because it limits quantizer, not frame quality
<Dark_Shikari> but yes it will
<Dark_Shikari> iirc
[BlackFlower] I know people who are doing it in order to use bitrate mode and possibly create undersized files if possible (bandwidth concerns), so I just wanted to ask.
[BlackFlower] Undersized files, right?
<Dark_Shikari> It's not the right thing to do, ever
[BlackFlower] I'll tell them :p
<Dark_Shikari> If you want to limit the quality, use 2-pass CRF
<Dark_Shikari> i.e. "use this quality, unless it goes beyond this bitrate"
[BlackFlower] Let me see if I got this right... bitrate mode + minimum quantizer creates undersized files? It's not a good thing to do, ever.
[BlackFlower] <Dark_Shikari> If you want to limit the quality, use 2-pass CRF <--- And by this you meant, crf 1st pass and bitrate second pass?
<Dark_Shikari> yes
<Dark_Shikari> bitrate + minimum quantizer is bad because it doesn't limit the quality in a smart fashion
<Tangerine> do slow first pass (full quality) and only reencode in 2nd if the filesize was too much
<Dark_Shikari> that works too

So think of this as SDA is finally moving to encoding "the correct way."
Cool. And coming from Dark_Shikari himself, so you'd best heed his advice.

Nate, what do you think? If this is a go, I'll start playing around with this and aim to get it into Anri 3.3.
Edit history:
Mystery: 2010-06-18 05:30:29 pm
Btw, Dark_Shikari tends to be around the IRC channel all days long. It is part of his job, after all.
Wait.. his job is developing x264?
Edit history:
Mystery: 2010-06-19 03:04:20 pm
Indeed. Or part of it anyway, from what I gather.
Paid to optimize x264 at least, I believe.
Maybe I should ask Tongue

EDIT: I guess it's better to say he's paid to do a lot of coding or consulting. Companies can hire him to do some work.
This is what I gathered from some digging.
Here's the preliminary dos batch version. The output is tee'd to the screen and a text file. The text file is parsed for the bitrate.

Code:
ECHO Beginning x264 first pass...
"%anri_dir%x264.exe" --profile high     --level 4.1 --pass 1 --crf 19 --stats "%2.stats"            --b-adapt 2 --direct auto                    --vbv-bufsize 24000 --vbv-maxrate 24000 --me umh                             --weightp 2             --trellis 2 --output "%2_video.mp4" %1 2>&1 | "%anri_dir%tee.exe" x264pass1_stdout.txt
REM Parse x264pass1_stdout.txt and compare average bitrate. If higher, run second pass.
FOR /F "tokens=6 delims= " %%G IN ('TYPE x264pass1_stdout.txt ^| FIND "encoded"') DO (
  IF %%G GTR %3 (
    ECHO Beginning x264 second pass...
    "%anri_dir%x264.exe" --profile high     --level 4.1 --pass 2 --bitrate %3 --stats "%2.stats"            --b-adapt 2 --direct auto         --vbv-bufsize 24000 --vbv-maxrate 24000 --me umh                             --weightp 2             --trellis 2 --output "%2_video.mp4" %1
  )
)
You aren't using presets? Ie x264 --preset veryslow. I see not a lot of options in your command line, so it seems like it might not be the best settings.
You may also want to use --slow-firstpass in the first pass or quality will suffer.
Edit history:
ballofsnow: 2010-06-19 04:31:29 pm
Thanks for pointing out --slow-firstpass. edit- wait, is this necessary on CRF pass 1?

iirc, I based the settings off x264's default values. Anyway, open to change on x264 settings but keep in mind hardware compatibility between X360/PS3/iPOD.

There is a bug in the dos code: 315.20 GTR 2048[.00] returns true... Going to have to chop off the decimals. x264 uses two decimal points even at xxxx.00 right??
The easiest would probably be to add --preset veryslow (or whatever the slowest speed you can muster is) --tune film.
You should be able to add the rest of the settings to keep hardware compatibility (I assume the VBV is required + a certain level profile?).
I mean, we shouldn't have to specify the level unless it goes above it.