Demtool appending woes

Edit history:

dex: 2007-02-09 07:44:59 pm

2007-02-09 07:30:03 pm

Invisible avatar

Okay, let me fill you with a little background story for this problem here. Today, doing my stuff around #qdq as always, Stubby pm'ed me, asking for help with appending demos, as i have already helped him with one of the blooper reels a few months ago.
Well anyway....his problem was a very big filesize after appending a few demos. His bloopers are 16 part bloopers, and the filesize pushed over 40 megs after 8th one. And mind you, i've agreed to append the demos for him, being the cool guy i am. They were 3 megs after unpacking if they were one by one.
I've made myself a cool batch file some time ago that basically does everything with demtooling for me through a text interface which is very easy to operate. So i started it up, chosen the appending function, then 16 demos for joining. I've started feeding the program with demos, one by one. After 8th one demtool took a ridiculously big amount of time to process the demos. Even lowering the framerate to 15 and removing entities outside fov took an unbearable 50 megs of space after 10 demos, which is clearly unacceptable. Then, i remembered the problem i had myself a few months ago, when making a blooper reel for r2m1. Basically, my file was over 50 megs unpacked with everything included in.

So i started to wonder - did it happen with every demo? I've tried on my drunk e1m1 demos...the result was exponentially rising filesize. I've decided to put comparison demos to good use and, for good of science, i've tried on e1m1 and e1m6. The demos are actually random ones i downloaded from the bunny comparison section of the site (and three on E1M1 are my drunk demos :P). The filesizes listed are for those files in the order they were appended.

The results:
E1M1:

file joined	size after in KB	size of the file itself
1	233	233
2	629	199
3	812	49
4	2,325	190
5	5,765	216
6	13,786	251
7	29,661	249
8	59,677	235

E1M6:

file joined	size after in KB	size of the file itself
1	153	153
2	452	150
3	895	111
4	1,825	117
5	5,274	216
6	10,568	166
7	25,287	230
8	54,715	230

As you can clearly see, the filesize gets bigger exponentially with more demos. With 10 demos, those things would break through 200 megs.

Also, while doing it i've noticed another weirdness. While using my batch file for demtool, i've accidentaly turned it off while making the fifth demo of the E1M6. So, i started it again and rushed through the first four demos appending (in like 20 seconds). Result? 5th demo was 8,539 KB. Compare with the result from the table. 8th demo was 89,922 KB after something like that.

This seems like a very big bug. Is it fault of demtool, or of the demo recording itself? Is it a known bug?

Edit: Also, in what programming language is demtool written?

Thread title:

Okay, let me fill you with a little background story for this problem here. Today, doing my stuff around #qdq as always, Stubby pm'ed me, asking for help with appending demos, as i have already helped him with one of the blooper reels a few months ago.
Well anyway....his problem was a very big filesize after appending a few demos. His bloopers are 16 part bloopers, and the filesize pushed over 40 megs after 8th one. And mind you, i've agreed to append the demos for him, being the cool guy i am. They were 3 megs after unpacking if they were one by one. 
I've made myself a cool batch file some time ago that basically does everything with demtooling for me through a text interface which is very easy to operate. So i started it up, chosen the appending function, then 16 demos for joining. I've started feeding the program with demos, one by one. After 8th one demtool took a ridiculously big amount of time to process the demos. Even lowering the framerate to 15 and removing entities outside fov took an unbearable 50 megs of space after 10 demos, which is clearly unacceptable. Then, i remembered the problem i had myself a few months ago, when making a blooper reel for r2m1. Basically, my file was over 50 megs unpacked with everything included in.

So i started to wonder - did it happen with every demo? I've tried on my drunk e1m1 demos...the result was exponentially rising filesize. I've decided to put comparison demos to good use and, for good of science, i've tried on e1m1 and e1m6. The demos are actually random ones i downloaded from the bunny comparison section of the site (and three on E1M1 are my drunk demos :P). The filesizes listed are for those files in the order they were appended.

The results:
E1M1:

[table]

[tr]
[td][center][b]file joined[/b][/center][/td]
[td][center][b]             size after in KB[/b][/center][/td]
[td][center][b]             size of the file itself[/b][/center][/td]
[/tr]

[tr]
[td][center]1[/center][/td]
[td]   233[/td]
[td]   233[/td]
[/tr]

[tr]
[td][center]2[/center][/td]
[td]   629[/td]
[td]   199[/td]
[/tr]

[tr]
[td][center]3[/center][/td]
[td]   812[/td]
[td]   49[/td]
[/tr]

[tr]
[td][center]4[/center][/td]
[td]   2,325[/td]
[td]   190[/td]
[/tr]

[tr]
[td][center]5[/center][/td]
[td]   5,765[/td]
[td]   216 [/td]
[/tr]

[tr]
[td][center]6[/center][/td]
[td]   13,786[/td]
[td]   251 [/td]
[/tr]

[tr]
[td][center]7[/center][/td]
[td]   29,661[/td]
[td]   249 [/td]
[/tr]

[tr]
[td][center]8[/center][/td]
[td]   59,677[/td]
[td]   235[/td]
[/tr]
[/table]

E1M6:

[table]

[tr]
[td][center][b]file joined[/b][/center][/td]
[td][center][b]             size after in KB[/b][/center][/td]
[td][center][b]             size of the file itself[/b][/center][/td]
[/tr]

[tr]
[td][center]1[/center][/td]
[td]   153[/td]
[td]   153[/td]
[/tr]

[tr]
[td][center]2[/center][/td]
[td]   452[/td]
[td]   150[/td]
[/tr]

[tr]
[td][center]3[/center][/td]
[td]   895[/td]
[td]   111[/td]
[/tr]

[tr]
[td][center]4[/center][/td]
[td]   1,825[/td]
[td]   117[/td]
[/tr]

[tr]
[td][center]5[/center][/td]
[td]   5,274[/td]
[td]   216 [/td]
[/tr]

[tr]
[td][center]6[/center][/td]
[td]   10,568[/td]
[td]   166[/td]
[/tr]

[tr]
[td][center]7[/center][/td]
[td]   25,287[/td]
[td]   230[/td]
[/tr]

[tr]
[td][center]8[/center][/td]
[td]   54,715[/td]
[td]   230[/td]
[/tr]
[/table]

As you can clearly see, the filesize gets bigger exponentially with more demos. With 10 demos, those things would break through 200 megs.

Also, while doing it i've noticed another weirdness. While using my batch file for demtool, i've accidentaly turned it off while making the fifth demo of the E1M6. So, i started it again and rushed through the first four demos appending (in like 20 seconds). Result? 5th demo was 8,539 KB. Compare with the result from the table. 8th demo was 89,922 KB after something like that.

This seems like a very big bug. Is it fault of demtool, or of the demo recording itself? Is it a known bug?

Edit: Also, in what programming language is demtool written?

Morfans

Edit history:

Morfans: 2007-02-09 07:59:53 pm

2007-02-09 07:58:57 pm

Hell is where the heart is.

How are you joining them? And don't just say "demtool". :-)

How are the demos cut, pre-processed, joined, post-processed?

Also, join two small demos together creating a disproportionately big demo and upload it somewhere so we can see what's padding it out.

Edit: pascal, if i remember correctly :-)

dex

Edit history:

dex: 2007-02-09 08:20:56 pm

2007-02-09 08:10:38 pm

Invisible avatar

In the first demo I use demtool -d to avoid problems with joining, then i append with -a. Those filesizes are without post-processing with removing entities outside fov (which reduces filesize by 4-5 megs at most) and changing the framerate to something low, like 15 (which salvages 10-15 megs, which is strange, it should go slightly lower...). If i lower the framerate and remove the entities outside fov before the processing, the filesize gets lower by a very marginal number, like 4-5 megs.
Demos were cut by Stubby for e2m1, by me for r2m1 and for the e1m1 and e1m6, i took whole comparison demos.
Two small demos to create a lot bigger video? I'd say 4 demos should be enough to see the problem clearly. I'll upload it in a sec.

EDIT: Made a bundle with four demos i used, one in which i joined 2 demos (jonandpeter is the name) and one with 4 demos. No real processing. You can see how the filesize is just a little off at two demos, but a lot different at 4 demos.

http://download.yousendit.com/10CCB81834EBBAC7

In the first demo I use demtool -d to avoid problems with joining, then i append with -a. Those filesizes are without post-processing with removing entities outside fov (which reduces filesize by 4-5 megs at most) and changing the framerate to something low, like 15 (which salvages 10-15 megs, which is strange, it should go slightly lower...). If i lower the framerate and remove the entities outside fov before the processing, the filesize gets lower by a very marginal number, like 4-5 megs.
Demos were cut by Stubby for e2m1, by me for r2m1 and for the e1m1 and e1m6, i took whole comparison demos.
Two small demos to create a lot bigger video? I'd say 4 demos should be enough to see the problem clearly. I'll upload it in a sec.

EDIT: Made a bundle with four demos i used, one in which i joined 2 demos (jonandpeter is the name) and one with 4 demos. No real processing. You can see how the filesize is just a little off at two demos, but a lot different at 4 demos.

http://download.yousendit.com/10CCB81834EBBAC7

Mandel

2007-02-09 11:48:51 pm

Mumma

you can always join demos manually by using lmpc. Just remove the cd track info from all demos except the first.

Two joined demos should not be bigger than the combined size of the original demos.

demtool is pascal, source is available off this site i think.

drunk and using a mobile phone seeya.

Radix

2007-02-10 02:27:53 am

I'm addicted to games

Ugh, dont lower the fps to save size. It's two thousand freaking seven!

dex

2007-02-10 11:05:08 am

Invisible avatar

Quote:

Ugh, dont lower the fps to save size. It's two thousand freaking seven!

Oh, i know. But i was trying every way possible to reduce insane 100+ MB filesizes.

Mandel: I'll be sure to check this out Smiley

GinKo_

2007-02-10 05:05:45 pm

I had the same problem while making a blooper reel for my e3m7_052 Careful run, which had 27 small demos. If I remember correctly, I solved it by using the demtool command to append files at the begining of the input instead of the end (or the opposite).

The other demos section takes so long to update that there's a good chance that demo won't be a record, as it's a tablefiller :-)