Calculating the Quality of a Speedrun

Mkt2015

2005-12-30 12:34:09 pm

MGS for PS1 forever.

A 10% bonus to a new run is silly. Lets take the Warrior Within run old and new.

3:39 multiple segments
3:19 100% single segment

When I wrote up my comments for 3:39 I admitted that there were a few minutes worth of mistakes in it (3-5). SnapDragon's 3:19 also has mistakes in it. His -20 minutes from my run I would say is 20 mins worth of new tricks and shortcuts if both he and I did at least 3-5 mins worth of mistakes.

I'm proposing that shortcuts and tricks don't matter...only the amount of minutes/seconds of mistakes. The reason why I say this is because some games just don't have shortcuts you can take that will shave chunks of time off and many people who view a run and who have not played the game may not know that a shortcut is a shortcut. Example: a friend of mine was playing Half-Life 2 and I made the comment that I had no idea where he was at. He had asked me a question about it since he knew I had watched the speedrun. The reason why I had no idea where he was was because in the speedrun the player completely bypassed the area.

However, even if tricks and shortcuts aren't even put into the mix when it comes to the quality of a speedrun, given that his is single-segment while mine is not would increase the quality factor by at least 10 fold.

While the new run is much better a 10% bonus means nothing. A run could be improved by a minute and people would think that's it's extraordiary or it could be improved by 30 mins. Because of this I think what I said earlier is a much better way of deciding quality...no matter what # the run is. (3 hours --> 5 mins of mistakes, etc.)

A 10% bonus to a new run is silly. Lets take the Warrior Within run old and new.

3:39 multiple segments
3:19 100% single segment

When I wrote up my comments for 3:39 I admitted that there were a few minutes worth of mistakes in it (3-5). SnapDragon's 3:19 also has mistakes in it. His -20 minutes from my run I would say is 20 mins worth of new tricks and shortcuts if both he and I did at least 3-5 mins worth of mistakes.

I'm proposing that shortcuts and tricks don't matter...only the amount of minutes/seconds of mistakes. The reason why I say this is because some games just don't have shortcuts you can take that will shave chunks of time off and many people who view a run and who have not played the game may not know that a shortcut is a shortcut. Example: a friend of mine was playing Half-Life 2 and I made the comment that I had no idea where he was at. He had asked me a question about it since he knew I had watched the speedrun. The reason why I had no idea where he was was because in the speedrun the player completely bypassed the area.

However, even if tricks and shortcuts aren't even put into the mix when it comes to the quality of a speedrun, given that his is single-segment while mine is not would increase the quality factor by at least 10 fold.

While the new run is much better a 10% bonus means nothing.  A run could be improved by a minute and people would think that's it's extraordiary or it could be improved by 30 mins. Because of this I think what I said earlier is a much better way of deciding quality...no matter what # the run is. (3 hours --> 5 mins of mistakes, etc.)

Gorash

2005-12-30 01:24:12 pm

${$uid.$user}{' usertext'}

I admit that my measure is not exactly quality.
It's more a measure how hard the run is to beat without new tricks (assuming the verifier did his job well).

A question to your rating, it only focusses on execution aswell, doesn't it? And by mistakes you only mean execution mistakes, not those from route planning or random events happening suboptimal.

Trevor

2005-12-30 01:51:02 pm

most people except the competitors won't know the difficulty of producing an excellent speedrun for a certain game.

RandomEngy

2005-12-30 04:53:07 pm

A Crab

Any mathematical rating of how good a speedrun is can only be an approximation. Any metric can be torn apart with a couple well-chosen examples, but you probably could craft a rating system with a decent correlation with the overall "goodness" of a run.

But I'd say that the "impressive" factor has the most to do with the amount of game experience the runner has, along with the amount of time spent running it. The "wow that shortcut is cool" factor is a combination of the inventiveness and amount of time screwing around in the game along with the amount of time spent debugging the game in development.

However the overall enjoyment is something much harder to pin down. Not only is the enjoyment different for every person, it's heavily affected by the game itself: the type, pacing and quality.

If you wanted to, you could probaly spend a long, long time doing statistical samples, measuring things on different speedruns and coming up with increasingly accurate models. However it kind of seems like a big waste of time.

Any mathematical rating of how good a speedrun is can only be an approximation.  Any metric can be torn apart with a couple well-chosen examples, but you probably could craft a rating system with a decent correlation with the overall "goodness" of a run.

But I'd say that the "impressive" factor has the most to do with the amount of game experience the runner has, along with the amount of time spent running it.  The "wow that shortcut is cool" factor is a combination of the inventiveness and amount of time screwing around in the game along with the amount of time spent debugging the game in development.

However the overall enjoyment is something much harder to pin down.  Not only is the enjoyment different for every person, it's heavily affected by the game itself: the type, pacing and quality.

If you wanted to, you could probaly spend a long, long time doing statistical samples, measuring things on different speedruns and coming up with increasingly accurate models.  However it kind of seems like a big waste of time.

jimsfriend

2005-12-30 05:06:27 pm

Like A Fox

Quote:

However it kind of seems like a big waste of time.

How big?

What would be the point of calculating the quality anyway?

And at Gorash's model, going on the game I know most (SMW), flying would not require any control? What if a human were to pull off the trick that saves 3 game time units in DP1 where you get the cape faster and immediately start flying. This flying is controlled only by directional input once you are off the ground, however you have to be able to gain altitude to reach the key. In order to make the trick work it is required that you press --> at the right time in order to bounce off those flying guys in order to gain speed and altitude.

Gorash

2005-12-30 06:27:59 pm

${$uid.$user}{' usertext'}

The thing why I excluded directional input is that you press the directional buttons almost always, and that would crush all statistics. Also analog input is completely not countable.

But we can settle for general input changes. That'd work on digital directional too, only problem remaining would be analog sticks.

Ekudeht

2005-12-30 07:57:53 pm

I love YaBB 1G - SP1!

Any measurement of how good a speedrun is would have to be a comparison of the finishing time over what a "perfect" time would be. Since it's not possible to determine what a perfect time is all you could do is estimate.

Overall I'd say a good speedrun should contain no more than 3 minutes of mistakes for every 1 hour of gameplay, a great speedrun should contain no more than 2 minutes of mistakes for every 1 hour of gameplay and a fantastic speedrun should have less than 1 minute worth of mistakes for every hour of the speedrun.

That is only my personal opinion.

Mkt2015

Edit history:

Mkt2015: 2005-12-30 08:03:43 pm

2005-12-30 07:58:57 pm

MGS for PS1 forever.

I know my thoughts leave out suboptimization of planning.

1. any person who hasn't played/seen the game won't know it and

2. this would fall under the "soon to be discovered tricks and shortcuts" idea of speedrunning

The idea is that a person could do a run that everyone including the runner himself looks optimal and has nearly no mistakes.

A month later someone beats it by 20 minutes. I believe that some people are able to find shorcuts and tricks easier than others and does a 20 minute better run mean that there are less mistakes? No.

The reason why 3 hrs will have at least 5 minutes worth of mistakes is because:

1. everyone is human and I'm sure that if you were to calculate every single error it would be 5 minutes worth and

2. Radix does want quality runs and a verify could tell if a 3 hr run has more than 5 mins worth of mistakes (if the verifier saw more than 5 mins I'm sure the run would be rejected)

Duke...a comparison of time to beat the game and a time for a "perfect" game, while it sounds nice in a logical way, would never work. It's like trying to sit around and estimate how big the universe is. No one knows how fast a game can be beaten and since there are no "perfect" runs, we will never know.

Even in the Id levels of Quake, runners are still able to slice seconds off of previously thought are optimized. The people who update the page have even said things like "I have no idea where he shaves off a second, but he does!"

I know my thoughts leave out suboptimization of planning.

1. any person who hasn't played/seen the game won't know it and

2. this would fall under the "soon to be discovered tricks and shortcuts" idea of speedrunning

The idea is that a person could do a run that everyone including the runner himself looks optimal and has nearly no mistakes.

A month later someone beats it by 20 minutes. I believe that some people are able to find shorcuts and tricks easier than others and does a 20 minute better run mean that there are less mistakes? No.

The reason why 3 hrs will have at least 5 minutes worth of mistakes is because:

1. everyone is human and I'm sure that if you were to calculate every single error it would be 5 minutes worth and

2. Radix does want quality runs and a verify could tell if a 3 hr run has more than 5 mins worth of mistakes (if the verifier saw more than 5 mins I'm sure the run would be rejected)

Duke...a comparison of time to beat the game and a time for a "perfect" game, while it sounds nice in a logical way, would never work. It's like trying to sit around and estimate how big the universe is. No one knows how fast a game can be beaten and since there are no "perfect" runs, we will never know.

Even in the Id levels of Quake, runners are still able to slice seconds off of previously thought are optimized. The people who update the page have even said things like "I have no idea where he shaves off a second, but he does!"

Enhasa

2005-12-31 03:13:10 am

everybody wanna tell you the meaning of music

First of all, I have to say, I'm not looking forward to the day a rating system is implemented. Maybe if it was done by SDA higher-ups like the moderators. Bisqwit's star system is as close to bearable as I can stand, but it's still overly cluttered.

I got the Drake equation, but since Yoshi348 got it before I saw this thread, you'd have to take my word for it. Wink

Last thing: Cid's Theme > everything else in FF7. If you understand the word "hyperbole," you'll find this a great review.

darkwasabi

2005-12-31 04:03:33 am

King of hearts

Considering time itself isn't constant, nothing can ever be perfect, then why are we even discussing this?

Anyway I don't think a rating system would work, or should be implemented. Many things are underrated, and overrated. It is completely opinion based and will only bring ambiguity and arguments.

Mkt2015

2005-12-31 09:25:58 am

MGS for PS1 forever.

Quote:

Considering time itself isn't constant, nothing can ever be perfect, then why are we even discussing this?

WTF does this even mean? Maybe your perception of time isn't constant but I believe time is.

So what if nothing can ever be perfect. We're not trying for perfection...we're just trying to get as close to it as we can. And to answer the question, we are discussing this because it is interesting reading what others think and coming up with somewhat of a definitive system. Not that we will use the system, but it's still interesting to think about it.

Quote:

Anyway I don't think a rating system would work, or should be implemented. Many things are underrated, and overrated. It is completely opinion based and will only bring ambiguity and arguments.

Thank-you thread killer. Are you quite satisfied now that you have hacked up your own opinion and declared that this is pointless? Here are a few philosophy lessons for you:

1. almost everything is completely based on a opinion
2. the only way to get to a conclusion is to question
3. i don't believe we are arguing...we are discussing

Even though I'm rather enjoying this back and forth discussion, I would hate to see this thread be pushed back and then a year from now someone else bring up the idea again. While there is a point in discussing it...there is no point in discussing and then doing nothing with the conclusion. That, in my opinion, is philosophical masterbation, which is a true waste of time.

[quote]Considering time itself isn't constant, nothing can ever be perfect, then why are we even discussing this?[/quote]

WTF does this even mean? Maybe your perception of time isn't constant but I believe time is.

So what if nothing can ever be perfect. We're not trying for perfection...we're just trying to get as close to it as we can. And to answer the question, we are discussing this because it is interesting reading what others think and coming up with somewhat of a definitive system. Not that we will use the system, but it's still interesting to think about it.
 
[quote]Anyway I don't think a rating system would work, or should be implemented.  Many things are underrated, and overrated.  It is completely opinion based and will only bring ambiguity and arguments.[/quote]

Thank-you thread killer. Are you quite satisfied now that you have hacked up your own opinion and declared that this is pointless? Here are a few philosophy lessons for you:

1. almost everything is completely based on a opinion
2. the only way to get to a conclusion is to question
3. i don't believe we are arguing...we are discussing

Even though I'm rather enjoying this back and forth discussion, I would hate to see this thread be pushed back and then a year from now someone else bring up the idea again. While there is a point in discussing it...there is no point in discussing and then doing nothing with the conclusion. That, in my opinion, is philosophical masterbation, which is a true waste of time.

darkwasabi

2005-12-31 09:45:20 am

King of hearts

http://forums.philosophyforums.com/thread/18001

i.e. the exact juxtaposition of yourself and console during the run will make slight variations. But this is on an extrmemely small scale...

Anyway, I meant why are we disscussing perfection in a run. There wouldn't be any way to tell what is or isn't, so you can't compare a human made run to perfection.

I was just stating my opinion. I don't want to see a ratings system up since I think they would promote the runs to get good ratings due to popularity. Fanboys/girls will give good ratings because they like the game.

Mkt2015

2005-12-31 10:39:41 am

MGS for PS1 forever.

Quote:

http://forums.philosophyforums.com/thread/18001

i.e. the exact juxtaposition of yourself and console during the run will make slight variations. But this is on an extrmemely small scale...

The people in that forum are just spinning their wheels and are anal-retentive. No one has ever gone the speed of light, so what's the point?

Quote:

Anyway, I meant why are we disscussing perfection in a run. There wouldn't be any way to tell what is or isn't, so you can't compare a human made run to perfection.

The only time someone mentioned perfection was the very first post and it had to do with a poem. The word was never brought up later on.

Quote:

I was just stating my opinion. I don't want to see a ratings system up since I think they would promote the runs to get good ratings due to popularity. Fanboys/girls will give good ratings because they like the game.

This is true. However, I believe this site already uses, relatively speaking, the 3 hr --> 5 minute thing. I doubt Radix would point anything that has more noticable mistakes.

BTW, sorry if I've seemed a bit annoyed, but I take philosophy very seriously.

[quote]http://forums.philosophyforums.com/thread/18001
 
i.e. the exact juxtaposition of yourself and console during the run will make slight variations.  But this is on an extrmemely small scale...[/quote]

The people in that forum are just spinning their wheels and are anal-retentive. No one has ever gone the speed of light, so what's the point?
 
[quote]Anyway, I meant why are we disscussing perfection in a run.  There wouldn't be any way to tell what is or isn't, so you can't compare a human made run to perfection.[/quote]

The only time someone mentioned perfection was the very first post and it had to do with a poem. The word was never brought up later on.
 
[quote]I was just stating my opinion.  I don't want to see a ratings system up since I think they would promote the runs to get good ratings due to popularity.  Fanboys/girls will give good ratings because they like the game.[/quote]

This is true. However, I believe this site already uses, relatively speaking, the 3 hr --> 5 minute thing. I doubt Radix would point anything that has more noticable mistakes.

BTW, sorry if I've seemed a bit annoyed, but I take philosophy very seriously.

Lag.Com

2005-12-31 10:50:49 am

sda loyalist

The whole point of the ratings system that most of us have been talking about is that it is not based on personal opinion, only on time saved/mistakes made.

darkwasabi

2005-12-31 11:40:39 am

King of hearts

Even if it is on time saved or mistakes made. What is the time compared to if there isn't a TAS run of it (in most cases there aren't)? And what if someone is careful, and moves at a slightly slower pace than normal, but then this wouldn't be too noticable, and wouldn't be counted off as much as it should.

You'd also need to account for the difficulty of the game/run, (which is also opinion based) and would make the ratings system too complex. A Zelda run like the old 5+ hour run would've seemed very impressive to even an advanced Zelda player, but to the Zelda elite, it's crap.

Also, improving a run by a very small amount can be a tremendous feat. But when rated, it was only a small amount faster so it can only get a small ratings boost.
If a run seemed optimized to the fullest, and got the highest rating. Then, another runner put in 20 times the effort and determination, but only gained a total of 3 seconds.

It's all relative to each individual game, but then relative to every other game as well. I mean, yeah it could be done, but it would be too complex and inaccurate to be worth the trouble.

Even if it is on time saved or mistakes made.  What is the time compared to if there isn't a TAS run of it (in most cases there aren't)?  And what if someone is careful, and moves at a slightly slower pace than normal, but then this wouldn't be too noticable, and wouldn't be counted off as much as it should.

You'd also need to account for the difficulty of the game/run, (which is also opinion based) and would make the ratings system too complex.  A Zelda run like the old 5+ hour run would've seemed very impressive to even an advanced Zelda player, but to the Zelda elite, it's crap.

Also, improving a run by a very small amount can be a tremendous feat.  But when rated, it was only a small amount faster so it can only get a small ratings boost.  
If a run seemed optimized to the fullest, and got the highest rating.  Then, another runner put in 20 times the effort and determination, but only gained a total of 3 seconds.

It's all relative to each individual game, but then relative to every other game as well.  I mean, yeah it could be done, but it would be too complex and inaccurate to be worth the trouble.