Continuation of Roundtable Discussion

2012-11-09 01:43:59 pm

HELLO!

Fair point. It may be that in 3 months verification is responsive with the current system. I'd say current process, but I can see that the team has been hard at work refining the process to make it work better, and harder for a run to slip through the cracks.

dunnius

Edit history:

dunnius: 2012-11-09 01:56:58 pm

dunnius: 2012-11-09 01:56:48 pm

2012-11-09 01:49:06 pm

Intruding N313 and F014

The idea of vote up/down for open verification is more useful as comments rather than a poll. Even though TASVideos has a poll, it is not what judges use to make the decision, it is the comments that matter. I don't see a poll being necessary for open verifications if there are good comments.

As for people not submitting runs because the process is too slow, I am certainly in that group. I am still improving my run, but I could submit my best for now because it is way better than what is currently up, and it will be a while before I can improve it. But because I know that I will improve the run before my current run gets posted, it seems like a waste of time to submit the run. I'll agree that the posting times for submissions needs to be improved.
Edit: That will help keep runs from being out of date for too long, and help the site.

ChristosOwen

2012-11-09 01:52:57 pm

Quote from dunnius:

The idea of vote up/down for open verification is more useful as comments rather than a poll. Even though TASVideos has a poll, it is not what judges use to make the decision, it is the comments that matter. I don't see a poll being necessary for open verifications if there are good comments.

As for people not submitting runs because the process is too slow, I am certainly in that group. I am still improving my run, but I could submit my best for now because it is way better than what is currently up, and it will be a while before I can improve it. But because I know that I will improve the run before my current run gets posted, it seems like a waste of time to submit the run. I'll agree that the posting times for submissions needs to be improved.

I completely agree with the first point, and am in the exact same boat for the second point.

Omnigamer

Edit history:

Omnigamer: 2012-11-09 03:13:09 pm

2012-11-09 03:10:21 pm

All the things

Quote from dunnius:

The idea of vote up/down for open verification is more useful as comments rather than a poll. Even though TASVideos has a poll, it is not what judges use to make the decision, it is the comments that matter. I don't see a poll being necessary for open verifications if there are good comments.

Up/down voting for the individual comments, or the run itself? If the up/down votes are for the submission, I don't see how it's much different from regular voting or a popularity contest. The formula changes slightly for individual ranking of comments so that the most relevant insight rises to the top, but that assumes the system is not abused. Showing popularity is a reasonable thing, but I don't think it should be the method of verification.

Something I proposed last night was instead a flagging system. The more important aspect of verifications is that the minority be heard because that is what allows the more subtle problems to come to light. The current verification already does this: the verifiers are supposed to be knowledgeable in the specific game and watch closely for suspicious effects or poor play. Opening up the verifications to everybody means that this focus is no longer assumed, but any problems must still be brought up. Players that see suspicious activity or think that the video does not meet the standards for SDA can flag the video for further review.

This kind of system is not without pitfalls or abuses, however. This means that staff must respond to flags individually, which may increase the burden depending on the number of submissions/flags. If the flag's reasoning is valid and more investigation needs to be done (or a not-insignificant number of flags for poor play appear), the submission can move to the classical verification style using either private or closed-but-visible threads to discuss if it is submission material. This can be abused if trolls issue unwarranted flags or a group bias comes into play. Counters can be created to those, but any way it's set up a lot of other mechanisms that would have to be built into the site.

As for a general open verification system, my thoughts are that SDA guidelines for submission-quality videos would need to become more concrete in order to prevent an overwhelming number of submissions. The runners of today have a pretty good idea of what they envision to be "site material", but newcomers are probably not as keen on what it means to get the SDA stamp of approval. The site has come a long way since the mid-2000s, particularly in what it expects from players. This is one of the main things that I hope will be maintained.

On the leaderboards idea, I'm unsure if SDA should try to pick up that end. It sounds like Cosmo has something in mind for that, and frankly trying to mix in leaderboard tracking with regular submissions would be too chaotic to handle smoothly. It starts to get further away from SDA's mission as well. However, I think it would be great if there were some ways for the two sites to talk to each other. Even something as simple as providing a link to the specific leaderboard or being able to associate the leaderboard submission with the SDA submission would be great.

tl;dr:
-Using flags in open verification one possible route
-Bring problems upfront first rather than letting it go to a popularity contest/majority decision
-Solidify what it means to have a "submission-quality" run
-SDA and leaderboards could work well as separate but complementary entities, but both serve their own purpose

[quote="dunnius"]The idea of vote up/down for open verification is more useful as comments rather than a poll.  Even though TASVideos has a poll, it is not what judges use to make the decision, it is the comments that matter.  I don't see a poll being necessary for open verifications if there are good comments.[/quote]

Up/down voting for the individual comments, or the run itself? If the up/down votes are for the submission, I don't see how it's much different from regular voting or a popularity contest. The formula changes slightly for individual ranking of comments so that the most relevant insight rises to the top, but that assumes the system is not abused. Showing popularity is a reasonable thing, but I don't think it should be the method of verification.

[img]http://imgs.xkcd.com/comics/tornadoguard.png[/img]

Something I proposed last night was instead a flagging system. The more important aspect of verifications is that the minority be heard because that is what allows the more subtle problems to come to light. The current verification already does this: the verifiers are supposed to be knowledgeable in the specific game and watch closely for suspicious effects or poor play. Opening up the verifications to everybody means that this focus is no longer assumed, but any problems must still be brought up. Players that see suspicious activity or think that the video does not meet the standards for SDA can flag the video for further review.

This kind of system is not without pitfalls or abuses, however. This means that staff must respond to flags individually, which may increase the burden depending on the number of submissions/flags. If the flag's reasoning is valid and more investigation needs to be done (or a not-insignificant number of flags for poor play appear), the submission can move to the classical verification style using either private or closed-but-visible threads to discuss if it is submission material. This can be abused if trolls issue unwarranted flags or a group bias comes into play. Counters can be created to those, but any way it's set up a lot of other mechanisms that would have to be built into the site.

As for a general open verification system, my thoughts are that SDA guidelines for submission-quality videos would need to become more concrete in order to prevent an overwhelming number of submissions. The runners of today have a pretty good idea of what they envision to be "site material", but newcomers are probably not as keen on what it means to get the SDA stamp of approval. The site has come a long way since the mid-2000s, particularly in what it expects from players. This is one of the main things that I hope will be maintained.

On the leaderboards idea, I'm unsure if SDA should try to pick up that end. It sounds like Cosmo has something in mind for that, and frankly trying to mix in leaderboard tracking with regular submissions would be too chaotic to handle smoothly. It starts to get further away from SDA's mission as well. However, I think it would be great if there were some ways for the two sites to talk to each other. Even something as simple as providing a link to the specific leaderboard or being able to associate the leaderboard submission with the SDA submission would be great.

tl;dr:
-Using flags in open verification one possible route
-Bring problems upfront first rather than letting it go to a popularity contest/majority decision
-Solidify what it means to have a "submission-quality" run
-SDA and leaderboards could work well as separate but complementary entities, but both serve their own purpose

Flip

2012-11-09 03:21:52 pm

1-Up!

Quote from UraniumAnchor:

I'm hoping that once things settle out with the new queue a bit, the lag times between submission and posting will also drop by a lot. I've already seen the average verification time drop by a huge amount since more of the mundane tasks got automated.

Don't want to get into long response right now. Couldn't be around for this last night and still listening.

Thanks to UA, the submission process has been streamlined and we're starting to see the benefits. Verification is in the best place that it's ever been right now, and the rest is on the way. Hopefully in the near future we can get to the point where submission to posting time is consistently lower.

yggdrizzle

2012-11-09 03:26:34 pm

The turnaround time for verification is the main problem I see with SDA. If a runner submits a high quality video, commentary track, and comments, we simply wait for verifiers and the game page, right? For more popular games you could probably find verifiers that will do the job within a week. See if you can't find more verifiers if they can't get it done in this timeframe. The game page could also be written at the time of submission by the runner or the community for the game, then edited by other users/admins before it goes up.

Putting this work on the submitter/small group of verifiers and then having it reviewed by an admin before posting would probably make it easy to do. This example is definitely one of the simplest, but it applies for a lot of games: I don't see any reason why a new, live-streamed Super Mario World any% run shouldn't be up within a week of it being done (or pushed out at the next update) assuming the runner does their commentary/comments quickly. There's a ton of people who could verify it within a day. If one of the verifiers are slow, you have 10 qualified people idling in IRC who would be able to take the job. Streaming attemps is a good indicator of the run being legitimate, and a lot of runs don't really need to go through the lengthy verification process.

The popular vote for submission isn't a terrible idea, but it might open the door to a lot of problems. Streamlining verification makes a lot more sense.

I think that having regular updates is a really good idea. I never see the updates on SDA when they happen because I have no real reason to check. My bookmark for the site is the game list to easily find info.

Good idea: put a 'coming soon' section on the front page, with "game | time | runner | link", linking to the twitch archive or Youtube, if the runner has a recorded run they intend to submit.

I don't know what's up behind the scenes, but I do know that while verification times have decreased, they are still painfully long, and it's obvious to anyone outside of forum regulars.

The turnaround time for verification is the main problem I see with SDA. If a runner submits a high quality video, commentary track, and comments, we simply wait for verifiers and the game page, right? For more popular games you could probably find verifiers that will do the job within a week. See if you can't find more verifiers if they can't get it done in this timeframe. The game page could also be written at the time of submission by the runner or the community for the game, then edited by other users/admins before it goes up.

Putting this work on the submitter/small group of verifiers and then having it reviewed by an admin before posting would probably make it easy to do. This example is definitely one of the simplest, but it applies for a lot of games: I don't see any reason why a new, live-streamed Super Mario World any% run shouldn't be up within a week of it being done (or pushed out at the next update) assuming the runner does their commentary/comments quickly. There's a ton of people who could verify it within a day. If one of the verifiers are slow, you have 10 qualified people idling in IRC who would be able to take the job. Streaming attemps is a good indicator of the run being legitimate, and a lot of runs don't really need to go through the lengthy verification process.

The popular vote for submission isn't a terrible idea, but it might open the door to a lot of problems. Streamlining verification makes a lot more sense.

I think that having regular updates is a really good idea. I never see the updates on SDA when they happen because I have no real reason to check. My bookmark for the site is the game list to easily find info.

Good idea: put a 'coming soon' section on the front page, with "game | time | runner | link", linking to the twitch archive or Youtube, if the runner has a recorded run they intend to submit.

I don't know what's up behind the scenes, but I do know that while verification times have decreased, they are still painfully long, and it's obvious to anyone outside of forum regulars.

your_name_here

2012-11-09 03:36:40 pm

I am against the idea of public verification being a pure voting system. verifiers should be explaining what the run does that makes it good, or what it does that makes it bad, and it shouldn't matter how many people point out a specific merit or problem with the run. The SDA staff should be able to look at what verifiers have brought up, and use that information to make a decision regarding whether the run is of sufficient quality.

Surreal

Edit history:

Surreal: 2012-11-09 04:05:14 pm

Surreal: 2012-11-09 04:04:18 pm

Surreal: 2012-11-09 04:01:43 pm

2012-11-09 03:51:26 pm

dabes

My views on verification: Pretty much the same as everyone else's. Though part of it lies in finding the verifiers and actually getting them to leave their feedback, the wait after everyone's given their vote is still really long. The obvious answer seems to be to increase the amount of "staff" members who can work on this whole process, since as far as I'm aware there's <10 people really running the whole site, I'm surprised no one really brought up this suggestion. As for the verification process, I brought this up during mike's stream (though I didn't get to stay long unfortunately and this probably got drowned out by the swarm of comments), but I feel like a good way to speed up verification would be to just simplify what a verifier needs to do: Did they cheat, are audio and video good, what's the time, Accept or Decline. The need for comments on the run seems like a pointless addition that only makes the verification process longer. I figure most people don't care to read what the verifiers' thoughts on every single section/level are (let alone the fact that some people bring out their inner Steven King and have to write a novel of notes first), not to mention that you can't even tell who is giving the feedback.

On Submitting: I'm like a lot of runners in that I'm holding back due to just wanting to make sure my submission will count, so that it either won't be beaten, or if it is improved by me or someone else, it will either take a long time after my submission and/or it will be a very small improvement (unless some major tricks/breaks were found). A big problem is that there are a lot of games that are optimizing too rapidly (routes, tricks found, optimizations, etc), and thus it's hard to ever work towards a specific goal, as that goal keeps changing in itself. Not to mention many runners tend to reach their goal times for submission, then realize they still have a bunch of mistakes and can keep improving (I've gone through this like 10 times with Spyro personally). As everyone has said, having a more swift service in having runs put up would motivate runners more to submit and just submit improvements later.

Another thing I feel is an issue is ingame timer vs real time (or real time with SDA timing). There are very few games that have completely accurate ingame timers (let alone ones that show seconds, which is a huge issue for most games that are ran on an optimized level), and Sonic is the only series that I can name that gets screwed over by not using it. Anyways, I won't go too in detail over that, but the reason I'm bringing it up is because there are a lot of games on here that use ingame time (usually unnecessarily other than laziness to manually time), that discourages people from submitting their runs as most runners go for real time and use real time strategies. Runners don't want to intentionally slow themselves down to submit a run to SDA, nor have their run be replaced by someone's run that was slower but had a faster ingame time. It also contributes to runners not submitting improvement runs, as it means they'll have to beat any run that's up by at least a minute (according to ingame timer, which may mean even more than a minute), which can be either impossible or really difficult for some games (Super Metroid was/is a prime example of this). There needs to be a change of using only real time (with SDA's timing), unless proven to negatively affect the run such as for Sonic.

On video quality: I personally prefer and enjoy SDA's quality standards. Seeing a clean, well encoded run with good game sound (and nothing else) makes for an enjoyable experience. As was said earlier, it's sort of like watching a movie. While I don't mind seeing good runs on twitch recordings, listening to the runner's commentary or discussions with little context to the run, as well as having the quiet game sound and perhaps other things (people in a skype call being dumb, dogs barking in background, etc), can kill the experience. I also feel that having the high quality standards helps with SDA's professional appearance, which I feel benefits the site when marathons are held. When people are introduced to speedrunning and see these wonders, we can say "We are SDA. If you want to learn more about speedrunning, come check out our site", as opposed to "We're a bunch of guys just getting together, you can find us all over Twitch".

On leaderboards: While such a thing can be nice, I feel like most people would only care to look at the top 3-5 times for any given game, just to see the best runs, and how they compare against each other. Runners will probably enjoy the idea of leaderboards more than someone watching runs as they can get that competitive satisfaction of "Yeah! I'm 4th best in the world!" (Humble as a lot of us may be, I'm sure almost all of us allow this guilty pleasure). A leaderboard system would also sort of need to replace the system SDA has for hosting runs if it existed, otherwise it would seem pointless to have any runs on the site if there's a leaderboard right next to it saying "this run has been beaten by a minute, the run in question wasn't submitted to our site".

On emulators: Too many inconsistencies between emulators, people's PC specs, plugins, difficulty verifying a run if it wasn't streamed, inaccurate emulation, advantages such as load times. Just...no lol

On SDA and SRL: <3 you all

My views on verification: Pretty much the same as everyone else's. Though part of it lies in finding the verifiers and actually getting them to leave their feedback, the wait after everyone's given their vote is still really long. The obvious answer seems to be to increase the amount of "staff" members who can work on this whole process, since as far as I'm aware there's <10 people really running the whole site, I'm surprised no one really brought up this suggestion. As for the verification process, I brought this up during mike's stream (though I didn't get to stay long unfortunately and this probably got drowned out by the swarm of comments), but I feel like a good way to speed up verification would be to just simplify what a verifier needs to do: Did they cheat, are audio and video good, what's the time, Accept or Decline. The need for comments on the run seems like a pointless addition that only makes the verification process longer. I figure most people don't care to read what the verifiers' thoughts on every single section/level are (let alone the fact that some people bring out their inner Steven King and have to write a novel of notes first), not to mention that you can't even tell who is giving the feedback.

On Submitting: I'm like a lot of runners in that I'm holding back due to just wanting to make sure my submission will count, so that it either won't be beaten, or if it is improved by me or someone else, it will either take a long time after my submission and/or it will be a very small improvement (unless some major tricks/breaks were found). A big problem is that there are a lot of games that are optimizing too rapidly (routes, tricks found, optimizations, etc), and thus it's hard to ever work towards a specific goal, as that goal keeps changing in itself. Not to mention many runners tend to reach their goal times for submission, then realize they still have a bunch of mistakes and can keep improving (I've gone through this like 10 times with Spyro personally). As everyone has said, having a more swift service in having runs put up would motivate runners more to submit and just submit improvements later.

Another thing I feel is an issue is ingame timer vs real time (or real time with SDA timing). There are very few games that have completely accurate ingame timers (let alone ones that show seconds, which is a huge issue for most games that are ran on an optimized level), and Sonic is the only series that I can name that gets screwed over by not using it. Anyways, I won't go too in detail over that, but the reason I'm bringing it up is because there are a lot of games on here that use ingame time (usually unnecessarily other than laziness to manually time), that discourages people from submitting their runs as most runners go for real time and use real time strategies. Runners don't want to intentionally slow themselves down to submit a run to SDA, nor have their run be replaced by someone's run that was slower but had a faster ingame time. It also contributes to runners not submitting improvement runs, as it means they'll have to beat any run that's up by at least a minute (according to ingame timer, which may mean even more than a minute), which can be either impossible or really difficult for some games (Super Metroid was/is a prime example of this). There needs to be a change of using only real time (with SDA's timing), unless proven to negatively affect the run such as for Sonic.

On video quality: I personally prefer and enjoy SDA's quality standards. Seeing a clean, well encoded run with good game sound (and nothing else) makes for an enjoyable experience. As was said earlier, it's sort of like watching a movie. While I don't mind seeing good runs on twitch recordings, listening to the runner's commentary or discussions with little context to the run, as well as having the quiet game sound and perhaps other things (people in a skype call being dumb, dogs barking in background, etc), can kill the experience. I also feel that having the high quality standards helps with SDA's professional appearance, which I feel benefits the site when marathons are held. When people are introduced to speedrunning and see these wonders, we can say "We are SDA. If you want to learn more about speedrunning, come check out our site", as opposed to "We're a bunch of guys just getting together, you can find us all over Twitch".

On leaderboards: While such a thing can be nice, I feel like most people would only care to look at the top 3-5 times for any given game, just to see the best runs, and how they compare against each other. Runners will probably enjoy the idea of leaderboards more than someone watching runs as they can get that competitive satisfaction of "Yeah! I'm 4th best in the world!" (Humble as a lot of us may be, I'm sure almost all of us allow this guilty pleasure). A leaderboard system would also sort of need to replace the system SDA has for hosting runs if it existed, otherwise it would seem pointless to have any runs on the site if there's a leaderboard right next to it saying "this run has been beaten by a minute, the run in question wasn't submitted to our site".

On emulators: Too many inconsistencies between emulators, people's PC specs, plugins, difficulty verifying a run if it wasn't streamed, inaccurate emulation, advantages such as load times. Just...no lol

On SDA and SRL: <3 you all

Hsanrb

2012-11-09 04:43:07 pm

Quote from Surreal:

Another thing I feel is an issue is ingame timer vs real time (or real time with SDA timing). There are very few games that have completely accurate ingame timers (let alone ones that show seconds, which is a huge issue for most games that are ran on an optimized level), and Sonic is the only series that I can name that gets screwed over by not using it. Anyways, I won't go too in detail over that, but the reason I'm bringing it up is because there are a lot of games on here that use ingame time (usually unnecessarily other than laziness to manually time), that discourages people from submitting their runs as most runners go for real time and use real time strategies. Runners don't want to intentionally slow themselves down to submit a run to SDA, nor have their run be replaced by someone's run that was slower but had a faster ingame time. It also contributes to runners not submitting improvement runs, as it means they'll have to beat any run that's up by at least a minute (according to ingame timer, which may mean even more than a minute), which can be either impossible or really difficult for some games (Super Metroid was/is a prime example of this). There needs to be a change of using only real time (with SDA's timing), unless proven to negatively affect the run such as for Sonic.

I think unless the in game timer is unreliable that it should be used. If a game has IF's, usually those help keep your run in line. Real-time is most likely to disguise fractions and wrap those fractions into seconds. Those who use timers care about those tenths, when in reality those tenths can be lost by just spliting later/earlier. Heck some RTA measurements can be gained/lost based exclusively on how its recorded (verifiers vs me came up with a 4s loss/gain, my Mean Bean RTA has 5 mysterious seconds added after encoding etc.) It would make a run improvement that can be made/lost exclusively on technical recording rather than on mechanical merit.

Also Leaderboards put too much emphasis on the result of a good run, not the journey of acquiring a run. Watching a run improvement personally is more satisfying than the bulk of the improvement. Streaming has just put a face to the journey of starting a game to SDA submission (if they don't drop it early.) It would be great for SRL when I tag a game, they could pull up stats like "Oh Hsan is playing Ristar (Any% bleh, 100% 38:41, 100% hard, 42:0X)" change game, flag new stats "Hsan Landstalker, Mir Tower 53 somethin, any% 2:31) and could compare contrast to fastest run of people on the site. But it also promotes a competitive atmosphere to those who not always want it. I hate reseting, so most of my good runs have some error in them (I even made a point to make Reset a censored word in chat.)

The SDA/SRL rift? I think the runners are less of a rift then the Viewers of a runner. Sure some runners have views of other runners in a positive/negative light, but that will always happen in groups. Not everyone can like everyone, ranging from stream content to attitude to the hobby (Casual VS competitive) Things people do will always rub someone the wrong way (Personally I hate those who greet the lurkers into irc, or have extra stuff around the game +/- timer.) The viewers however can create a rift the runners won't perceive. SRL viewers want to see WR's or are generally game focused. Ever notice that spike of 3x-5x when someone either has lots of green on their splits, or word spreads on a memorable milestone for their game falling. Chances are you won't see those viewers again unless you trim off time in a run in progress. People who watch through wooty are more likely to watch around a runner. I'll always watch Uyama, or Axon and more likely to hang out regardless what the runner is playing.

I had more, but the idea came and gone like the wind.

[quote="Surreal"] Another thing I feel is an issue is ingame timer vs real time (or real time with SDA timing). There are very few games that have completely accurate ingame timers (let alone ones that show seconds, which is a huge issue for most games that are ran on an optimized level), and Sonic is the only series that I can name that gets screwed over by not using it. Anyways, I won't go too in detail over that, but the reason I'm bringing it up is because there are a lot of games on here that use ingame time (usually unnecessarily other than laziness to manually time), that discourages people from submitting their runs as most runners go for real time and use real time strategies. Runners don't want to intentionally slow themselves down to submit a run to SDA, nor have their run be replaced by someone's run that was slower but had a faster ingame time. It also contributes to runners not submitting improvement runs, as it means they'll have to beat any run that's up by at least a minute (according to ingame timer, which may mean even more than a minute), which can be either impossible or really difficult for some games (Super Metroid was/is a prime example of this). There needs to be a change of using only real time (with SDA's timing), unless proven to negatively affect the run such as for Sonic. [/quote]

I think unless the in game timer is unreliable that it should be used. If a game has IF's, usually those help keep your run in line. Real-time is most likely to disguise fractions and wrap those fractions into seconds. Those who use timers care about those tenths, when in reality those tenths can be lost by just spliting later/earlier. Heck some RTA measurements can be gained/lost based exclusively on how its recorded (verifiers vs me came up with a 4s loss/gain, my Mean Bean RTA has 5 mysterious seconds added after encoding etc.) It would make a run improvement that can be made/lost exclusively on technical recording rather than on mechanical merit.

Also Leaderboards put too much emphasis on the result of a good run, not the journey of acquiring a run. Watching a run improvement personally is more satisfying than the bulk of the improvement. Streaming has just put a face to the journey of starting a game to SDA submission (if they don't drop it early.) It would be great for SRL when I tag a game, they could pull up stats like "Oh Hsan is playing Ristar (Any% bleh, 100% 38:41, 100% hard, 42:0X)" change game, flag new stats "Hsan Landstalker, Mir Tower 53 somethin, any% 2:31) and could compare contrast to fastest run of people on the site. But it also promotes a competitive atmosphere to those who not always want it. I hate reseting, so most of my good runs have some error in them (I even made a point to make Reset a censored word in chat.)

The SDA/SRL rift? I think the runners are less of a rift then the Viewers of a runner. Sure some runners have views of other runners in a positive/negative light, but that will always happen in groups. Not everyone can like everyone, ranging from stream content to attitude to the hobby (Casual VS competitive) Things people do will always rub someone the wrong way (Personally I hate those who greet the lurkers into irc, or have extra stuff around the game +/- timer.) The viewers however can create a rift the runners won't perceive. SRL viewers want to see WR's or are generally game focused. Ever notice that spike of 3x-5x when someone either has lots of green on their splits, or word spreads on a memorable milestone for their game falling. Chances are you won't see those viewers again unless you trim off time in a run in progress. People who watch through wooty are more likely to watch around a runner. I'll always watch Uyama, or Axon and more likely to hang out regardless what the runner is playing.

I had more, but the idea came and gone like the wind.

nate

2012-11-09 04:50:54 pm

Quote from CosmoWright:

the KB seems to be flooded with spam and what I'm envisioning wouldn't use wiki software but rather custom code + its own database.

i agree with the second part, but why does the first part matter? can you see any evidence of spam if you don't look at the recent changes page? i'd love to solve this problem but i honestly have no idea how.

yggdrizzle

2012-11-09 04:53:37 pm

Quote from Hsanrb:

I think unless the in game timer is unreliable that it should be used. If a game has IF's, usually those help keep your run in line. Real-time is most likely to disguise fractions and wrap those fractions into seconds. Those who use timers care about those tenths, when in reality those tenths can be lost by just spliting later/earlier. Heck some RTA measurements can be gained/lost based exclusively on how its recorded (verifiers vs me came up with a 4s loss/gain, my Mean Bean RTA has 5 mysterious seconds added after encoding etc.) It would make a run improvement that can be made/lost exclusively on technical recording rather than on mechanical merit.

This isn't really true. You can always re-time real time accurately. The 'mysterious seconds' have already been explained. I'm not sure what you're getting at with this splits. Most real time vs game time cases come down to menus or routing and what run is actually faster. I think most people would agree real-time is the more impressive/difficult category in games where this makes a difference (Super Metroid, Mega Man 9/10), unless the extra time is just cosmetic (Sonic).

[quote="Hsanrb"]I think unless the in game timer is unreliable that it should be used. If a game has IF's, usually those help keep your run in line. Real-time is most likely to disguise fractions and wrap those fractions into seconds. Those who use timers care about those tenths, when in reality those tenths can be lost by just spliting later/earlier. Heck some RTA measurements can be gained/lost based exclusively on how its recorded (verifiers vs me came up with a 4s loss/gain, my Mean Bean RTA has 5 mysterious seconds added after encoding etc.) It would make a run improvement that can be made/lost exclusively on technical recording rather than on mechanical merit.
[/quote]

This isn't really true. You can always re-time real time accurately. The 'mysterious seconds' have already been explained. I'm not sure what you're getting at with this splits. Most real time vs game time cases come down to menus or routing and what run is actually faster. I think most people would agree real-time is the more impressive/difficult category in games where this makes a difference (Super Metroid, Mega Man 9/10), unless the extra time is just cosmetic (Sonic).

Alko

2012-11-09 04:54:43 pm

Caution: This user contains Kana ^_^

While I don't think, the amount of people working on/with/amoung/whatever the submission process is close to the 'too many cooks spoil the broth' limit, I also don't think, adding more people to it would speed it up in any way, save times when someone's on holiday or in hospital.

I also have a feeling, that there is way too much lost time between the run being verified and accepted and actually being posted. While I haven't checked this, I assume a run doesn't actually get published on the game page until when or shortly before when they are posted on the front page. For those runs I follow (i.e. I like the game) they are only published ages after the verification is through, and as somebody way ahead said: I have no idea, what would still need to happen in between. Encoding is the only sensible thing coming to mind.

Somebody mentioned Sonic realtiming to be inaccurate; I myself haven't played that game, so I'ld be very happy if you could explain (or link the threads which explain) =3 Occasions where shorter ingame time leads to longer realtime should probably be realtime timed imho, but again, I can't think of why that would actually happen.

I wish I knew how to address the problem of many runs just not being submitted, because while they might be awesome, they will be improved within weeks and the runner doesn't feel comfortable enough with them. But since I haven't submitted yet, I don't know what makes makes people submit or keeps them from it, nor what might make them submit even though they don't atm.

I'm agreeing with other people on video quality (don't let it decline), and the question of public verificaion (SDA ain't broke, so why fix it?). I'm not sure what I think about leaderboards. Not going to comment on perceived or actual rifts between SDA and other sites either.

If I missed something, I'll add that later

While I don't think, the amount of people working on/with/amoung/whatever the submission process is close to the 'too many cooks spoil the broth' limit, I also don't think, adding more people to it would speed it up in any way, save times when someone's on holiday or in hospital.

I also have a feeling, that there is way too much lost time between the run being verified and accepted and actually being posted. While I haven't checked this, I assume a run doesn't actually get published on the game page until when or shortly before when they are posted on the front page. For those runs I follow (i.e. I like the game) they are only published ages after the verification is through, and as somebody way ahead said: I have no idea, what would still need to happen in between. Encoding is the only sensible thing coming to mind.

Somebody mentioned Sonic realtiming to be inaccurate; I myself haven't played that game, so I'ld be very happy if you could explain (or link the threads which explain) =3 Occasions where shorter ingame time leads to longer realtime should probably be realtime timed imho, but again, I can't think of why that would actually happen.

I wish I knew how to address the problem of many runs just not being submitted, because while they might be awesome, they will be improved within weeks and the runner doesn't feel comfortable enough with them. But since I haven't submitted yet, I don't know what makes makes people submit or keeps them from it, nor what might make them submit even though they don't atm.

I'm agreeing with other people on video quality (don't let it decline), and the question of public verificaion (SDA ain't broke, so why fix it?).  I'm not sure what I think about leaderboards. Not going to comment on perceived or actual rifts between SDA and other sites either.

If I missed something, I'll add that later

Surreal

2012-11-09 05:00:21 pm

dabes

Dragondarch

2012-11-09 05:00:50 pm

Pudding%

Quote from Alko:

Somebody mentioned Sonic realtiming to be inaccurate; I myself haven't played that game, so I'ld be very happy if you could explain (or link the threads which explain) =3 Occasions where shorter ingame time leads to longer realtime should probably be realtime timed imho, but again, I can't think of why that would actually happen.

In the case of Sonic, having a faster in-game time causes a MUCH longer points countdown at the end of a stage. For Sonic 1/2, it's 50,000 points if under 30 sec, but only 10,000 for 30-59 (Not 100% sure on exact time and point values...someone correct me if I'm wrong here). So what happens is that getting a 0:29 would be significantly slower real time than a 0:30 thanks to that countdown.

Golden

2012-11-09 05:05:41 pm

Voice

Obvious conclusion: Don't speedrun Sonic games since the developers didn't intend you to.

Wink

usernameguy

2012-11-09 05:10:21 pm

FWIW, there is a solid solution to the "the problem with averaging star ratings". It just takes a bit of math. I implemented it pretty successfully at another site.

Here's the original proposal.

http://www.evanmiller.org/how-not-to-sort-by-average-rating.html

Even if you don't code it up, read it if you're interested in math or programming. I think it's a pretty interesting idea. And you can pretty easily find on Google implementations for this with variations of "Wilson score star ratings" search strings.

Darkwing Duck

2012-11-09 06:22:51 pm

Highly Evolved

Quote from Flip:

Hopefully in the near future we can get to the point where submission to posting time is consistently lower.

Would it be possible to give an estimate on how much lower? Six months to five months wouldn't seem to appease the issues with the long gap between completion of a run and posting of a run, while six to, say, three would be a significant dent.

dunnius

Edit history:

dunnius: 2012-11-09 06:58:33 pm

dunnius: 2012-11-09 06:41:34 pm

2012-11-09 06:36:23 pm

Intruding N313 and F014

Quote from Omnigamer:

Quote from dunnius:

The idea of vote up/down for open verification is more useful as comments rather than a poll. Even though TASVideos has a poll, it is not what judges use to make the decision, it is the comments that matter. I don't see a poll being necessary for open verifications if there are good comments.

Up/down voting for the individual comments, or the run itself?

For the run. I mean that a poll is not necessary because the comments and reasons mean far more than any number. Also, the end decision should not made by popular vote, it should be made by a judge(admin) whose decision is based solely on the comments, which avoids spammed votes, etc.

Rakuen

2012-11-09 06:57:41 pm

Weegee Time

Lots and lots of stuff in this thread. I've read most, but not all, so apologies if I retread some ground.

I like the idea of public verification for a different reason than just speeding up the process. I've verified a whopping one run on this site. Impressive, I know. It was Mega Man X4, a game I know very well. I always feel daunted whenever I think of verifying something. I'm not terrible at games by any means, but I'm rarely the best at them either. What if I don't know enough? What if I miss something? Perhaps I over think it, but it's my personality. It's a difficult beast to tame. I imagine a lot of new users feel a similar way to a lesser extent. I like that SDA has high standards yet those same high standards can have an insulating effect. We're not just an archive site. We're a community. It's important we work to build that community. Making verification more public can help the process continue along. It would allow the new and average users to play a role in the process, or at the very least feel like they play a role.

As for SDA, SRL, whatever other sites and systems you want to bring into this, I think there's one thing which hasn't been described yet. Speedrunning is in a bubble right now. Thanks to Twitch and to a lot of word-of-mouth and publicity, speedrunning is "a thing to do". That's good. Really, it is. Be careful though. Bubbles have a nasty habit of bursting, or at the very least contracting, over time. You can't just keep building upwards and outwards if you want to ensure it survives. You also have to continue building a firm foundation for everything to sit on. If you think I'm illustrating the difference between SRL and SDA, you are correct. I see SRL continuing on as the bleeding edge of speedrunning, while SDA plays the vital role of keeping track of the mess.

I'm not just talking about quality speedruns either. I'm talking about quality information. The site counter says we have 779 games on the site. We only have 80 games listed on the KB index. Some simple math will tell you only 10% of the games are documented. A lot of the games have some good and even extensive run comments. That's great. But even run comments can gloss over the nuance in the game.

To illustrate, my job suffers from the "hit by a bus" problem. If any one person on my team were suddenly incapacitated by, say, getting hit by a bus, we would hurt in a big way. We have overlap in our duties and knowledge of other systems, but only one or two people can be considered "experts" on any given system. The depth of knowledge is just as important as the breadth of knowledge. Speedrunning is not quite as serious business as a Fortune 50 corporation (well, maybe to some people...). Even so, if a big player suddenly disappears where does that leave the state of his game(s)? That's knowledge and experience that has gone into the ether. You can get some or even all of it back given enough time. Still, it's a lot more productive to not lose it at all.

I'm not saying SDA shouldn't look at ways to improve it's turn around time. We should always strive to improve it. Rather, I'm saying we shouldn't try to make SDA more like SRL. It's not that, it's never been that. It's an archive. It's right in the name. What does an archive do? It preserves and provides knowledge. My vision is for SDA to be the #1 stop for knowledge in the speedrunning community.

If we work together across all the communities, when the bubble bursts or contracts or whatever happens, the tower and the foundation will remain sound.

And if you've read this far, well, thanks for humoring me. Smiley

Lots and lots of stuff in this thread.  I've read most, but not all, so apologies if I retread some ground.

I like the idea of public verification for a different reason than just speeding up the process.  I've verified a whopping one run on this site.  Impressive, I know.  It was Mega Man X4, a game I know very well.  I always feel daunted whenever I think of verifying something.  I'm not terrible at games by any means, but I'm rarely the best at them either.  What if I don't know enough?  What if I miss something?  Perhaps I over think it, but it's my personality.  It's a difficult beast to tame.  I imagine a lot of new users feel a similar way to a lesser extent.  I like that SDA has high standards yet those same high standards can have an insulating effect.  We're not just an archive site.  We're a community.  It's important we work to build that community.  Making verification more public can help the process continue along.  It would allow the new and average users to play a role in the process, or at the very least [i]feel[/i] like they play a role.

As for SDA, SRL, whatever other sites and systems you want to bring into this, I think there's one thing which hasn't been described yet.  Speedrunning is in a bubble right now.  Thanks to Twitch and to a lot of word-of-mouth and publicity, speedrunning is "a thing to do".  That's good.  Really, it is.  Be careful though.  Bubbles have a nasty habit of bursting, or at the very least contracting, over time.  You can't just keep building upwards and outwards if you want to ensure it survives.  You also have to continue building a firm foundation for everything to sit on.  If you think I'm illustrating the difference between SRL and SDA, you are correct.  I see SRL continuing on as the bleeding edge of speedrunning, while SDA plays the vital role of keeping track of the mess.

I'm not just talking about quality speedruns either.  I'm talking about quality [i]information[/i].  The site counter says we have 779 games on the site.  We only have 80 games listed on the KB index.  Some simple math will tell you only 10% of the games are documented.  A lot of the games have some good and even extensive run comments.  That's great.  But even run comments can gloss over the nuance in the game.

To illustrate, my job suffers from the "hit by a bus" problem.  If any one person on my team were suddenly incapacitated by, say, getting hit by a bus, we would hurt in a big way.  We have overlap in our duties and knowledge of other systems, but only one or two people can be considered "experts" on any given system.  The depth of knowledge is just as important as the breadth of knowledge.  Speedrunning is not quite as serious business as a Fortune 50 corporation (well, maybe to some people...).  Even so, if a big player suddenly disappears where does that leave the state of his game(s)?  That's knowledge and experience that has gone into the ether.  You can get some or even all of it back given enough time.  Still, it's a lot more productive to not lose it at all.

I'm not saying SDA shouldn't look at ways to improve it's turn around time.  We should [i]always[/i] strive to improve it.  Rather, I'm saying we shouldn't try to make SDA more like SRL.  It's not that, it's never been that.  It's an archive.  It's right in the name.  What does an archive do?  It preserves and provides knowledge.  My vision is for SDA to be the #1 stop for [i]knowledge[/i] in the speedrunning community.

If we work together across all the communities, when the bubble bursts or contracts or whatever happens, the tower [i]and[/i] the foundation will remain sound.

And if you've read this far, well, thanks for humoring me. :)

dunnius

2012-11-09 07:26:19 pm

Intruding N313 and F014

Quote from Rakuen:

My vision is for SDA to be the #1 stop for knowledge in the speedrunning community.

There is also TASVideos, which also has information that can be used for a speedrun. Sometimes it will only be there because that is where runners got the info, or in my case, I TASed the games first, and I don't feel like duplicating all the info that is scattered in the forum posts.

Quote from Rakuen:

Even so, if a big player suddenly disappears where does that leave the state of his game(s)?

That has happened for the ActRaiser TAS. Fortunately, there is enough info left behind to understand what was done.

Naegleria

(user is banned)

Naegleria

Edit history:

Naegleria: 2012-11-09 07:34:49 pm

Naegleria: 2012-11-09 07:34:34 pm

Naegleria: 2012-11-09 07:34:26 pm

2012-11-09 07:31:18 pm

Quote from Omnigamer:

Even something as simple as providing a link to the specific leaderboard or being able to associate the leaderboard submission with the SDA submission would be great.
-SDA and leaderboards could work well as separate but complementary entities, but both serve their own purpose

5 stars, having SDA and SRL (or whatever Cosmo is working on) being brother sites having each game page on one site tied to the other and filling their own niche (which is pretty much what the sites do right now, this topic is about refining or redefining what that niche is).

Also, everything else Omnigamer said was pretty sound and answered most of the questions asked in this thread, I just picked this quote out for better representing what I was trying to say.

Freezard

2012-11-09 07:32:31 pm

Rakuen I don't see how that's a problem with SDA though, the comment option and wiki are already there, it's up to the speedrunners to decide what information they want to provide. This is one thing other than the high quality videos I really like with SDA because you can find information about a lot of games in either the comments, forum or wiki. I personally have written articles in the wiki for all three games I've submitted just for the documentation purpose since unfortunately any comments are removed from SDA if the run is obsoleted. It kind of annoys me when people write troll comments and I'm left with no in-depth knowledge of a game I want to watch, and nowadays most people only comment while playing on stream cause it doesn't require as much effort as putting it down in a text format. Sadly this is just the direction media has been heading towards the last few years; instead of looking up something in a game FAQ when you're stuck, you have to find YT walkthroughs and spend minutes trying to find the actual information you're looking for... it has nothing to do with SDA really.

Flip

2012-11-09 08:01:47 pm

1-Up!

Quote from Darkwing Duck:

Quote from Flip:

Hopefully in the near future we can get to the point where submission to posting time is consistently lower.

Would it be possible to give an estimate on how much lower? Six months to five months wouldn't seem to appease the issues with the long gap between completion of a run and posting of a run, while six to, say, three would be a significant dent.

Any estimation i could give you would be pretty arbitrary, tbh. That's why I'm not jumping out here with bold promises. My goal would be closer to 3 months, but only time will tell. As always, how long a run takes to get verified plays a huge part, and nearly all of that is out of my hands.

Rakuen

2012-11-09 08:07:58 pm

Weegee Time

Quote from dunnius:

There is also TASVideos, which also has information that can be used for a speedrun. Sometimes it will only be there because that is where runners got the info, or in my case, I TASed the games first, and I don't feel like duplicating all the info that is scattered in the forum posts.

Ayep, you're right. I totally overlooked TASVideos in my post. Sorry about that.

Quote from Freezard:

Rakuen I don't see how that's a problem with SDA though, the comment option and wiki are already there, it's up to the speedrunners to decide what information they want to provide. ... Sadly this is just the direction media has been heading towards the last few years; instead of looking up something in a game FAQ when you're stuck, you have to find YT walkthroughs and spend minutes trying to find the actual information you're looking for... it has nothing to do with SDA really.

But why does it have to be that way? You're right, it's not a problem with SDA specifically. Further, it is up to the people to share their knowledge. But why don't we all work together to make SDA that source? Or if not at SDA, at least put it all somewhere. I think it'd be a lot better for the community to have a one-stop-shop than to make people search all over creation for something.

[quote="dunnius"]There is also TASVideos, which also has information that can be used for a speedrun.  Sometimes it will only be there because that is where runners got the info, or in my case, I TASed the games first, and I don't feel like duplicating all the info that is scattered in the forum posts.[/quote]
Ayep, you're right.  I totally overlooked TASVideos in my post.  Sorry about that.

[quote="Freezard"]Rakuen I don't see how that's a problem with SDA though, the comment option and wiki are already there, it's up to the speedrunners to decide what information they want to provide. ... Sadly this is just the direction media has been heading towards the last few years; instead of looking up something in a game FAQ when you're stuck, you have to find YT walkthroughs and spend minutes trying to find the actual information you're looking for... it has nothing to do with SDA really.[/quote]
But why does it have to be that way?  You're right, it's not a problem with SDA specifically.  Further, it [i]is[/i] up to the people to share their knowledge.  But why don't we all work together to make SDA that source?  Or if not at SDA, at least put it all somewhere.  I think it'd be a lot better for the community to have a one-stop-shop than to make people search all over creation for something.

AlecK47

Edit history:

AlecK47: 2012-11-09 08:18:59 pm

2012-11-09 08:13:43 pm

I have to agree with Rakuen that knowledge is an essential part of SDA, and also with everyone who mentioned that SDA and SRL/Twitch each serve their own unique role in speedrunning. The fact that each place contributes something different to speedrunning in general seems like a good and healthy thing to me.

I also believe that the matter of lowering SDA's standards (say what you will, stream video still isn't as good as it could be) would be a significant mistake, although a separate section for such things does seem to be a workable solution. Some people may make fun of SDA for the way it does things, but that's par for the course when high standards are part of the MO. The main problem is the delay, and that's being worked on.

On that note, I do have a potential suggestion. I may be wrong, but do the updaters/other admins have to go through all the final encodes after verification? If so, then that seems an ideal place for "open verification" in the sense that anyone could verify a video's quality, even if they know little to nothing about the game in question.

There was another suggestion I wanted to make, but I forgot what it was while I was writing this... hopefully I can remember soon.

Edit: Remembered as soon as I hit post. Figures. SDA has the tagline "Playing though games quickly, skillfully and legitimately" on the main page, but it doesn't have a "mission statement" or easily accessed explanation of what the site is beyond that. It may be a touch pretentious, but it would help explain the site and its goals to people who may otherwise not know. Calling it something else may be recommended. In fact, a general "about SDA" section with history, standards, etc. would be a nice addition to the front page, imho. There are certainly enough articulate people here to make it happen.

I have to agree with Rakuen that knowledge is an essential part of SDA, and also with everyone who mentioned that SDA and SRL/Twitch each serve their own unique role in speedrunning.  The fact that each place contributes something different to speedrunning in general seems like a good and healthy thing to me.

I also believe that the matter of lowering SDA's standards (say what you will, stream video still isn't as good as it could be) would be a significant mistake, although a separate section for such things does seem to be a workable solution.  Some people may make fun of SDA for the way it does things, but that's par for the course when high standards are part of the MO.  The main problem is the delay, and that's being worked on.

On that note, I do have a potential suggestion.  I may be wrong, but do the updaters/other admins have to go through all the final encodes after verification?  If so, then that seems an ideal place for "open verification" in the sense that anyone could verify a video's quality, even if they know little to nothing about the game in question.

There was another suggestion I wanted to make, but I forgot what it was while I was writing this... hopefully I can remember soon.

Edit:  Remembered as soon as I hit post.  Figures.  SDA has the tagline "Playing though games quickly, skillfully and legitimately" on the main page, but it doesn't have a "mission statement" or easily accessed explanation of what the site is beyond that.  It may be a touch pretentious, but it would help explain the site and its goals to people who may otherwise not know.  Calling it something else may be recommended.  In fact, a general "about SDA" section with history, standards, etc. would be a nice addition to the front page, imho.  There are certainly enough articulate people here to make it happen.