Continuation of Roundtable Discussion

2012-11-14 08:49:57 pm

Weegee Time

On the other hand, verifiers can answer each other's questions about the run and debate any special circumstances the run may create. There's pros and cons to everything, and I'd say that's more of a pro than a con.

PJ

2012-11-14 09:01:16 pm

Is PJ

I agree that seeing the other responses is more of a good thing than a bad thing. Speaking personally, if I didn't see any other responses in the thread, I'd more likely to have the "nobody else verified so I've still got some time before I do" mentality. Getting pressure from the other verifiers is good. Tongue

But yea, I've had times when I don't agree with the other verifiers and I didn't let it affect me at all. IMO it's much better to have that discussion; several of the verifications have some really productive back-and-forth discussion.

Darkwing Duck

2012-11-14 10:16:01 pm

Highly Evolved

Quote from PJ:

I agree that seeing the other responses is more of a good thing than a bad thing. Speaking personally, if I didn't see any other responses in the thread, I'd more likely to have the "nobody else verified so I've still got some time before I do" mentality. Getting pressure from the other verifiers is good. Tongue

But yea, I've had times when I don't agree with the other verifiers and I didn't let it affect me at all. IMO it's much better to have that discussion; several of the verifications have some really productive back-and-forth discussion.

If that's the case, wouldn't public verification handle that even better?

Serris

2012-11-14 11:42:10 pm

Why not make verifiers submit their first impressions separately to the admin and open discussion among them (and reveal their identities to each other) afterwards, with all of the first impressions being posted by the admin as the thread opener?

presjpolk

2012-11-14 11:44:22 pm

HELLO!

Is adding more complexity to the verification process something that is really desired?

Paraxade

2012-11-15 07:03:28 am

Not saying anonymity is bad, more just.. pointless. I rarely ever see verifiers who care whether they stay anonymous or not. Hell, most of the time verifiers are fine revealing their identities before the verification even happens.

Verifiers who reject runs aren't doing it just to be dicks, they just believe in SDA's gameplay quality standards and don't think the run meets them. I'd imagine most of the time they'd be happy to see a new run that does, which is why rejects typically have a lot of really good constructive feedback. So for them I really can't imagine that being anonymous matters.

I dunno, it's obviously not a big deal either way. It just makes sense to me to do it this way, and then allow verifiers to be anonymous on request for the odd verifier that does care about anonymity, the same way verifiers can request not to have their responses publicly posted.

UraniumAnchor

Edit history:

UraniumAnchor: 2012-11-15 11:35:53 am

UraniumAnchor: 2012-11-15 07:14:01 am

UraniumAnchor: 2012-11-15 07:13:21 am

UraniumAnchor: 2012-11-15 07:05:00 am

UraniumAnchor: 2012-11-15 07:04:40 am

2012-11-15 07:04:02 am

Not a walrus

There's nothing stopping verifiers from sending their responses in a separate PM rather than using the same thread, and if you want to stop yourself from being influenced by the discussion, you can remove yourself from the group thread (or just not read anything but the first post). Makes it a little harder for Flip to keep track of things, though, so you might want to buy him a coffee if you do that a lot. Wink

As for anonymity, it's possible to reveal yourself if you want to (in your actual comment or after the fact). On the other hand, you can also have Flip completely withhold your actual comments and simply post your overall decision. The fact that most people don't do either of these suggests that they'd rather not do so in general (or don't care enough for it to matter), so I think it works out fine this way.

mike89

2012-11-15 07:09:00 am

SEGA Junkie

You guys are reminding me that I have to verify Pokemon Snap, shit

Totally Not Lee At All

2012-11-15 09:38:56 am

Hi, I just anonymously posted in a public thread. I think this can also work for public verification. I believe admins still have access to ip addresses, so any abuse of this system would be tied back to the original person.

Tiberius

2012-11-15 09:39:13 am

Make it so.

Quote from mike89:

You guys are reminding me that I have to verify Pokemon Snap, shit

I heard (publically) that it's good. I wouldn't bother watching, just accept.

Not that I think that would happen all the time with public verification, I just think there is a might higher likelyhood of risking such things.

Lee_tg

2012-11-15 10:28:35 am

Quote from Totally Not Lee At All:

Hi, I just anonymously posted in a public thread. I think this can also work for public verification. I believe admins still have access to ip addresses, so any abuse of this system would be tied back to the original person.

This must be some impostor like Lee_sda or Lee_srl. Down with the fake Lees.

Signed,
The Great Lee_tg

UraniumAnchor

2012-11-15 11:37:17 am

Not a walrus

Not speaking for the staff for this question, but I'm curious how you guys would feel about this:

After 60 days, 2 verifiers is enough to go through verification.

After 120 days, open a game up for public verification.

Tiberius

2012-11-15 11:46:57 am

Make it so.

Quote from UraniumAnchor:

Not speaking for the staff for this question, but I'm curious how you guys would feel about this:

After 60 days, 2 verifiers is enough to go through verification.

After 120 days, open a game up for public verification.

Actually I really like the idea of the necessities of verification changing with time like that. The only minor issue I have is that if there are not even 2 verifiers by 120 days then evidently not many people in the community have played the game at all and therefore public verification would only serve to judge the entertainment value of the run rather than the actual skill, which you need to have played the game to have knowledge of in most cases.

UraniumAnchor

2012-11-15 11:56:40 am

Not a walrus

That's the same issue I have with public verification at all, really, but that approach solves itself because if a game is obscure enough that 2 people don't step forward after 4 months, it's probably not going to be particularly heavily contested.

Public verification still doesn't mean 'majority rules', and hopefully if there's an obvious error somebody will catch it.

presjpolk

2012-11-15 12:32:41 pm

HELLO!

Well, verification checks four things, right?

- Time
- Cheating
- AV quality
- Run quality

SDA doesn't expect every game to be at the same level. To take the extreme example, because SMB is nearly frame-perfect, it doesn't mean every game out there has to be nearly frame-perfect. When a game gets thrown to public verification, you're not going to get verification of run quality that would pick up that level of error. But you don't need it, because the community interest and therefore skill level just aren't at that level, as the community has reached for an SMB or a Contra.

The community as a group likely won't be able to distinguish an amazing run from a merely good run, in the case of a game with no verifiers, but the community ought to be able to distinguish ordinary skill from a good run.

Is that good enough, for a game at a low interest level? I don't know the answer to that. I think that's a value judgement: Is it better for SDA to have fewer games at a higher standard, or to be more responsive to people trying to get a game kickstarted, with the risk that some less popular games will have runs that aren't quite as good?

Well, verification checks four things, right?

- Time
- Cheating
- AV quality
- Run quality

SDA doesn't expect every game to be at the same level.  To take the extreme example, because SMB is nearly frame-perfect, it doesn't mean every game out there has to be nearly frame-perfect.  When a game gets thrown to public verification, you're not going to get verification of run quality that would pick up that level of error. But you don't need it, because the community interest and therefore skill level just aren't at that level, as the community has reached for an SMB or a Contra.

The community as a group likely won't be able to distinguish an amazing run from a merely good run, in the case of a game with no verifiers, but the community ought to be able to distinguish ordinary skill from a good run.

Is that good enough, for a game at a low interest level?  I don't know the answer to that.  I think that's a value judgement: Is it better for SDA to have fewer games at a higher standard, or to be more responsive to people trying to get a game kickstarted, with the risk that some less popular games will have runs that aren't quite as good?

Kotti

2012-11-15 12:38:59 pm

Quote from UraniumAnchor:

Not speaking for the staff for this question, but I'm curious how you guys would feel about this:

After 60 days, 2 verifiers is enough to go through verification.

After 120 days, open a game up for public verification.

I like this idea, not because I'd think it would necessarily help to get things verified but because it would give the obscure games at least a tiny bit of publicity. Even if I don't think I'm knowledgeable enough to verify a run I might still have interest in just watching it. And after watching the run I might come to the conclusion that I am after all knowledgeable enough and verify it.

Paraxade

Edit history:

Paraxade: 2012-11-15 01:52:24 pm

2012-11-15 01:10:59 pm

My question there is if a run hasn't been able to find anyone for normal verification, why would it for public verification? And on that end, if a run goes to public verification, what do we do if it gets very few or no posts? How many public verifier responses are needed to render a verdict? I assume we won't accept a run because some random person guest-posts an accept.

UraniumAnchor

Edit history:

UraniumAnchor: 2012-11-15 02:23:08 pm

UraniumAnchor: 2012-11-15 02:22:45 pm

2012-11-15 02:20:45 pm

Not a walrus

Quote from Paraxade:

My question there is if a run hasn't been able to find anyone for normal verification, why would it for public verification? And on that end, if a run goes to public verification, what do we do if it gets very few or no posts? How many public verifier responses are needed to render a verdict? I assume we won't accept a run because some random person guest-posts an accept.

It would ultimately come down to whatever the current verification dictator (ie Flip) decides.

Sheer volume has never been enough to post a decision, I can think of at least one example I was personally involved in where only one verifier voted reject and that was enough to reject it, simply because the reasoning was sound.

Or if you want a more extreme example of what might potentially happen in public verification, look at what happened when Demonstrate pulled in a bunch of Youtube subscribers to try to drum up support for his Portal run. Well reasoned arguments have won out against volume in the past, so I don't think that would be an issue.

As for there simply not being enough interest in a run, there's no way to completely fix that. Public verification would only mitigate the issue, not eliminate it.

[quote="Paraxade"]My question there is if a run hasn't been able to find anyone for normal verification, why would it for public verification? And on that end, if a run goes to public verification, what do we do if it gets very few or no posts? How many public verifier responses are needed to render a verdict? I assume we won't accept a run because some random person guest-posts an accept.[/quote]

It would ultimately come down to whatever the current verification dictator (ie Flip) decides.

Sheer volume has never been enough to post a decision, I can think of at least one example I was personally involved in where only one verifier voted reject and that was enough to reject it, simply because the reasoning was sound.

Or if you want a more extreme example of what might potentially happen in public verification, look at what happened when Demonstrate pulled in a bunch of Youtube subscribers to try to drum up support for his Portal run. Well reasoned arguments have won out against volume in the past, so I don't think that would be an issue.

As for there simply not being enough interest in a run, there's no way to completely fix that. Public verification would only mitigate the issue, not eliminate it.

Paraxade

2012-11-15 03:00:19 pm

I'm talking more about low volume than high volume. I'm not really seeing what public verification will do for runs that aren't finding verifiers. That sounds like there's just no one around interested or able to verify. Going public doesn't fix that as the option to be a verifier is still open to the exact same pool of people. The theoretical situation I was describing was if that was the only public response period. (Kind of an extreme hypothetical case, yeah, but given the things I just outlined I can see it happening, if maybe to a lesser degree.)

UraniumAnchor

2012-11-15 03:15:48 pm

Not a walrus

Increased exposure. Maybe that won't help much, but it's worth a try.

Flip

2012-11-15 04:04:29 pm

1-Up!

Quote from Paraxade:

I'm talking more about low volume than high volume. I'm not really seeing what public verification will do for runs that aren't finding verifiers. That sounds like there's just no one around interested or able to verify. Going public doesn't fix that as the option to be a verifier is still open to the exact same pool of people. The theoretical situation I was describing was if that was the only public response period. (Kind of an extreme hypothetical case, yeah, but given the things I just outlined I can see it happening, if maybe to a lesser degree.)

My opinion is that if there are no "experts" to be found, then our next best option is a community groupthink. Maybe individuals don't know a game all that well, but with enough exposure and feedback, my hope is that I can come to a reasonable verdict.

Other than that anything else I would say would basically echo what UraniumAnchor has said since my last post.

Brossentia

Edit history:

Brossentia: 2012-11-15 04:53:03 pm

2012-11-15 04:52:52 pm

No pain, no gain

Quote from Flip:

Quote from Paraxade:

I'm talking more about low volume than high volume. I'm not really seeing what public verification will do for runs that aren't finding verifiers. That sounds like there's just no one around interested or able to verify. Going public doesn't fix that as the option to be a verifier is still open to the exact same pool of people. The theoretical situation I was describing was if that was the only public response period. (Kind of an extreme hypothetical case, yeah, but given the things I just outlined I can see it happening, if maybe to a lesser degree.)

My opinion is that if there are no "experts" to be found, then our next best option is a community groupthink. Maybe individuals don't know a game all that well, but with enough exposure and feedback, my hope is that I can come to a reasonable verdict.

I think this is a great point. Do we want to punish runs because they simply aren't popular? Probably not. I've looked through some of the games that are in the queue, and while I would love to watch them, I don't know enough or have interest in shelling out some mulah to try them out. In some cases, I don't have the required hardware, and few if any people do. These problems aren't the submitter's fault, and it'd be good to get rid of issues they didn't cause.

This community is pretty darn good at knowing what is reasonable in a run. More exposure can only help these runs get through the limbo of verification.

[quote="Flip"][quote="Paraxade"]I'm talking more about low volume than high volume. I'm not really seeing what public verification will do for runs that aren't finding verifiers. That sounds like there's just no one around interested or able to verify. Going public doesn't fix that as the option to be a verifier is still open to the exact same pool of people. The theoretical situation I was describing was if that was the only public response period. (Kind of an extreme hypothetical case, yeah, but given the things I just outlined I can see it happening, if maybe to a lesser degree.)[/quote]
My opinion is that if there are no "experts" to be found, then our next best option is a community groupthink. Maybe individuals don't know a game all that well, but with enough exposure and feedback, my hope is that I can come to a reasonable verdict.[/quote]

I think this is a great point.  Do we want to punish runs because they simply aren't popular?  Probably not.  I've looked through some of the games that are in the queue, and while I would love to watch them, I don't know enough or have interest in shelling out some mulah to try them out.  In some cases, I don't have the required hardware, and few if any people do.  These problems aren't the submitter's fault, and it'd be good to get rid of issues they didn't cause.

This community is pretty darn good at knowing what is reasonable in a run.  More exposure can only help these runs get through the limbo of verification.

dballin

2012-11-15 07:38:27 pm

I still feel like a dick for not verifying the Deep Fear or Vampire Panic runs because I do have the hardware to run the games and it's not like they're expensive or anything (I bought Deep Fear for $7), but then again I'm reminded that my first verification was for VG's Asteroids run that was sitting in hell for 2+ years so I feel better. Until I realize I rejected the run. teehee

I also haven't signed in for like 4 months? gogo guest posting!! Sup Lee or fake Lee whoever you REALLY are

UraniumAnchor

2012-11-15 07:46:55 pm

Not a walrus

Vampire Panic got cancelled because the person who submitted it never responded after a year and a half anyway, so don't feel too bad about that one.

dballin

2012-11-15 09:03:17 pm

I wouldn't respond after a year and a half either!