Public Verification Implementation Plan

ktwo

SDAVerification

Edit history:

SDAVerification: 2021-04-14 09:16:55 am

ShadowWraith: 2015-08-09 02:21:37 pm

ShadowWraith: 2015-08-09 02:21:35 pm

ktwo: 2015-08-09 02:01:55 pm

Flip: 2013-06-27 11:43:20 am

Flip: 2013-06-27 11:42:54 am

Flip: 2013-06-27 11:42:36 am

Flip: 2013-04-10 03:05:50 pm

Flip: 2013-04-07 03:15:43 pm

Flip: 2013-03-26 06:56:15 am

Flip: 2013-03-21 02:29:17 pm

Flip: 2013-03-20 09:42:59 pm

Flip: 2013-03-06 09:36:18 pm

Flip: 2013-03-06 09:34:47 pm

Flip: 2013-03-06 09:32:59 pm

Flip: 2013-02-25 02:08:29 pm

Flip: 2013-02-24 08:54:35 pm

Flip: 2013-02-24 12:05:01 pm

Flip: 2013-02-24 12:01:35 pm

Flip: 2013-02-24 11:07:58 am

Flip: 2013-02-24 11:07:12 am

Flip: 2013-02-24 11:06:56 am

Flip: 2013-02-24 11:06:40 am

Flip: 2013-02-24 11:04:34 am

Flip: 2013-02-24 11:03:32 am

Flip: 2013-02-24 11:01:39 am

2013-02-18 01:42:10 pm

Original post content (from Flip):

Public verification threads will live in this sub-board, and once the deadline is met, I will move the thread into the normal verification board and post my verdict there. That way it is always easy to find the active public verification threads.

Refer to the Verification Guidelines to read the public verification procedures. Remember that these games didn't get verifiers, so if you haven't played it, you're in the majority. Collaborative effort is vital for the success of this system.

I think it's time to update this thread and use it going forward for posting relevant information on how public verification works.

From the site, it's me, ktwo, who currently take care of everything surrounding sda-verification (both private and public). This includes posting the decision (accept/reject) for all verification topics.

I'm trying to follow what's going on in the different verification topics, so if you have any questions, feel free to post them there. If I don't respond to specific questions directed my way, you are also welcome to send it in a pm. This particular thread will also be open in case someone has more general comments regarding public verification they wish to discuss.

As things currently stand, runs will be posted for verification in the same pace as they become available to us. I'm trying to roughly stick to the current schedule:
- Verification deadline is set to Tuesday two weeks out. That means some verifications will be open for a little more than 14 days, some less.
- When the verification deadline is up, I will post the decision (accept/reject). If the verification responses are very split, not conclusive enough or if there is an ongoing discussion, the verification will be left open.

Thread title:

Original post content (from Flip):
[hidden]Public verification threads will live in this sub-board, and once the deadline is met, I will move the thread into the normal verification board and post my verdict there. That way it is always easy to find the active public verification threads.

Refer to the [url=https://forum.speeddemosarchive.com/post/verification_guidelines.html]Verification Guidelines[/url] to read the public verification procedures. Remember that these games didn't get verifiers, so if you haven't played it, you're in the majority. Collaborative effort is vital for the success of this system.[/hidden]

I think it's time to update this thread and use it going forward for posting relevant information on how public verification works.

From the site, it's me, ktwo, who currently take care of everything surrounding sda-verification (both private and public). This includes posting the decision (accept/reject) for all verification topics.

I'm trying to follow what's going on in the different verification topics, so if you have any questions, feel free to post them there. If I don't respond to specific questions directed my way, you are also welcome to send it in a pm. This particular thread will also be open in case someone has more general comments regarding public verification they wish to discuss.

As things currently stand, runs will be posted for verification in the same pace as they become available to us. I'm trying to roughly stick to the current schedule:
- Verification deadline is set to Tuesday two weeks out. That means some verifications will be open for a little more than 14 days, some less.
- When the verification deadline is up, I will post the decision (accept/reject). If the verification responses are very split, not conclusive enough or if there is an ongoing discussion, the verification will be left open.

ktwo

2015-08-24 11:48:38 am

Just some quick info on what's going on in this section.

The last two updates have been pretty small. There is a step on the backend between receiving the files until they're ready to go out for verification. It has built up a backlog in this step in the wake of sgdq. Tomorrow, there will be a pretty big influx of runs in public verification though as the backlog in the mentioned step has more or less been cleared out since last week.

ktwo

2015-09-22 01:24:17 pm

Some more news from my side.

I will from now on take a quick look at the videos before sending them out. That should hopefully help avoid some of the most obvious cases of insufficient a/v quality. It will just be quick peaks though, so don't expect me to catch everything.

You will also see more cases of runs moving from private to public verification. It will be done on a case-by-case basis. More time will of course be granted for longer runs, but there are also some shorter runs for which the verification should just be a formality. Instead of chasing replies in private verification, I will just move them to public.

ktwo

2015-12-01 02:19:53 pm

The verification guidelines have been updated. https://kb.speeddemosarchive.com/Verification_Guidelines
All of the sections have been re-written, but I don't think there should be any shocking news. There is now also included a section explaining the difference between public and private verification, which had been missing until now.

ktwo

Edit history:

ktwo: 2016-05-24 09:38:35 am

2016-05-22 05:52:22 am

From https://forum.speeddemosarchive.com/post/assassins_creed_brotherhood_available_for_verification__due_2016.05.17_16.html#assassins_creed_brotherhood_available_for_verification__due_2016.05.17_16:

Quote from LotBlind:

Ktwo: Do you think this should be included as an instruction in these public verification threads? If you don't mean to finish your verification in one go, leave a note. Right now it would have been useful. I'm still going to watch the rest, but if you don't want to HonorableJay, you can definitely leave it now.

LotBlind, I'm not sold on that idea. A few reasons come to mind.

1. There are not many private verifications these days, but they are a testimony of how hard it is to get timely replies from people signing up for something (generally speaking). I know you don't function that way and if you write you'll do something, I have full confidence that you will. And that's assuming that people will actually read those instructions and post that they are planning to verify. In this thread, both you and honorableJay wrote that you intended to verify the run, which should have been an indication as good as any, but still we got two replies out of the blue.

2. It's hard to put a clearly defined limit of when a run is done being verified. If a couple of people with limited experience from SDA would write that they were planning to verify, then you would need to assess when the "verification quota" is filled. Let's take this thread as an example. Two competent people from the game community turned up to verify and posted very sensible replies. But none of them have much of an SDA-history (judging by the number of posts). So it could just as well have been two random people who ended up replying "looks ok, accept".

3. There has always been an element to verification that is more than just the accept/reject. It's also about giving feedback to the runner. I know a lot of runners are interested in hearing what other people have to say about their run. So the more replies they get, the better experience they get from submitting. Trying to bring the replies down to the bare minimum goes a bit against this spirit. That's also why I think that people should not stop replying to threads they're interested in, even though there are enough replies already and the decision is obvious.

From https://forum.speeddemosarchive.com/post/assassins_creed_brotherhood_available_for_verification__due_2016.05.17_16.html#assassins_creed_brotherhood_available_for_verification__due_2016.05.17_16:
[quote="LotBlind"]Ktwo: Do you think this should be included as an instruction in these public verification threads? If you don't mean to finish your verification in one go, leave a note. Right now it would have been useful. I'm still going to watch the rest, but if you don't want to HonorableJay, you can definitely leave it now.[/quote]

LotBlind, I'm not sold on that idea. A few reasons come to mind.

1. There are not many private verifications these days, but they are a testimony of how hard it is to get timely replies from people signing up for something (generally speaking). I know you don't function that way and if you write you'll do something, I have full confidence that you will. And that's assuming that people will actually read those instructions and post that they are planning to verify. In this thread, both you and honorableJay wrote that you intended to verify the run, which should have been an indication as good as any, but still we got two replies out of the blue.

2. It's hard to put a clearly defined limit of when a run is done being verified. If a couple of people with limited experience from SDA would write that they were planning to verify, then you would need to assess when the "verification quota" is filled. Let's take this thread as an example. Two competent people from the game community turned up to verify and posted very sensible replies. But none of them have much of an SDA-history (judging by the number of posts). So it could just as well have been two random people who ended up replying "looks ok, accept".

3. There has always been an element to verification that is more than just the accept/reject. It's also about giving feedback to the runner. I know a lot of runners are interested in hearing what other people have to say about their run. So the more replies they get, the better experience they get from submitting. Trying to bring the replies down to the bare minimum goes a bit against this spirit. That's also why I think that people should not stop replying to threads they're interested in, even though there are enough replies already and the decision is obvious.

LotBlind

2016-05-22 12:55:49 pm

Ktwo: I see what you're saying, but you can look at it the other way round as well: the less verifiers one game gets, the more some other game may get, at least in theory. Someone getting their easy accept for a game with tons of history, and everyone knows who they are - I'd like to argue - doesn't need the feedback in nearly the same way as someone waiting on the word on their first submission. I'm sure those people are even more delighted that their run got some attention by the deadline in the first place. I think that's probably the kind of people you're talking about too. I think I'm still operating on the principle of "if I don't verify it, it's probably going to be stuck for another 2 weeks", and any time I save doing a verification will be put to something equally useful, but I'm not the average I know.

I get the part about defining when it's enough - the force of habit and also the pragmatism arising from a low number of verifiers has got "2 is enough" stuck in my head, especially since I never tend to do verifications that are getting more than just enough. Are you saying (do I get you right) that the two runners who showed up here might have been lenient on the run because of unfamiliarity with SDA? I can't see what they've written equating to "random people", they've addressed everything exactly as they should, and they can be expected to know exactly what the standards are like for this game. Maybe I'm misunderstanding you.

Another thing about feedback: I think it's really the same as your point #2 - 2 really good verifications with some extra feedback etc. comments fill the "feedback" quota more so than 5 short, undetailed replies, so... whatever that means.

" people should not stop replying to threads they're interested in, even though there are enough replies already and the decision is obvious." yes, but again the pragmatic thing - if they were never going to be doing any other run then I suppose it doesn't matter, but with verifiers of my sort that's different.

I'll be done in another 3 days I predict. I'll watch the rest partially because I know I'll probably end up writing about it (and because it's kinda entertaining I'll admit).

Will you consider adding the "let us know you're getting started on the verification on long runs" in the guidelines anyway? If people don't read them, that doesn't hurt it either, but there's always someone who wants to do things as they should who wouldn't have thought of that. That's probably all I'm going to say on that in either case.

Ktwo: I see what you're saying, but you can look at it the other way round as well: the less verifiers one game gets, the more some other game may get, at least in theory. Someone getting their easy accept for a game with tons of history, and everyone knows who they are - I'd like to argue - doesn't need the feedback in nearly the same way as someone waiting on the word on their first submission. I'm sure those people are even more delighted that their run got some attention by the deadline in the first place. I think that's probably the kind of people you're talking about too. I think I'm still operating on the principle of "if I don't verify it, it's probably going to be stuck for another 2 weeks", and any time I save doing a verification will be put to something equally useful, but I'm not the average I know.

I get the part about defining when it's enough - the force of habit and also the pragmatism arising from a low number of verifiers has got "2 is enough" stuck in my head, especially since I never tend to do verifications that are getting more than just enough. Are you saying (do I get you right) that the two runners who showed up here might have been lenient on the run because of unfamiliarity with SDA? I can't see what they've written equating to "random people", they've addressed everything exactly as they should, and they can be expected to know exactly what the standards are like for this game. Maybe I'm misunderstanding you.

Another thing about feedback: I think it's really the same as your point #2 - 2 really good verifications with some extra feedback etc. comments fill the "feedback" quota more so than 5 short, undetailed replies, so... whatever that means.

" people should not stop replying to threads they're interested in, even though there are enough replies already and the decision is obvious." yes, but again the pragmatic thing - if they were never going to be doing any other run then I suppose it doesn't matter, but with verifiers of my sort that's different.

I'll be done in another 3 days I predict. I'll watch the rest partially because I know I'll probably end up writing about it (and because it's kinda entertaining I'll admit).

Will you consider adding the "let us know you're getting started on the verification on long runs" in the guidelines anyway? If people don't read them, that doesn't hurt it either, but there's always someone who wants to do things as they should who wouldn't have thought of that. That's probably all I'm going to say on that in either case.

ktwo

2016-05-24 10:30:05 am

Quote from LotBlind:

Are you saying (do I get you right) that the two runners who showed up here might have been lenient on the run because of unfamiliarity with SDA? I can't see what they've written equating to "random people", they've addressed everything exactly as they should, and they can be expected to know exactly what the standards are like for this game. Maybe I'm misunderstanding you.

It sounds like you misunderstood me:

Quote from ktwo:

Let's take this thread as an example. Two competent people from the game community turned up to verify and posted very sensible replies.

However, if they had posted beforehand that they planned to verify, it would have been difficult based on their SDA-history to evaluate how their replies would look like (and you can't really ask people to write their speedrun cv for this kind of thing either). It would also have been difficult to know the chances of actually getting a reply in a timely manner or at all (point #1).

Quote from LotBlind:

Will you consider adding the "let us know you're getting started on the verification on long runs" in the guidelines anyway? If people don't read them, that doesn't hurt it either, but there's always someone who wants to do things as they should who wouldn't have thought of that. That's probably all I'm going to say on that in either case.

I appreciate the feedback and I don't mind hearing you out about this or other ideas. I think I see where you're coming from here. However, I'm just not convinced that it will have the wished for result, all things considered. On the other hand, it doesn't mean I disapprove of your habit of voluntarily posting about (particularly long) runs you plan to verify.

[quote="LotBlind"]Are you saying (do I get you right) that the two runners who showed up here might have been lenient on the run because of unfamiliarity with SDA? I can't see what they've written equating to "random people", they've addressed everything exactly as they should, and they can be expected to know exactly what the standards are like for this game. Maybe I'm misunderstanding you.[/quote]
It sounds like you misunderstood me:
[quote="ktwo"]Let's take this thread as an example. Two competent people from the game community turned up to verify and posted very sensible replies.[/quote]
However, if they had posted beforehand that they planned to verify, it would have been difficult based on their SDA-history to evaluate how their replies would look like (and you can't really ask people to write their speedrun cv for this kind of thing either). It would also have been difficult to know the chances of actually getting a reply in a timely manner or at all (point #1).

[quote="LotBlind"]Will you consider adding the "let us know you're getting started on the verification on long runs" in the guidelines anyway? If people don't read them, that doesn't hurt it either, but there's always someone who wants to do things as they should who wouldn't have thought of that. That's probably all I'm going to say on that in either case.[/quote]
I appreciate the feedback and I don't mind hearing you out about this or other ideas. I think I see where you're coming from here. However, I'm just not convinced that it will have the wished for result, all things considered. On the other hand, it doesn't mean I disapprove of your habit of voluntarily posting about (particularly long) runs you plan to verify.

LotBlind

Edit history:

LotBlind: 2016-05-25 02:08:39 pm

2016-05-25 02:07:06 pm

Okay, I see... I guess I instinctively always go "hey guys I know this game well" if that's true of me but without exposure to SDA might not be the case.

What if you phrased it like this "If you know you're definitely going to verify a run but you're not doing it straight away, let the thread know you're on the job so everyone can see which runs aren't getting enough verifiers. If you want, you can mention any experience with the game in your comments." This would discourage anyone from saying anything unless it's more or less guaranteed, and the last thing lets you assume a very concise "looks good" is by a random.

Meanwhile if you see no follow-up within a certain length of time, you could go as far as to delete or edit their initial posts so it's clear they're not being counted on anymore.

ktwo

2016-05-28 03:31:19 am

How about we do like this. Today, if a run gets overdue, it gets stickied, but usually without any comments. If it's a long run, I could then also make a post asking for people to post if they're planning to verify. That way we target the ones that really would be the ones in most need.

Judgy

Edit history:

Judgy: 2016-05-28 07:46:27 am

2016-05-28 07:43:19 am

Borderlands 2 Glitch Hunter/ router.

Quote from ktwo:

How about we do like this. Today, if a run gets overdue, it gets stickied, but usually without any comments. If it's a long run, I could then also make a post asking for people to post if they're planning to verify. That way we target the ones that really would be the ones in most need.

This sounds good to me, usually if a run is a long one (or has a huge amount of segments ) I'll do it in maybe 2 or 3 batches of watching at which point I will post in the topic saying that I am looking at it particularly if that run has had no other verification before me and has been sitting a while.

I'm not sure if it is a good Idea or not but maybe twitch could be utilized to help with verification if we have a page like www.twitch.tv/SDAVerification where runs can be streamed to whomever wants to watch them (maybe set up a schedule week-by-week). at the end of the stream we then ask them to come here to post their decisions for Gameplay and Cheating ONLY!! because the quality tests get done later in PRC anyway the limitations of not having the raw encodes quality on the stream doesn't matter, this way you may be able to get a few more verifiers who would have otherwise not have watched at all.

just my 2 cents Tongue

[quote="ktwo"]How about we do like this. Today, if a run gets overdue, it gets stickied, but usually without any comments. If it's a long run, I could then also make a post asking for people to post if they're planning to verify. That way we target the ones that really would be the ones in most need.[/quote]

This sounds good to me, usually if a run is a long one (or has a huge amount of segments ) I'll do it in maybe 2 or 3 batches of watching at which point I will post in the topic saying that I am looking at it particularly if that run has had no other verification before me and has been sitting a while.

I'm not sure if it is a good Idea or not but maybe twitch could be utilized to help with verification if we have a page like www.twitch.tv/SDAVerification where runs can be streamed to whomever wants to watch them (maybe set up a schedule week-by-week). at the end of the stream we then ask them to come here to post their decisions for Gameplay and Cheating ONLY!! because the quality tests get done later in PRC anyway the limitations of not having the raw encodes quality on the stream doesn't matter, this way you may be able to get a few more verifiers who would have otherwise not have watched at all.

just my 2 cents :P

LotBlind

2016-05-28 01:15:47 pm

Do whatever you feel is best, I just wanted to bring out all the pov I had.

Stream verifications! Where did you come up with that? Cheesy

I fear having no comments (well you could link to them) and no ability to stop and replay parts might make it difficult to do a proper verification; also what kind of crowd would that attract? Tongue

Tigger77

2016-05-29 02:40:15 am

I survived MIKE-Fest 1

This streaming idea sounds really interesting Cheesy

The no stop and replay thingis an issue for sure but we could direkt those people to the forum where they can download the run ... if they have enough interesst to replay certain parts they might have enough intresst to download the run.

As for the crowed ... I have no idea who would want do to that but I also thought that watching a Speedrunner resets over a million times live is boring as hell Wink

As a thought: If we had a bot that would watch the twitch comment chat and then post it here as a verification response?
Of curse it would have to have some key words in comments it actuality only records comments that start with something special otherwise we would have a mess.

LotBlind

2016-05-29 11:59:56 am

Just as a way to "advertise" public verification and incoming runs - sure.

TheMG2

Edit history:

TheMG2: 2016-05-29 12:40:35 pm

2016-05-29 12:31:18 pm

Could always make specific bot commands to record comments. These typically use the format !command so !verifycomment CommentGoesHere could work.

I'm not sure how I feel about this in general but I'm certainly intrigued.

EDIT: I guess the big problem I see with this is the potential for trolls to exploit it. You can't IP ban on twitch, nor can you even confirm if two users have the same IP.

ymh

2016-05-29 01:30:44 pm

As advertisement - maybe. But I don't think as actual verification. A stream is anything but practical for that, for reasons already named. And in my opinion the current system (a thread notifying about which runs require verification) is already pretty comfortable and sufficient.
Beside the risk of abuse or spam, I also think that signing up in the forums is the least thing someone can do to participate as a (public) verifier.

Maybe getting in touch with other (gaming/speedrunning) communities might be a way to get the word out there ? I don't know how dramatic the situation is, at least the number of public verifiers seems to be countable on the fingers of one, two hands.
And yet the runs get through, don't they.

ktwo

2016-05-30 11:19:30 am

I think the streaming idea is pretty neat. It might not be exactly aligned with the original idea and the values behind how SDA-verification was intented to work, but it could be worth a shot. I do see a few practical issues, but I'm guessing they can be overcome if someone would be interested in hosting it. It's not going to be me, but I can of course support with some arrangements, if necessary, here on the site. If anyone reading this is interested, you can post here or pm me.

The idea with twitch chat verification replies can probably be dropped. People watch there and then reply here.

If this ever sees the light of the day, I have a feeling it will still mostly be the same people already doing most of the verifications. But I figure there is nothing wrong with doing it together instead of each one on their own.

ais523

2016-06-13 02:51:30 am

Being able to rewind and go "what did I just watch?" is a vital tool for verification. Otherwise, it can be very hard to distinguish a genius strategy from an acceptable backup strategy from a sloppy mistake from outright cheating. (I remember verifying a run when the runner just randomly died for no obvious reason. I eventually figured it out after concentrating on the replay, and decided that the death was acceptable because it was near the end of a single-segment run and I missed it too, despite being fairly good at the game. In general, I don't public-verify runs unless I'd be willing to private-verify them.)

LotBlind

2016-06-14 09:17:23 am

ais523: May I ask if your aversion to doing random public verifications is because you're afraid you'll make some mistake?

ais523

2016-06-14 11:17:55 am

It's partly because it takes a lot of time to verify (and I don't have the network bandwidth to verify at the moment either).

If I don't know the game, it would take even longer because I have to learn about the game first, ask questions to the runner, etc..

If the game's obscure enough not to have plenty of speedruns already, it's quite likely that (for example) the runner will make a routing mistake, start an autoscroller late, or the like. All these things are things that verifiers should be able to catch, but if you aren't learning about the game you aren't able to find them.

LotBlind

Edit history:

LotBlind: 2016-06-15 01:15:41 pm

2016-06-15 01:14:57 pm

Okay. Well, my philosophy is you can't expect public verification to be as good as private, and you can't ask verifiers to really go that far out of their way, we just wouldn't be getting the runs processed. Most verifications are really clear-cut too. In some cases, yes, I've asked them a whole load of questions and waited for them to explain it all, but then that's usually been enough.

Also I think it's more the case that if the runner is new, there's a higher chance of mistakes like that, not if a new game is being done by an old runner.

Sonikkustar

2016-06-15 08:53:24 pm

The Great Farming Empire

I kinda figured that if a person needs more time to offer a proper verification response, then they can just come on the forums and replay the video for themselves. But the general idea of streaming runs to verify does sound kinda neat. This does seem to beg the question of whether or not these streams should be just for verification as they can also provide an opportunity to advertise runs of games that are not typically brought up like what ymh said.

If a verification stream were to be set up, I'd figure there should be at least 2-3 people perhaps commentating about the run, with them offering their thoughts afterwards. However, I feel as though this might cause some people to think that SDA as a community may be a bit snobbish when it comes to the runs wanted as it may differ from how people view speedrunning now as opposed to how it's viewed here. I dunno, maybe I'm just being paranoid.

Anyways, it seems like an interesting idea and I'd definitely be down for it.

LotBlind

2016-06-16 12:49:30 pm

If it's meant for actually verifying the run, you can't have extra stuff going on. Or do you mean those people are verifiers voicing their thoughts in real-time? It's really hard to come by verifiers as it is, and getting several people together at the same time. Tongue

I think I can see a less obscure use for such a stream now -- you used the word "advertize"... well how about simply playing the runs back after they've been published? Seems Twitch could act naturally as a sort of rec room for those who like watching them together with others. If only it weren't for that damn buffering... You'd probably want to set the stream to be automatic and loop the most recently published runs the clock round.

ymh

2016-06-22 03:00:53 am

@Lotblind
This idea might be unnecessary, but maybe we (all who are interested) should add a few concrete workflow tips for (public) verification along with the general guidelines, to encourage new verifiers and also share some methods with each other.
It might be too obvious or uncalled-for, but just to give an example, after scanning SDA I like to check speedrun.com for any existing runs of the particular game, categories, or any route/trick explanations on the game page there.
Stuff like that.

ktwo

2016-06-23 02:16:22 pm

ymh, there is no harm in sharing your approach. I'm guessing that if a potential new verifier turns up, the first thing they would read (if anything) would be the verification guidelines (https://kb.speeddemosarchive.com/Verification_Guidelines), since that's linked to in the first post of every verification thread. The guidelines do talk a bit about what's expected from a verifier and also touch upon the difference between private and public verificaiton. However, if you have any ideas on how to improve the text (it could for example be a checklist of things to do when verifying), we can easily make some changes to it.