Move towards Public Verification permanently for verifying runs?

2020-06-16 10:10:39 pm

The Great Farming Empire

Hey. So I know I'm not really an authority here, but considering that a majority of runs that are being submitted now go through Public Verification instead of Private Verification, would it be good to start phasing out Private Verification and move towards one verification system? I know the idea could be considered blasphemous since Private Verification has been a staple of SDA since its inception, however the only run I can think of this year that has actually gone through Private Verification was this run of Contra 3, with every other run going through Public Verification with little issue.

Again feel free to disagree all you want. All I'm saying is that maybe now might be a good time if any to start discussing the whole verification system and whether or not things could change in the future.

Thread title:

LotBlind

2020-06-18 12:30:26 pm

You don't have to be an authority to post a suggestion. Cheesy

I don't immediately know why you think that would make anything easier though. Also the situation could change again.

ktwo

2020-06-18 03:30:47 pm

Sonikkustar, I share your feeling.

In practice, what you're suggesting is also what's happening. As a general rule, whenever you see runs in the "runs needing verifiers" thread, it's because the runners haven't uploaded the video or because we're preparing the verification encode.

The above means that other than the rare exception, where enough verifiers are found before we have a video for verification, runs go into public verification.

Would the Contra 3 verification have looked the same in public verification? In my experience, the response rate is usually better in private verifications, where people have voluntarily signed up beforehand and "committed" to check out the run. I've seen enough times signed-up verifiers not posting any reply if the run ends up going to public verification. Maybe potential verifiers look at a thread in public verification and think that it's already done and won't bother posting? From experience, I can definitely say that many runners like to see that other people have watched their run and provided feedback. So more replies usually mean a better experience for the runner. Not to mention that more eyes on a run is also better for obvious reasons.
Time-wise, it wouldn't have changed anything. The verification time is the same in public and private verifications (that hasn't always been the case, but it is now).

I'd like to address one more thing. If you have looked at the submission form, you will have seen that the submitter has three options to select from on how the run should be verified. In my opinion, that is just a cause of confusion for most people not already quite familiar with SDA. The options are there because it was judged to be pertinent when the back-end was created. But times change... I can honestly say that whatever you select there today, it's going to be corrected to "immediate public verification". As explained above, if some specific criteria are fulfilled, it might still end up going to private verification though.

Sonikkustar, I share your feeling.

In practice, what you're suggesting is also what's happening. As a general rule, whenever you see runs in the "runs needing verifiers" thread, it's because the runners haven't uploaded the video or because we're preparing the verification encode.

The above means that other than the rare exception, where enough verifiers are found before we have a video for verification, runs go into public verification.

Would the Contra 3 verification have looked the same in public verification? In my experience, the response rate is usually better in private verifications, where people have voluntarily signed up beforehand and "committed" to check out the run. I've seen enough times signed-up verifiers not posting any reply if the run ends up going to public verification. Maybe potential verifiers look at a thread in public verification and think that it's already done and won't bother posting? From experience, I can definitely say that many runners like to see that other people have watched their run and provided feedback. So more replies usually mean a better experience for the runner. Not to mention that more eyes on a run is also better for obvious reasons.
Time-wise, it wouldn't have changed anything. The verification time is the same in public and private verifications (that hasn't always been the case, but it is now).

I'd like to address one more thing. If you have looked at the submission form, you will have seen that the submitter has three options to select from on how the run should be verified. In my opinion, that is just a cause of confusion for most people not already quite familiar with SDA. The options are there because it was judged to be pertinent when the back-end was created. But times change... I can honestly say that whatever you select there today, it's going to be corrected to "immediate public verification". As explained above, if some specific criteria are fulfilled, it might still end up going to private verification though.

LotBlind

2020-06-19 08:35:07 am

I have no real opinions, you've got the better view ktwo.

Sonikkustar

2020-06-21 12:03:42 am

The Great Farming Empire

Quote from ktwo:

Sonikkustar, I share your feeling.

In practice, what you're suggesting is also what's happening. As a general rule, whenever you see runs in the "runs needing verifiers" thread, it's because the runners haven't uploaded the video or because we're preparing the verification encode.

The above means that other than the rare exception, where enough verifiers are found before we have a video for verification, runs go into public verification.

Would the Contra 3 verification have looked the same in public verification? In my experience, the response rate is usually better in private verifications, where people have voluntarily signed up beforehand and "committed" to check out the run. I've seen enough times signed-up verifiers not posting any reply if the run ends up going to public verification. Maybe potential verifiers look at a thread in public verification and think that it's already done and won't bother posting? From experience, I can definitely say that many runners like to see that other people have watched their run and provided feedback. So more replies usually mean a better experience for the runner. Not to mention that more eyes on a run is also better for obvious reasons.
Time-wise, it wouldn't have changed anything. The verification time is the same in public and private verifications (that hasn't always been the case, but it is now).

I'd like to address one more thing. If you have looked at the submission form, you will have seen that the submitter has three options to select from on how the run should be verified. In my opinion, that is just a cause of confusion for most people not already quite familiar with SDA. The options are there because it was judged to be pertinent when the back-end was created. But times change... I can honestly say that whatever you select there today, it's going to be corrected to "immediate public verification". As explained above, if some specific criteria are fulfilled, it might still end up going to private verification though.

I'm not entirely sure if the response rate being better with private verification is a unique thing with SDA though. The same thing could probably be said with how moderators verify runs on SRC since becoming a moderator is also a commitment people tend to make voluntarily. I definitely agree on how runners see feedback on their runs being a good thing. I think SDA is in a nice position where many different people who don't run the same game can look at a specific run and give feedback from different perspectives. It's that kind of cross-over that I feel makes the way the SDA community verifies run actually pretty great. Whether or not it's enough justification to have private verification still around however is still something I'd want to try and discuss however since there are some issues that I'd like to bring up. Some of them are more structural and some of them are personal grievances that I've come across over the years.

The confusion is partly the reason why I wanted to start this discussion and open this thread in the first place. As it stands, the way verification is handled is honestly rather cumbersome because with there being multiple options for verification, it becomes hard to asses where our attention should be diverted to. Speaking from the verification side of things, I've had multiple instances where I've signed up to verify for certain runs only for them to end up in public verification with no notification or anything to know beforehand. So more often than not, I don't really bother with public verification in the long run because it ends up becoming more of a hassle over anything else trying to keep up with everything.

It kinda speaks to how there are just way too many cogs in the machine so to speak that makes things more convoluted than they need to be, which is why I think that only having public verification would help make things just a little bit easier. Having only one type of verification could certainly help make things more focused and easier to keep tack of rather than having to select from three types whenever you submit a run. Also, because of all the focus being diverted to one place, you'd probably have a greater chance of people coming in to give responses because they wouldn't have to worry about signing up to verify beforehand. Granted there is still the valid point about having to wait for proper video/verification files, but you could probably wait until those files are ready before sending it to public verification instead of having it sit in the "runs needing verifiers" thread, almost effectively "cutting out the middleman" of sorts with that thread no longer serving much of a purpose. But maybe I'm getting too much ahead of myself.

Obviously doing this isn't going to suddenly improve things as there are probably a lot of other things that need to change in order to really make a difference. However considering how people felt about bringing massive change to SDA, I figured it would be better to try and take things in baby steps. As for other ways of improving things, I'll leave that to those willing to share ideas of their own.

[quote="ktwo"]Sonikkustar, I share your feeling.

In practice, what you're suggesting is also what's happening. As a general rule, whenever you see runs in the "runs needing verifiers" thread, it's because the runners haven't uploaded the video or because we're preparing the verification encode.

The above means that other than the rare exception, where enough verifiers are found before we have a video for verification, runs go into public verification.

Would the Contra 3 verification have looked the same in public verification? In my experience, the response rate is usually better in private verifications, where people have voluntarily signed up beforehand and "committed" to check out the run. I've seen enough times signed-up verifiers not posting any reply if the run ends up going to public verification. Maybe potential verifiers look at a thread in public verification and think that it's already done and won't bother posting? From experience, I can definitely say that many runners like to see that other people have watched their run and provided feedback. So more replies usually mean a better experience for the runner. Not to mention that more eyes on a run is also better for obvious reasons.
Time-wise, it wouldn't have changed anything. The verification time is the same in public and private verifications (that hasn't always been the case, but it is now).

I'd like to address one more thing. If you have looked at the submission form, you will have seen that the submitter has three options to select from on how the run should be verified. In my opinion, that is just a cause of confusion for most people not already quite familiar with SDA. The options are there because it was judged to be pertinent when the back-end was created. But times change... I can honestly say that whatever you select there today, it's going to be corrected to "immediate public verification". As explained above, if some specific criteria are fulfilled, it might still end up going to private verification though.[/quote]
I'm not entirely sure if the response rate being better with private verification is a unique thing with SDA though. The same thing could probably be said with how moderators verify runs on SRC since becoming a moderator is also a commitment people tend to make voluntarily. I definitely agree on how runners see feedback on their runs being a good thing. I think SDA is in a nice position where many different people who don't run the same game can look at a specific run and give feedback from different perspectives. It's that kind of cross-over that I feel makes the way the SDA community verifies run actually pretty great. Whether or not it's enough justification to have private verification still around however is still something I'd want to try and discuss however since there are some issues that I'd like to bring up. Some of them are more structural and some of them are personal grievances that I've come across over the years.

The confusion is partly the reason why I wanted to start this discussion and open this thread in the first place. As it stands, the way verification is handled is honestly rather cumbersome because with there being multiple options for verification, it becomes hard to asses where our attention should be diverted to. Speaking from the verification side of things, I've had multiple instances where I've signed up to verify for certain runs only for them to end up in public verification with no notification or anything to know beforehand. So more often than not, I don't really bother with public verification in the long run because it ends up becoming more of a hassle over anything else trying to keep up with everything.

It kinda speaks to how there are just way too many cogs in the machine so to speak that makes things more convoluted than they need to be, which is why I think that only having public verification would help make things just a little bit easier. Having only one type of verification could certainly help make things more focused and easier to keep tack of rather than having to select from three types whenever you submit a run. Also, because of all the focus being diverted to one place, you'd probably have a greater chance of people coming in to give responses because they wouldn't have to worry about signing up to verify beforehand. Granted there is still the valid point about having to wait for proper video/verification files, but you could probably wait until those files are ready before sending it to public verification instead of having it sit in the "runs needing verifiers" thread, almost effectively "cutting out the middleman" of sorts with that thread no longer serving much of a purpose. But maybe I'm getting too much ahead of myself.

Obviously doing this isn't going to suddenly improve things as there are probably a lot of other things that need to change in order to really make a difference. However considering how people felt about bringing massive change to SDA, I figured it would be better to try and take things in baby steps. As for other ways of improving things, I'll leave that to those willing to share ideas of their own.

LotBlind

2020-06-22 01:36:58 pm

This is ktwo's field more than anyone's but just a few idle thoughts:

I think the idea with private verification ever having been a thing was that way the verifiers get to talk about the run "in peace", sort of like a jury. I can imagine sensitive topics coming up, like if someone was suspected of cheating. Completely removing the possibility of this kind of thing would make it more awkward in a situation like that. I'm not exactly sure why anonymity has been an option but that part could be arranged through PM'ing the SDAverification user instead.

If you only want to verify select games in public verification, I guess subscribing to the thread where the list is wouldn't be enough. The wider your range, the more useful it is, of course.

Could start posting what runs are in verification currently in each front page post, even though the front page cycle will only seldom perfectly coincide with verification.

ktwo

2020-06-23 03:21:19 pm

I've made a request to remove the option of choosing the verification method in the submission form. I don't have a firm answer on how much work it would take to change it and if it's worth it. Seen from the outside, it doesn't seem like a monumental change, so hopefully it can be pushed through.

Whenever people sign up as verifiers and the run ends up in public verification, I send out a PM to alert them. This is done manually, so there could of course be cases where I've missed out. I would be surprised if it were that frequent though, so you must have been pretty unlucky to never have received any of those notifications. As I mentioned in my previous post, the response rate tends to be very low from people having signed up as verifiers to then post a reply in public verification despite receiving notifications. I can only speculate why that is, but it at least seems to me that some people are more willing to do private verifications than public ones.

I've also asked how much it would take to automate sending out a PM to verifiers. Getting that implemented would be pretty marginal though. There is very little manual work involved in this with the current site activity, so I doubt it would get noticed by the site users.

Personally, I think private verification generally works pretty well for the submissions that end up there. Not well enough to delay a run to go into public verification for a significant amount of time though. So the way I see it, I don't see much harm in keeping that track open for the few submissions it's applicable for. However, I must admit that I'm not sure I'm able to completely follow your reasoning to take in your perspective. Because private verification exists, you say that public verifications are a hassle, but if private verifications were to go, then you'd be all over public verifications? If that's a correct summary of it, I'm not convinced that's a general feeling among the potential verifiers. Your help in verification is definitely appreciated, so I hope you'll find the motivation again to check out the public verification section of the forum from time to time, despite private verification still being a thing.

I've made a request to remove the option of choosing the verification method in the submission form. I don't have a firm answer on how much work it would take to change it and if it's worth it. Seen from the outside, it doesn't seem like a monumental change, so hopefully it can be pushed through.

Whenever people sign up as verifiers and the run ends up in public verification, I send out a PM to alert them. This is done manually, so there could of course be cases where I've missed out. I would be surprised if it were that frequent though, so you must have been pretty unlucky to never have received any of those notifications. As I mentioned in my previous post, the response rate tends to be very low from people having signed up as verifiers to then post a reply in public verification despite receiving notifications. I can only speculate why that is, but it at least seems to me that some people are more willing to do private verifications than public ones.

I've also asked how much it would take to automate sending out a PM to verifiers. Getting that implemented would be pretty marginal though. There is very little manual work involved in this with the current site activity, so I doubt it would get noticed by the site users.

Personally, I think private verification generally works pretty well for the submissions that end up there. Not well enough to delay a run to go into public verification for a significant amount of time though. So the way I see it, I don't see much harm in keeping that track open for the few submissions it's applicable for. However, I must admit that I'm not sure I'm able to completely follow your reasoning to take in your perspective. Because private verification exists, you say that public verifications are a hassle, but if private verifications were to go, then you'd be all over public verifications? If that's a correct summary of it, I'm not convinced that's a general feeling among the potential verifiers. Your help in verification is definitely appreciated, so I hope you'll find the motivation again to check out the public verification section of the forum from time to time, despite private verification still being a thing.