Marathon Donation App - Call for testers

2012-06-10 02:41:27 pm

SGL Scrublord

One oddball thing I noticed was a ping pong effect, where I would apply a bid and it'd somehow reappear for being read off on the other laptop and vice versa. It would reappear with the "Donation Read" checked off or a bid attached to the bottom already, but would come back regardless. Beyond that I am totally clueless amid all the technobabble.

UraniumAnchor

Edit history:

UraniumAnchor: 2012-06-10 04:42:50 pm

UraniumAnchor: 2012-06-10 04:40:31 pm

2012-06-10 04:35:58 pm

Not a walrus

I know exactly why that's happening, and it hopefully will be far less of an issue once we get the interface redesign done.

We're completely redesigning how the app interacts with the database, anyway. It won't touch the DB directly any more, it'll just be using requests to the website and rely on the webserver to do all the heavy lifting.

Worst case I can run a local copy of the web app if the webserver gets overloaded, since the database server is a separate machine and should be far less likely to have issues. Even when the page was down during AGDQ2 because of the excess traffic, the database itself was fine.

I moved the merging logic into the web app, for example, and even a full merge (as in downloading the entire chipin history and processing all of it from scratch) only took 30 seconds, even when running the web app locally and using a remote db. A partial merge (deleted 100 donations so they would be reprocessed) was essentially instant.

CyberBotX

2012-06-10 06:19:50 pm

Insane Killer Robot

I'll be trying to hack something together here in the next week or so in C#, I just haven't had much of a chance to do so yet (mostly because of my job, partially because of my side-job, partially because I'm a slacker [yes I'll admit that]). My first step was going to be trying to see if I could get something slight-similar to SMK's tracker, but in C#, and then we could work on enhancing it. I would hack onto SMK's if I knew Java, but I am a lot more familiar with C# and from what I have seen between Java and C#, C# is a cleaner language in my opinion. More to come, I guess. Tongue

UraniumAnchor

Edit history:

UraniumAnchor: 2012-06-14 01:00:22 pm

2012-06-14 12:59:28 pm

Not a walrus

Found something that might be relevant to why the merges were taking so long and why doing the merges on the webapp was so much faster:

http://stackoverflow.com/questions/9819271/why-is-mysql-innodb-insert-so-slow

InnoDB hates random integer primary keys, which is what the Java side generates, but the webapp merge generates sequential primary keys. I'm not sure why this wasn't a problem at AGDQ, unless the logic in the Java app changed that much. I'm guessing this just horribly multiplied the latency issues the Java app was experiencing.

I'm actually hoping to migrate the data to PostgreSQL but I need to find a reliable (and hopefully cheap) host first.

Sir VG

Edit history:

Sir VG: 2012-06-14 01:44:39 pm

2012-06-14 01:42:08 pm

Fucking Weeaboo

I use Hostgator for my stuff UA, which is quite cheap and been completely reliable for me. Some configurations allow that type of SQL, available upon request:

http://support.hostgator.com/articles/specialized-help/postgresql

UraniumAnchor

2012-06-16 03:58:37 pm

Not a walrus

VPS or Dedicated servers are the opposite of 'cheap', unfortunately.

But I have months to figure it out, so I'm not too concerned yet.

Sir VG

2012-06-16 10:42:29 pm

Fucking Weeaboo

Quote from UraniumAnchor:

VPS or Dedicated servers are the opposite of 'cheap', unfortunately.

But I have months to figure it out, so I'm not too concerned yet.

Well, I just have basic webhosting though them and don't have to deal with all the SQL stuff.

mikwuyma

2012-10-10 02:11:41 pm

My feelings on The Demon Rush

So there was talk about converting this program to C#, but nothing was ever really confirmed or completed, so I'm bumping this topic to ask a couple of questions.

-Is work being done to convert this program to C#? The program needs to be optimized way more than it was for SGDQ, because the biggest source of time wasted in donation processing was waiting for the program to load.

We still have months to figure out the dedicated server, UA, but have you (or someone else) made any progress on this end?

I want to stress something. I am 99% sure that donation traffic will be even higher for 2013 than it was this year, so we need a donation tracker that runs as quickly as possible. I don't care who optimizes the program, or if it's a team, but it needs to be done.

UraniumAnchor

Edit history:

UraniumAnchor: 2012-10-10 02:46:23 pm

UraniumAnchor: 2012-10-10 02:45:29 pm

2012-10-10 02:37:14 pm

Not a walrus

I haven't been able to work on the backend, but the backend wasn't really a problem because the website itself was always relatively quick. I'm still not sure what specifically was causing the Java app to be so slow.

I want to migrate the existing data into a new form before anybody bothers working on a C# port. I'll try to work on that later this month. I've had a few discussions with CyberBotX about it, but nothing concrete yet.

If we can't get a native app figured out by January, we still have the option of using the django admin interface, which has the advantage of people not having to muck with a bunch of putty nonsense to get the connection working. If the remote website starts being slow, we can run a local copy of the website, pointing at the same database, and do that instead. The database server itself was never slow, even at AGDQ2. The only disadvantage to that approach is that we'll need somebody to donate a laptop and run a Virtual Machine.

Cool Matty

2012-10-10 06:00:00 pm

Crawlathon WR, get down on my level.

UA: Just responding to a 4 month old host, but as far as hosting, my server does offer PostgreSQL. I know we needed to get Python going and I am willing to commit to doing that this week if you actually want to go that route.

UraniumAnchor

2012-10-10 06:01:39 pm

Not a walrus

If you're still offering that, that's the route I would like to go. Dreamhost isn't all that great for handling heavy cpu loads.

Or heavy much of anything, really.

Cool Matty

2012-10-10 06:03:33 pm

Crawlathon WR, get down on my level.

Or billing. Or customer support. Or reliability.

I'll poke you on Skype about it.

SMK

2012-10-11 01:20:32 pm

Yeah, some stuff was discussed, but then the line went quiet.

The problem with the java app was roundtrip database access: basically the ORM I was using (solely in an attempt to cut down on code) was really, _really_ bad about making multiple roundtrip requests for a single data object (I didn't really even think it would be a problem, turns out I was wrong). I tried tuning some stuff with it, but nothing really helped. At this point, I feel kind of lost, because I have zero experience in how to optimize database access properly. If anybody knows of some good solution(s) for doing C# data model stuff, I'm all ears.

One thing we have been looking at was migrating some of the tasks to the server side. I think we have the chipin data merge, and gdocs data merge implemented in django (UA can correct me on that), and I think that they're both relatively functional and efficient. Assuming we could also make prize drawing work in a similar fashion, that would remove the 3 biggest communication problems with the current app.

Yeah, some stuff was discussed, but then the line went quiet.

The problem with the java app was roundtrip database access: basically the ORM I was using (solely in an attempt to cut down on code) was really, _really_ bad about making multiple roundtrip requests for a single data object (I didn't really even think it would be a problem, turns out I was wrong).  I tried tuning some stuff with it, but nothing really helped.  At this point, I feel kind of lost, because I have zero experience in how to optimize database access properly.  If anybody knows of some good solution(s) for doing C# data model stuff, I'm all ears.

One thing we have been looking at was migrating some of the tasks to the server side.  I think we have the chipin data merge, and gdocs data merge implemented in django (UA can correct me on that), and I think that they're both relatively functional and efficient.  Assuming we could also make prize drawing work in a similar fashion, that would remove the 3 biggest communication problems with the current app.

UraniumAnchor

Edit history:

UraniumAnchor: 2012-10-11 01:48:50 pm

UraniumAnchor: 2012-10-11 01:43:20 pm

UraniumAnchor: 2012-10-11 01:42:29 pm

2012-10-11 01:41:16 pm

Not a walrus

What I want to do is just remove any direct database access from the local app entirely and rely instead on the app interfacing with the website instead, probably through JSON requests since Django's pretty good at serializing database objects that way. The reason to have a local app at all is that the website is going to have enough to do serving up pages to the general public without having to also be the primary edit interface, though maybe that won't be as much of an issue as we think.

Chipin merges work, or at least they did last I checked which was admittedly back in July, but I don't think I ever got Google Docs merge working, since that's less resource intensive.

I'll try to have prize drawings working within the next couple of weeks. There's other stuff I need/want to do first.

Question for those familiar with C#, though. How hard would it be to embed a mini web browser to handle login cookies?

CyberBotX

2012-10-11 02:44:55 pm

Insane Killer Robot

Well, the .NET framework already has a WebControl class, which may do what you are asking for.

As for the Google Docs merging, is that also going to be done on the server as opposed to in the client? I know how to work with the Google Docs API in C# but if it's going to be server side, that's a moot point. Tongue

UraniumAnchor

2012-10-11 02:47:39 pm

Not a walrus

I don't think it's as important to do that directly server side for a couple of reasons.

-It's not as high volume
-It can submitted as a giant json blob instead of a bunch of smaller requests

As long as WebControl supports cookies, then I think we're good. It'll make Access Control a lot simpler.

SMK

2012-10-11 02:48:07 pm

Quote from UraniumAnchor:

But I don't think I ever got Google Docs merge working, since that's less resource intensive.

Did you ever try the code I sent you? It worked locally, but I wasn't able to test it on a live server.

Quote from UraniumAnchor:

I'll try to have prize drawings working within the next couple of weeks. There's other stuff I need/want to do first.

I can give that a shot if you want. Given that I wrote the original logic anyways.

Quote from UraniumAnchor:

Question for those familiar with C#, though. How hard would it be to embed a mini web browser to handle login cookies?

Yeah, I'm not sure what you mean, but pretty much any HTTP control library has support for setting/sending cookies; its how I handle logging into chipin in the Java app (in fact, there's a .net port of the exact HTTP library I used, so I think we're okay if you're saying what I think you are).

UraniumAnchor

2012-10-11 03:11:57 pm

Not a walrus

Quote from SMK:

Quote from UraniumAnchor:

But I don't think I ever got Google Docs merge working, since that's less resource intensive.

Did you ever try the code I sent you? It worked locally, but I wasn't able to test it on a live server.

Must have slipped my mind, I'll go look at it again when I have time.

Quote from SMK:

Quote from UraniumAnchor:

I'll try to have prize drawings working within the next couple of weeks. There's other stuff I need/want to do first.

I can give that a shot if you want. Given that I wrote the original logic anyways.

Go for it. I might have to tweak it but anything I don't have to do is helpful.

Quote from SMK:

Quote from UraniumAnchor:

Question for those familiar with C#, though. How hard would it be to embed a mini web browser to handle login cookies?

Yeah, I'm not sure what you mean, but pretty much any HTTP control library has support for setting/sending cookies; its how I handle logging into chipin in the Java app (in fact, there's a .net port of the exact HTTP library I used, so I think we're okay if you're saying what I think you are).

Well, the issue is that the easiest way to log in to the tracker is to do so via a google account, which requires some user interaction besides just entering in a login and password. So it's not just HTTP control. It needs a browser.

[quote="SMK"][quote="UraniumAnchor"]But I don't think I ever got Google Docs merge working, since that's less resource intensive.[/quote]
Did you ever try the code I sent you?  It worked locally, but I wasn't able to test it on a live server.[/quote]

Must have slipped my mind, I'll go look at it again when I have time.

[quote="SMK"][quote="UraniumAnchor"]I'll try to have prize drawings working within the next couple of weeks. There's other stuff I need/want to do first.[/quote]
I can give that a shot if you want.  Given that I wrote the original logic anyways.[/quote]

Go for it. I might have to tweak it but anything I don't have to do is helpful.

[quote="SMK"][quote="UraniumAnchor"]Question for those familiar with C#, though. How hard would it be to embed a mini web browser to handle login cookies?[/quote]
Yeah, I'm not sure what you mean, but pretty much any HTTP control library has support for setting/sending cookies; its how I handle logging into chipin in the Java app (in fact, there's a .net port of the exact HTTP library I used, so I think we're okay if you're saying what I think you are).
[/quote]

Well, the issue is that the easiest way to log in to the tracker is to do so via a google account, which requires some user interaction besides just entering in a login and password. So it's not just HTTP control. It needs a browser.

SMK

2012-10-11 03:40:25 pm

Quote from UraniumAnchor:

Well, the issue is that the easiest way to log in to the tracker is to do so via a google account, which requires some user interaction besides just entering in a login and password. So it's not just HTTP control. It needs a browser.

You're referring to using the openID based login correct? If so, I'm sure somebody's implemented something that can do it transparently.

CyberBotX

2012-10-11 03:46:23 pm

Insane Killer Robot

There is apparently a library called DotNetOpenAuth which handles OpenID. Could look into that.

UraniumAnchor

Edit history:

UraniumAnchor: 2012-10-11 04:25:17 pm

UraniumAnchor: 2012-10-11 04:19:56 pm

UraniumAnchor: 2012-10-11 04:16:17 pm

UraniumAnchor: 2012-10-11 04:13:30 pm

UraniumAnchor: 2012-10-11 04:12:19 pm

2012-10-11 04:11:01 pm

Not a walrus

OpenID is not what needs to be figured out, it's how to log into your Google account so the OpenID system can call the correct hooks.

And Google has some intermediary pages that you need to go through first.

Edit: Let me clarify. App wants to log into the donation site. App visits http://donations.uranium-anchor.com/openid/login/, gets redirected to a Google Login page because the App isn't logged in to Google or the website itself needs permission from Google. Now what? *That's* what needs solving. It might be as simple as logging in via the website first to avoid the permission interstitial and having a popup that asks for your Google login in the App itself (maybe with an app-specific password set up in your account if you're paranoid).

mikwuyma

2012-11-08 08:00:49 am

My feelings on The Demon Rush

I was just wondering how progress on this program was going.

UraniumAnchor

2012-11-08 09:29:55 am

Not a walrus

I've been doing some work on the website side of things and last I heard SMK at least was working on the client-side integration. There's code in place on the website side to search/add/edit/delete in a low-bandwidth, low-latency(ish) way without having to directly touch the database, so that's a step up from the old Java client that needed an SSH tunnel and full database access to do anything useful.

SMK

Edit history:

SMK: 2012-11-08 12:07:08 pm

2012-11-08 10:24:59 am

School has recently dropped several trains on me, so I don't really have much spare time to devote to this as I'd like. However, come December, I will have _nothing_ to do in the span of time before Christmas, so that's when I hope to pour the coals on. As far as I'm concerned, anyone with experience with C#/WinForms is welcome to lend a hand in the meantime (and I'll be willing to discuss/explain anything necessary), though you'll have to talk to CBX about getting access to the repo if you're serious about it.

Edit:

Quote from UraniumAnchor:

If you do want to help, SMK can probably add you to the PM discussion thread we have going on, too.

Yeah, I don't think we've insulted too many people so far.. Tongue

UraniumAnchor

2012-11-08 10:44:29 am

Not a walrus

If you do want to help, SMK can probably add you to the PM discussion thread we have going on, too.