SDA 2 Prototype (sda2p)

2010-07-19 09:51:23 am

Seriously, what the hell? Sorry to be dragging this into another topic, but this is the reason your run could possibly be taken down:

Quote from DJGrenola:

His SS is full of little tricks that were missed by my predecessors and I,

The run is very improvable. I played through the first five segments last night and was about 1:30 faster than you at the first Torvus save; that's after spending one night on it. If you're against the fact that your runs could be obsoleted, well, SDA was never the place for you.

Yes, people were idiots 5 years ago. I'm sorry about that. But it has no bearing on how SDA is run today.

DJGrenola

2010-07-19 10:08:44 am

guffaw

Mike: no, I'm revoking all rights. Sorry.

moooh

2010-07-19 10:22:42 am

Exoray

Quote from DJGrenola:

Mike: no, I'm revoking all rights. Sorry.

This is most unfortunate. Your time spent on this promising project was very appreciated by the community and we were looking forward to seeing it actually come to use. Would you please reconsider and actually make this decision when you are less angry as you originally intended?
Just ignore those two blockheads coming here trying to provoke you even more. They are way out of line bringing that discussion to this topic.

Pootrain

2010-07-19 11:13:40 am

train kept rollin

It would be a shame to let all that work go to waste over a disagreement with a few members of the community. Cry

ridd3r.

2010-07-19 12:03:12 pm

we have lift off

Quote from DJGrenola:

Mike: no, I'm revoking all rights. Sorry.

You've put way too much time into this to just throw it all away, surely you could at least wait a few days before making a decision. You owe it to yourself even if you don't think the community deserves it. No matter how you feel right now you may seriously regret acting out of anger.

arkarian

2010-07-19 05:33:56 pm

gamelogs.org

i don't really think he owes anyone anything ... he can do whatever he wants with code he's written. that said, yeah it's a setback for sda 2.0 but that means someone else will have to rise to the challenge (or not).

Hattori HANZo

2010-07-19 08:04:57 pm

contraddicted

Quote from moooh:

Would you please reconsider and actually make this decision when you are less angry as you originally intended?

I second this.
I do not know what the reasons for this sudden change of opinion are, but by guts I'd say it's something rather heated. And by personal experience I can tell that heated decisions rarely are the ones you'd make two days later. If in this case they should be, then so be it. But assuring yourself not only by thought but also by time of important things is surely nothing one would regret.

And how about closing this topic for the public in the meantime?
Grenola's made his statement and obviously there's a lot of people that appreciated his work. Leaving this open is likely just to attract drama, flies and mold. (If someone disagrees with my opinion, please don't reply here.)

DJGrenola

Edit history:

DJGrenola: 2010-10-25 04:52:19 pm

2010-10-25 04:43:20 pm

guffaw

so here's the sources and test data SQL script for sda2p r3. It is almost the same as the version that was hosted on SDA for a while (r2), but I couldn't resist going after some bugs before releasing it, mainly because the game page was totally broken. It should at least work now, but one bug I don't think can be structurally fixed without a change to the database schema, and I wasn't delving into that -- instead I just fudged one case (filter by system) so PHP does the filtering instead of the database.

I also backported a fix for a bug I found later in my bbcode parser, and improved one or two other things.

I'll probably post some more comments about this later, but if you want to install and play with it then knock yourself out. I would suggest that if people are going to hack on this code then they might want to set up a repository for collaborative development. I hear google code is pretty good.

edit: about the db: there are a bunch of random performance statistics in the plog table which I didn't bother to delete. They might be of interest to someone, although I can't remember which machine I was testing on that generated them (might have been my wii). Also, I did have a load of email addresses for runners populated but they were deleted before release obviously.

so here's the sources and test data SQL script for sda2p r3. It is almost the same as the version that was hosted on SDA for a while (r2), but I couldn't resist going after some bugs before releasing it, mainly because the game page was totally broken. It should at least work now, but one bug I don't think can be structurally fixed without a change to the database schema, and I wasn't delving into that -- instead I just fudged one case (filter by system) so PHP does the filtering instead of the database.

I also backported a fix for a bug I found later in my bbcode parser, and improved one or two other things.

I'll probably post some more comments about this later, but if you want to install and play with it then knock yourself out. I would suggest that if people are going to hack on this code then they might want to set up a repository for collaborative development. I hear google code is pretty good.

edit: about the db: there are a bunch of random performance statistics in the plog table which I didn't bother to delete. They might be of interest to someone, although I can't remember which machine I was testing on that generated them (might have been my wii). Also, I did have a load of email addresses for runners populated but they were deleted before release obviously.

Attachment:

sda2p_r3_dist.zip

mikwuyma

2010-10-25 07:03:05 pm

My feelings on The Demon Rush

Thanks Grenola! Glad you're willing to help us out despite what has happened.

tenfoLd

2010-10-25 10:13:02 pm

Three cheers for DJGrenola!!! Smiley

DJGrenola

2010-10-26 02:16:33 am

guffaw

I'll answer questions if anyone has 'em.

DJGrenola

Edit history:

DJGrenola: 2010-10-26 08:31:19 am

2010-10-26 08:19:34 am

guffaw

First off, the ZIP I shipped contains an old version of the JW flash player used on the site. Although JW is free to use it is not without license restrictions which means I've probably broken various laws by including it. Further development would best be done by replacing the swfobject.js and player.swf files with the newest versions from here anyway. I just threw this in the ZIP so it would have a chance of all working out of the box.

The database is currently MySQL and uses the MyISAM table type, which is the default one. My understanding is that MyISAM has no transactional support or foreign key support, so there are no explicit foreign keys. There are no views or stored procedures and I don't think there are any subselects used since MySQL typically optimises all of these things very poorly. The majority of the SQL has been tuned for performance, which is very important for a web application that would get at least tens of thousands of hits per day. If you are not in the habit of using EXPLAIN SELECT to optimise your queries then you had better start doing it, because once MySQL decides to start creating temporary tables during its queries it won't be returning results on a live web server in 0.02s like it does on your quad-core gaming rig. Quite a bit of development was done running Linux/Apache/MySQL/PHP on a Wii whose modest CPU and even more modest RAM provided a good simulation of a web server under load. Quite a few changes were made as a result of this.

There are some weirdnesses in the database. One of the more important ones is that some tables contain a dummy entry; these are present to make various joined SELECT queries work. It wasn't until later that I came across LEFT JOIN as a means of solving this problem without having to have dummy rows in a load of tables. Try:

select * from categories where category_id=0;
select * from people where person_id=0;

You should get the idea. Ideally these entries would be removed and the relevant queries reworked to use better joins.

The database also contains this ubiquitous update_id column in virtually all of the tables. Currently everything should have an update_id of 0. This feature is not implemented yet and I haven't thought it through 100%, but it exists as a way of trying to make it possible for work to be done on the site "behind the scenes" and then be published all at once in a single update. It should also allow rollback facilities if an update is revoked. An updater would start by creating an update. The site would assign an ID to this update (call it U) and the updater would work under the context of that update. Any changes made to the site would be entered into the database as new rows with update_id=U, but the live site would continue to run with "where update_id=0" in all of its queries so would not notice the changes. At update time:

- appropriate tables are read/write locked
- rows with update_id=0 have their update_id changed to (-U)
- rows with update_id=U have their update_id changed to 0
- other stuff happens
- tables are unlocked

This also means that old data is kept in the database with a negative update_id of -U, which allows the querier to figure out in which update a change was made, and enables a wiki-style history of updates. Of course this feature also allows multiple updates to be worked on at the same time.

There are probably a lot of complex issues that would need to be sorted out before this would work though. Now I think about it I may have decided that old updates should be given a complete set of tables to themselves in order to avoid compromising performance on the live site, but the principle of the thing is still valid.

Much of the site's textual content is stored in the database as a BBCode variant. sda2p includes a parser that can be used to render this as HTML 4. Although the parser is fast enough in most cases, performance statistics gathered during the public test of r2 suggest that for e.g. really big runner comments (some are 100K+) it might be necessary to use a database cache of the resulting HTML instead. One thing worth mentioning about the parser is that its output must be treated as a block-level HTML element, because it may create block-level HTML objects like tables. Don't try to put its output inside a <span> because you'll be violating the HTML 4 spec. I believe that most if not all one-line data fields in sda2p are text-only and do not allow BBCode for this reason. Plans were to extend the parser to allow linking to objects in SDA. So you could have [game 1234] to link to a particular game, or [run 12345] to link to a run, or [person 123] to link to a runner, or [category 123] to link to a run category. The parser would probably be set up to ignore anything after the IDs, so [person 123 Mike Uyama] or [category 123 Single-Segment] would be parsed correctly but would also have the advantage of being humanly readable. Links like this have various advantages (and disadvantages) over static URLs, but none of this work has been done yet. For the most part you can forget about the parser and treat it as a black box.

There is a hidden debug mode for some pages that may be activated by adding "?budge" or "&budge" to the end of the URL as appropriate (budge being an anagram of debug).

The PHP is improvable; the whole thing basically started out life as a big hack. In particular the code might well be easier to understand if a lot of functionality that is currently in global functions were to be moved inside the data classes as class or instance methods. There are also still some instances of objects being passed around as arrays rather than as proper class instances. The code is not particularly well organised and not always self-consistent. Also, some of the IL category stuff is quite grim. :]

can't think of anything else I wanted to say right now

First off, the ZIP I shipped contains an old version of the JW flash player used on the site. Although JW is free to use it is not without license restrictions which means I've probably broken various laws by including it. Further development would best be done by replacing the swfobject.js and player.swf files with the newest versions from [url=http://www.longtailvideo.com/players/jw-flv-player/]here[/url] anyway. I just threw this in the ZIP so it would have a chance of all working out of the box.

The database is currently MySQL and uses the MyISAM table type, which is the default one. My understanding is that MyISAM has no transactional support or foreign key support, so there are no explicit foreign keys. There are no views or stored procedures and I don't think there are any subselects used since MySQL typically optimises all of these things very poorly. The majority of the SQL has been tuned for performance, which is very important for a web application that would get at least tens of thousands of hits per day. If you are not in the habit of using EXPLAIN SELECT to optimise your queries then you had better start doing it, because once MySQL decides to start creating temporary tables during its queries it won't be returning results on a live web server in 0.02s like it does on your quad-core gaming rig. Quite a bit of development was done running Linux/Apache/MySQL/PHP on a Wii whose modest CPU and even more modest RAM provided a good simulation of a web server under load. Quite a few changes were made as a result of this.

There are some weirdnesses in the database. One of the more important ones is that some tables contain a dummy entry; these are present to make various joined SELECT queries work. It wasn't until later that I came across LEFT JOIN as a means of solving this problem without having to have dummy rows in a load of tables. Try:

select * from categories where category_id=0;
select * from people where person_id=0;

You should get the idea. Ideally these entries would be removed and the relevant queries reworked to use better joins.

The database also contains this ubiquitous update_id column in virtually all of the tables. Currently everything should have an update_id of 0. This feature is not implemented yet and I haven't thought it through 100%, but it exists as a way of trying to make it possible for work to be done on the site "behind the scenes" and then be published all at once in a single update. It should also allow rollback facilities if an update is revoked. An updater would start by creating an update. The site would assign an ID to this update (call it U) and the updater would work under the context of that update. Any changes made to the site would be entered into the database as new rows with update_id=U, but the live site would continue to run with "where update_id=0" in all of its queries so would not notice the changes. At update time:

- appropriate tables are read/write locked
- rows with update_id=0 have their update_id changed to (-U)
- rows with update_id=U have their update_id changed to 0
- other stuff happens
- tables are unlocked

This also means that old data is kept in the database with a negative update_id of -U, which allows the querier to figure out in which update a change was made, and enables a wiki-style history of updates. Of course this feature also allows multiple updates to be worked on at the same time.

There are probably a lot of complex issues that would need to be sorted out before this would work though. Now I think about it I may have decided that old updates should be given a complete set of tables to themselves in order to avoid compromising performance on the live site, but the principle of the thing is still valid.

Much of the site's textual content is stored in the database as a BBCode variant. sda2p includes a parser that can be used to render this as HTML 4. Although the parser is fast enough in most cases, performance statistics gathered during the public test of r2 suggest that for e.g. really big runner comments (some are 100K+) it might be necessary to use a database cache of the resulting HTML instead. One thing worth mentioning about the parser is that its output must be treated as a block-level HTML element, because it may create block-level HTML objects like tables. Don't try to put its output inside a <span> because you'll be violating the HTML 4 spec. I believe that most if not all one-line data fields in sda2p are text-only and do not allow BBCode for this reason. Plans were to extend the parser to allow linking to objects in SDA. So you could have [game 1234] to link to a particular game, or [run 12345] to link to a run, or [person 123] to link to a runner, or [category 123] to link to a run category. The parser would probably be set up to ignore anything after the IDs, so [person 123 Mike Uyama] or [category 123 Single-Segment] would be parsed correctly but would also have the advantage of being humanly readable. Links like this have various advantages (and disadvantages) over static URLs, but none of this work has been done yet. For the most part you can forget about the parser and treat it as a black box.

There is a hidden debug mode for some pages that may be activated by adding "?budge" or "&budge" to the end of the URL as appropriate (budge being an anagram of debug).

The PHP is improvable; the whole thing basically started out life as a big hack. In particular the code might well be easier to understand if a lot of functionality that is currently in global functions were to be moved inside the data classes as class or instance methods. There are also still some instances of objects being passed around as arrays rather than as proper class instances. The code is not particularly well organised and not always self-consistent. Also, some of the IL category stuff is quite grim. :]

can't think of anything else I wanted to say right now

tenfoLd

2010-10-26 10:08:27 am

Can my name be spelled "tenfoLd" instead of "TeNFoLD" haha. It really doesn't matter....I don't even know if this is a place to post such a request.

DJGrenola

2010-10-26 12:00:41 pm

guffaw

I'm not changing the test data. It's just test data.

DJGrenola

Edit history:

DJGrenola: 2010-10-27 11:00:34 am

2010-10-27 10:58:06 am

guffaw

a few other things. category_temp was an intermediary table that was used to migrate to the current category/metacategory system. I don't think anything uses it any more and it can probably be deleted. The revision_notes fields that are on most of the tables were intended as part of the update system along with update_id. I think having them in the tables is pretty stupid though and this information should almost certainly be stored somewhere else, so the revision_notes columns should probably all be deleted too.

There's currently no text localisation support.