DJGrenola appreciation thread

2007-10-12 06:27:41 am

I love YaBB 1G - SP1!

I'm one of the perpetrators. Sad

Let me say I'm very sorry if my comments upset you. To be honest, I did not look at the news very often until after I submitted my run. Once I sent the tapes, I started checking almost every day. Now I know that you are invaluable to the site, and I meant no offense.

DJGrenola

2007-10-12 06:52:28 am

guffaw

It's fine. It was probably wrong of me to single you out for criticism, because you're not the first and I doubt you'll be the last -- just getting two like that in one update made me annoyed.

DJGrenola

2007-10-12 07:29:05 am

guffaw

OK, here's that checklist I mentioned, or a slightly slimmed down version of it, anyway ...

The observant will notice that there aren't 40 points in this checklist, but some steps have been omitted for various reasons and some have been amalgamated into one for simplicity. This is the basic shape of the operation, though.

[ ] upload to archive
- actually this is something of a lie, because nate tends to do this nowadays.

[ ] make jpegs
- takes a lot longer than one point on the checklist would suggest. With the videos staged on the new dedicated server, I run a bash script of my own devising which clips out frames at random from the run. It also remultiplexes the first 90 seconds of the first segment of the set into an AVI container, and zips up the JPEGs and the AVI file into a ZIP file for me to download. Once I've got it, I have to hope that the runner included the title screen in the first segment of the run so I can rip that from the AVI file. (If they didn't, things start to get really time consuming.) I sift through the 60 or so frame grabs from the run and pick something like a dozen reasonable looking ones for the rotating pictures on each game page (I think a lot of people don't realise that those pictures are actually on a rotation in many cases). The images often need to be cropped and resized, or their borders blackened so they don't look ugly against SDA's black background, and Radix has a maximum size constraint on game images to avoid putting too much strain on SDA's main server bandwidth, so they often need to be saved multiple times at different quality settings until they meet this target (I should automate this, but never got round to it). It takes a while.

[ ] collect file information
- Don't want to go into too much detail here, but another script is run (one of Radix's this time) which outputs information about the files comprising the run and their sizes. This information is recorded for use on the game pages (the 30 MB/100 MB/500 MB low/normal/high quality part), and is also outputted in a format that later on is fed to the page that generates the actual links to the video files themselves.

[ ] collect video length information
- archive.org's "run time" field needs to be filled in, so again I use one of my own scripts to tot up the total video length of each run. This information is again just recorded in a text file at this stage for use later on.

[ ] prepare game pages
- Mike actually does most of the work here, sending me the updated game pages via email with runs and runner comments already included. I make sure these files are in the correct character set (for some reason he often sends me them in UTF-16). Sometimes I have to run more of my own software to look for "evil" characters like smart quotes and so on that are inserted by Microsoft Word and friends. I check the pages over for mistakes, correct any I find, and fill in the file size information from Radix's script.

[ ] prepare news post
- This takes ages. Generally I don't have a lot of information about runs coming through the queue, and unless I've verified them I won't have seen the videos. I'll often read what runners have written in their comments. I also look to see if the runs are improved versions of old runs, and if they are, I'll make a note of who the vanquished runner was, the date of the old run, and work out the time improvement so I can mention these things in the news. Sometimes I'll have an idea for a cool theme or something I can place in the news update, but usually I just have to find creative ways of saying "runner A beat game B for system C in time D on category E". Because of the data collection involved in writing the news, it's a time consuming business. Writing the news for one of the 7-run updates recently (probably the mega man one, I forget) took over 2 hours.

[ ] move to directories
- A script is run (nate's -- I was elsewhere when this job needed doing) that places all the files on the dedicated server into a canonical directory structure appropriate for distribution via HTTP and BitTorrent.

[ ] make torrents
- I run scripts on the dedicated server to build torrents for whatever we're doing torrents for.

[ ] rename torrents for SS runs
- Sometimes the torrents need renaming so that the links on the run distribution page work properly.

[ ] assemble local update copy
- The JPEGs and HTML I prepared earlier are placed into a folder and zipped up ...

[ ] upload HTML and JPEGs to SDA
... and uploaded to a private directory on the SDA server, where they are unpacked. Another trivial but important operation occurs at this stage which I won't go into detail about because it has implications for security.

[ ] make backups
- Backups of anything I'm going to modify, including the news, any pictures I might be replacing, any HTML I might be replacing, the game lists, and the files that contain the information to generate the links to the video files are made at this point. This is partly automated, partly not. I introduced this step after the momentous occasion when I accidentally overwrote SilentHill.html with the contents of SilentHill2.html. It has since saved my skin on several occasions. Tongue

[ ] get archive ready
- If the items at archive haven't been checked in yet, this happens now. I also look through the archive metadata pages and note the archive servers and paths that point to the videos. I'll need this information in a minute. I used to have some tools which did this automatically, but a change at archive.org meant my tools stopped working, and I never got round to writing fixed ones, so I do it manually.

[ ] insert run data
- The file information and archive servers and paths from the previous step are now inserted into the files which generate the links to the videos.

[ ] send torrents to SDA's server
- Any torrent files I prepared earlier on FDC are uploaded to SDA's server and placed in a private directory for the time being.

[ ] take videos and torrents live on the dedicated server
- Self explanatory ... I just move them from a private directory to the live one, but as always when I'm moving files I triple check to make sure I'm not about to screw anything up.

[ ] download, check torrents
- The created torrents are downloaded to my machine and loaded into Azureus. I start them going, check that I'm getting a download on them all, and check that they contain the files they're supposed to contain.

[ ] take torrents live on SDA's server
- The torrent files I uploaded to SDA earlier are moved into their distribution directory. Sometimes we seed torrents from the main server as well if we have spare bandwidth, in which case I'll let them download from the FDC dedicated machine until a second seed is created.

[ ] deploy flash
- The flash video files are moved to their deployment directory.

[ ] spider
- A perl script (my own) is run which goes through the video and torrent links on the site and checks each and every one to make sure it's a good link. This can take a while for long multisegment runs, but again this is a tool which has prevented botched releases on a number of occasions.

[ ] install jpegs
- With the videos and torrents all ready to go, all that's left is to update the site itself. The pictures I made right at the start of the process are installed.

[ ] post game page
- New and updated game pages are installed, and I check through them as quickly as I can to make sure that there's nothing glaringly wrong with them.

[ ] check game page links
- I check to make sure that the clickable times on the game pages work.

[ ] post news
- Self explanatory, but do not underestimate the time it takes to check everything.

[ ] edit queue
- The runs I've posted are removed from the queue.

[ ] update gamelists
- This is actually several points on the real checklist, but the game lists are updated with new games and revised times for improved runs.

[ ] remove obsoletes
- Obsolete runs are removed from the run list files. This is mainly just to keep them tidy.

[ ] populate archive page
- This is the last thing I want to do at this point, but archive.org detail pages need to be filled in for every run I've posted. These are done manually, with a little help from some JavaScript snippets devised by Radix. This is one step that really could do with more automation. I'd quite like to write a Firefox extension that would make this easier, but I doubt I'll get round to it any time soon.

[ ] generate news feeds
- News feeds are generated and checked to make sure they're OK.

[ ] deploy live news feeds
- News feeds are deployed.

[ ] back up
- All files changed in the update are backed up offsite.

OK, here's that checklist I mentioned, or a slightly slimmed down version of it, anyway ...

The observant will notice that there aren't 40 points in this checklist, but some steps have been omitted for various reasons and some have been amalgamated into one for simplicity. This is the basic shape of the operation, though.

[ ] upload to archive
- actually this is something of a lie, because nate tends to do this nowadays.

[ ] make jpegs
- takes a lot longer than one point on the checklist would suggest. With the videos staged on the new dedicated server, I run a bash script of my own devising which clips out frames at random from the run. It also remultiplexes the first 90 seconds of the first segment of the set into an AVI container, and zips up the JPEGs and the AVI file into a ZIP file for me to download. Once I've got it, I have to hope that the runner included the title screen in the first segment of the run so I can rip that from the AVI file. (If they didn't, things start to get really time consuming.) I sift through the 60 or so frame grabs from the run and pick something like a dozen reasonable looking ones for the rotating pictures on each game page (I think a lot of people don't realise that those pictures are actually on a rotation in many cases). The images often need to be cropped and resized, or their borders blackened so they don't look ugly against SDA's black background, and Radix has a maximum size constraint on game images to avoid putting too much strain on SDA's main server bandwidth, so they often need to be saved multiple times at different quality settings until they meet this target (I should automate this, but never got round to it). It takes a while.

[ ] collect file information
- Don't want to go into too much detail here, but another script is run (one of Radix's this time) which outputs information about the files comprising the run and their sizes. This information is recorded for use on the game pages (the 30 MB/100 MB/500 MB low/normal/high quality part), and is also outputted in a format that later on is fed to the page that generates the actual links to the video files themselves.

[ ] collect video length information
- archive.org's "run time" field needs to be filled in, so again I use one of my own scripts to tot up the total video length of each run. This information is again just recorded in a text file at this stage for use later on.

[ ] prepare game pages
- Mike actually does most of the work here, sending me the updated game pages via email with runs and runner comments already included. I make sure these files are in the correct character set (for some reason he often sends me them in UTF-16). Sometimes I have to run more of my own software to look for "evil" characters like smart quotes and so on that are inserted by Microsoft Word and friends. I check the pages over for mistakes, correct any I find, and fill in the file size information from Radix's script.

[ ] prepare news post
- This takes ages. Generally I don't have a lot of information about runs coming through the queue, and unless I've verified them I won't have seen the videos. I'll often read what runners have written in their comments. I also look to see if the runs are improved versions of old runs, and if they are, I'll make a note of who the vanquished runner was, the date of the old run, and work out the time improvement so I can mention these things in the news. Sometimes I'll have an idea for a cool theme or something I can place in the news update, but usually I just have to find creative ways of saying "runner A beat game B for system C in time D on category E". Because of the data collection involved in writing the news, it's a time consuming business. Writing the news for one of the 7-run updates recently (probably the mega man one, I forget) took over 2 hours.

[ ] move to directories
- A script is run (nate's -- I was elsewhere when this job needed doing) that places all the files on the dedicated server into a canonical directory structure appropriate for distribution via HTTP and BitTorrent.

[ ] make torrents
- I run scripts on the dedicated server to build torrents for whatever we're doing torrents for.

[ ] rename torrents for SS runs
- Sometimes the torrents need renaming so that the links on the run distribution page work properly.

[ ] assemble local update copy
- The JPEGs and HTML I prepared earlier are placed into a folder and zipped up ...

[ ] upload HTML and JPEGs to SDA
... and uploaded to a private directory on the SDA server, where they are unpacked. Another trivial but important operation occurs at this stage which I won't go into detail about because it has implications for security.

[ ] make backups
- Backups of anything I'm going to modify, including the news, any pictures I might be replacing, any HTML I might be replacing, the game lists, and the files that contain the information to generate the links to the video files are made at this point. This is partly automated, partly not. I introduced this step after the momentous occasion when I accidentally overwrote SilentHill.html with the contents of SilentHill2.html. It has since saved my skin on several occasions. :P

[ ] get archive ready
- If the items at archive haven't been checked in yet, this happens now. I also look through the archive metadata pages and note the archive servers and paths that point to the videos. I'll need this information in a minute. I used to have some tools which did this automatically, but a change at archive.org meant my tools stopped working, and I never got round to writing fixed ones, so I do it manually.

[ ] insert run data
- The file information and archive servers and paths from the previous step are now inserted into the files which generate the links to the videos.

[ ] send torrents to SDA's server
- Any torrent files I prepared earlier on FDC are uploaded to SDA's server and placed in a private directory for the time being.

[ ] take videos and torrents live on the dedicated server
- Self explanatory ... I just move them from a private directory to the live one, but as always when I'm moving files I triple check to make sure I'm not about to screw anything up.

[ ] download, check torrents
- The created torrents are downloaded to my machine and loaded into Azureus. I start them going, check that I'm getting a download on them all, and check that they contain the files they're supposed to contain.

[ ] take torrents live on SDA's server 
- The torrent files I uploaded to SDA earlier are moved into their distribution directory. Sometimes we seed torrents from the main server as well if we have spare bandwidth, in which case I'll let them download from the FDC dedicated machine until a second seed is created.

[ ] deploy flash
- The flash video files are moved to their deployment directory.

[ ] spider
- A perl script (my own) is run which goes through the video and torrent links on the site and checks each and every one to make sure it's a good link. This can take a while for long multisegment runs, but again this is a tool which has prevented botched releases on a number of occasions.

[ ] install jpegs
- With the videos and torrents all ready to go, all that's left is to update the site itself. The pictures I made right at the start of the process are installed.

[ ] post game page
- New and updated game pages are installed, and I check through them as quickly as I can to make sure that there's nothing glaringly wrong with them.

[ ] check game page links
- I check to make sure that the clickable times on the game pages work.

[ ] post news
- Self explanatory, but do not underestimate the time it takes to check everything.

[ ] edit queue
- The runs I've posted are removed from the queue.

[ ] update gamelists
- This is actually several points on the real checklist, but the game lists are updated with new games and revised times for improved runs.

[ ] remove obsoletes
- Obsolete runs are removed from the run list files. This is mainly just to keep them tidy.

[ ] populate archive page
- This is the last thing I want to do at this point, but archive.org detail pages need to be filled in for every run I've posted. These are done manually, with a little help from some JavaScript snippets devised by Radix. This is one step that really could do with more automation. I'd quite like to write a Firefox extension that would make this easier, but I doubt I'll get round to it any time soon.

[ ] generate news feeds
- News feeds are generated and checked to make sure they're OK.

[ ] deploy live news feeds
- News feeds are deployed.

[ ] back up
- All files changed in the update are backed up offsite.

trihex

2007-10-12 07:47:28 am

Yoshi's eggs are at my mercy!

....

HOLY. FUCKING. SHIT.

Do you still have energy left to breathe after doing those mega-updates?!

BTW, when Radix was doing all the updates, was there far less automation in the process for him compared to now?

DJGrenola

Edit history:

DJGrenola: 2007-10-12 08:17:49 am

2007-10-12 08:15:05 am

guffaw

yeah, I took one look at the state of things when I started and immediately started writing tools. The JPEG generator, link spider, auto-archive server lookup script (although that no longer works), video length summer (Radix had his own tool to do this -- only runs on win32 though, so not much use when the files are all sitting on a UNIX machine) and a script to check codecs / framerates / resolutions were all added to the toolbox at this point. I also started writing a perl thing to populate archive pages, but it was never finished. The other thing that happens now which didn't happen in the old days is that I maintain spreadsheets for individual-level runs, which saves time in the long run.

The site's throughput is higher now than it was, though.

It's not too awful if there are only 2-3 runs in an update, but doing 6 or 7 at once as I've done three times recently is an all-day job.

Sir VG

2007-10-12 09:44:07 am

Fucking Weeaboo

I created this some time ago to use at another forum, but it sooooo needs to be used right now.

VorpalEdge

2007-10-12 05:28:56 pm

welcome to the machine

That list really is quite... amazing.

I must say, neither archive nor the rotating game pics script seems to be very friendly to you. :/

TheQuietMan

Edit history:

TheQuietMan: 2007-10-13 12:48:42 am

2007-10-13 12:47:26 am

Complete. Global. Saturation.

Quote:

Oh no wait, I hate Final Fantasy X.

With shit like this, I don't blame you:

Psychochild

2007-10-13 03:17:33 am

Grenola?! **** Grenola! I eat Grenola for breakfast. With yogurt. I just mix it right in there. It's delicious.

But y'know I love. I guess.

Cizol

2007-10-13 04:33:48 pm

I love YaBB 1G - SP1!

Just registered to say thanks to everyone who is involved in making this site what it is.

DigiMe

2007-10-14 12:56:38 pm

Thank you for your hard work on my favorite website for the last 2 and a half year or so. I really enjoy comming to this site and lurk in the shadows & watch speedruns.

I think it's good to say thank you to the staff sometimes Smiley

Thread_Starter

2007-10-15 02:37:39 am

yeah.... thanks for deleting my first post.

anyways.. Dj and the rest of SDA staff rule!

Axel Ryman

2007-10-15 07:14:13 am

We all scream for Eyes Cream

DJGrenola thanks for putting up my CC run. Sorry for forgetting you. >_< Still don't know everyone on the site so I'll make sure to remember you on my next run.

Do I get my Christmas Card now? >_>

DJGrenola

2007-10-15 10:11:48 am

guffaw

sure. and I'm sorry for picking on you.

AngerFist

Edit history:

AngerFist: 2007-10-15 12:42:30 pm

2007-10-15 12:41:27 pm

Angered Me, You have!

Dj Grenola is the heart of SDA. I wonder when we DJ Grenola fans can see him speedrunning Metroid Prime 2 Echoes?

DJGrenola

2007-10-15 02:26:15 pm

guffaw

you can count on angerfist.

ShadowWraith

2007-10-15 06:25:28 pm

.

Didn't he already make an MP2E run? Skipping Spacejump or something. I remember enjoying it because of the voice commentary that came with it. Smiley

Bottles

2007-10-15 06:26:36 pm

It's on archive right?It has boundry violation I believe.

DJGrenola

2007-10-15 08:45:07 pm

guffaw

It was low% (the no space jump game never interested me that much -- too arbitrary)... I skipped 6 "unskippable" items (two translators, gravity boost, annihilator beam, grapple beam, and echo visor). Lower percentages than this are possible, but are probably less suitable for speed runs (I think 16% might be plausible, but I'm not doing it).

http://www.archive.org/details/MetroidEchoes_17p_155

The run that Angerfist is talking about is a theoretical improvement to the 1:43 any% on the site, done by SDA's rules this time. I've wanted to do it for a while, but have so far failed to get around to it. Don't know how much faster I could go, but 1:40 seems to be a fair target. The last run took 3 months of pretty solid work though, so don't expect anything to happen in the near future.

Another run it would be fun to do would be an any% with boundary violations, really because I'd like to know how much faster it would be than the any% without them.

AngerFist

2007-10-15 09:02:33 pm

Angered Me, You have!

Quote:

The run that Angerfist is talking about is a theoretical improvement to the 1:43 any%

Theoretical? Aw come on! Smiley

We discussed about this a few times in irc and did we not conclude that you could improve it by at least 10 minutes or so?

HeroPenguin

2007-10-16 10:11:34 pm

Rock on, Earthbound. Rock on.

I like seeing DJGrenola's posts on the main page. Always good grammar and proper punctuation, despite his workload and the coding monster he tackles which I can't even begin to fathom, and he even bothers to make fun titles to his posts and slips jokes in there.

We love ya, DJ, don't ever doubt it!

(I came out of forum hibernation to do this! If I hadn't found this thread I was about to make it myself)