Baseball Toaster Fairpole
Help
Fairpole Status
2005-06-02 23:50
by Ken Arneson

I haven't written much about the Fairpole software lately, and frankly, I haven't done much about it, either. I'm in one of those thinking phases, where you keep mulling over the issues you're trying to deal with, letting them stew and simmer in your mind until that light bulb goes on and you suddenly have the solution you've been looking for.

Every system has its bottleneck, the part of the system where things move the slowest. When I designed Fairpole, I did so under the assumption retrieving data out of the database would be the slowest part of the system. So I essentially designed the system from the back to the front, focusing on making the database access as efficient as possible.

But after watching it in action, that assumption turned out to be wrong. The slow part is the dynamic page generation: turning the data into a web page. So now I need to turn my architecture around and design it front to back, instead. Hopefully, as a result, we'll have a system that's efficient on both ends.

I haven't quite figured it out yet, but I kinda have an idea what the new architecture is going to look like. At the very front, I'm going to have an Atom server to handle all the incoming data. Page generation will take place in two phases: first when the data is saved, and at runtime. To speed up page generation, we'll try to limit what is generated at runtime to those things you truly cannot know until runtime: member information and preferences, mostly. Everything else will be generated as the data is saved.

Also, I had a bit of a revelation when I was reading Stefan Geens' essay about the limits of Spotlight, the new Mac search tool. I had planned to build in some sort of folksonomy along the lines of the one on Flickr. But his desire to have name-value pairs instead of just single-word tags lit a light bulb for me, and by using name-value pairs, I realized that would solve a lot of problems for me, helping me kill about five birds with one stone. The bad part is that in order to do that, I'll have to change my database schema, which in turn means I'll have to rewrite nearly all of my code. Again.

But I'm in this to see if I can do this right and to have fun in the process; I'm not trying to compete with anyone else but myself; so if it takes me a little longer to get it done, so be it. I'm in no hurry. I think that sometime next week or so, after I figure out the last few missing pieces, I'll begin the recoding. I appreciate your patience.

Comments
2005-06-04 17:52:55
1.   Doug N
Not only do I think Baseball Toaster is working exceptionally well, I think it's very cool that you share your thoughts on the production of the system. It all goes pretty much straight over my head, but the compliment and thanks are genuine.
2005-06-05 09:22:38
2.   dbt
One thing I'd suggest taking a look at -- there's a web toolkit called clearsilver. See www.clearsilver.net. It's got a very fast C template engine, and bindings to write code in anything (ocaml, perl, python, ruby, ...).

I've been using it for a while. Trac, the python open source bug tracking/wiki/subversion interface available from edgewall.com, uses it.

2005-06-09 10:04:24
3.   everett
Since webpage generation is a performance suck, maybe for your comment pages you can create a button or link that says give me all comments (and the original web page) after a certain comment number...

On the dodger thoughts webpages you've got tons of people refreshing some 200 - 300 comments who are only really interested in comments > 325 and the ability to post more comments... each time they refresh they're only interested in the new comments generally...

seems like this might save you quite a bit of performance suckage.

2005-06-10 17:08:06
4.   Ken Arneson
Thanks for the clearsilver suggestion, dbt. I've been using Template Toolkit, but clearsilver might be a better fit for what I need. I'll try it out.

I've got some simliar ideas on the comment refreshing, everett. But first, I gotta fix the templating itself.

Comment status: comments have been closed. Baseball Toaster is now out of business.