I haven't written much about the Fairpole software lately, and frankly, I haven't done much about it, either. I'm in one of those thinking phases, where you keep mulling over the issues you're trying to deal with, letting them stew and simmer in your mind until that light bulb goes on and you suddenly have the solution you've been looking for.

Every system has its bottleneck, the part of the system where things move the slowest. When I designed Fairpole, I did so under the assumption retrieving data out of the database would be the slowest part of the system. So I essentially designed the system from the back to the front, focusing on making the database access as efficient as possible.

But after watching it in action, that assumption turned out to be wrong. The slow part is the dynamic page generation: turning the data into a web page. So now I need to turn my architecture around and design it front to back, instead. Hopefully, as a result, we'll have a system that's efficient on both ends.

I haven't quite figured it out yet, but I kinda have an idea what the new architecture is going to look like. At the very front, I'm going to have an Atom server to handle all the incoming data. Page generation will take place in two phases: first when the data is saved, and at runtime. To speed up page generation, we'll try to limit what is generated at runtime to those things you truly cannot know until runtime: member information and preferences, mostly. Everything else will be generated as the data is saved.

Also, I had a bit of a revelation when I was reading Stefan Geens' essay about the limits of Spotlight, the new Mac search tool. I had planned to build in some sort of folksonomy along the lines of the one on Flickr. But his desire to have name-value pairs instead of just single-word tags lit a light bulb for me, and by using name-value pairs, I realized that would solve a lot of problems for me, helping me kill about five birds with one stone. The bad part is that in order to do that, I'll have to change my database schema, which in turn means I'll have to rewrite nearly all of my code. Again.

But I'm in this to see if I can do this right and to have fun in the process; I'm not trying to compete with anyone else but myself; so if it takes me a little longer to get it done, so be it. I'm in no hurry. I think that sometime next week or so, after I figure out the last few missing pieces, I'll begin the recoding. I appreciate your patience.