Saw this in my feedlist the other day, and wanted to share it with the world: MySQL and Python and Unicode. Excellent coverage of the issues making a full toolchain Unicode compliant when using Python and MySQL.

I learned this the ‘hard way’, and I am glad someone came to the same conclusion. I was using a library to get the connection from MySQL, so I had to mod the code to make the “SET NAMES utf8″ call on init. Now I am sorry for not blogging it sooner.

Tahpot, looks like we are covering a lot of the same territory, feel like sharing notes?

Popularity: 13%

Mark Fletcher, CEO of Bloglines, posted a nice piece: Stealth Start-Ups Suck.

He actually has a lot of bright things to say in this so-called rant.

My rule of thumb is that it should take no more than 3 months to go from conception to launch of a new web service.

I would say that might be a little bit tight, but it is definetly in the ballpark. The heavy hitting on FeedLounge started in April, and we launched the Alpha in June, so we are good there.

Why go fast? Many reasons:
[1]- First mover advantage is important.
[2]- There is no such thing as a unique idea. I guarantee that someone else has already thought of your wonderful web service, and is probably way ahead of you. Get over yourself.
[3]- It forces you to focus on the key functionality of the site.
[4]- Being perfect at launch is an impossible (and unnecessary and even probably detrimental) goal, so don’t bother trying to achieve it. Ship early, ship often.
[5]- The sooner you get something out there, the sooner you’ll start getting feedback from users.

1. Important when you can get it, however, in the end, its the users that choose the leader. If a competitor can come in head and shoulders above the leader in functionality, it’s often possible to unseat the first mover.
2. Agreed. What we are doing is not unique in its various bits and pieces, in fact it’s the combinations of non-unique ideas in interesting ways that help make FeedLounge compelling to the user. Without a great user experience, there is no permanent user (and that’s no unique idea, just often forgotten in the ‘next rev’ mentality).
3. Absolutely. Our focus ended up being sharp enough to kill some really cool stuff before we launched. Do we wish we launched with it? Yes. Are we sad for launching when we did? No. Are our alpha users jumping up and down about how kick-ass we are? You bet.
4. Sure, early and often works in open source, there is no reason it wouldn’t work here. With only one deployment, your service should be updated regularly.
5. This is the key point people! If you sit back in a vacuum for a year, you are not getting the feedback that makes the service great. You might think it is absolutely the coolest thing, but if you are out that long, when you come into the light, people will be saying ‘What the hell is that???’ Knowing what features our users want in the service by direct communication right from the start? PRICELESS.

The success of a web service is inversely proportional to the secrecy that surrounded its development.

I don’t think this is necessarily true. I think most people that are keeping something secret are also paranoid and just straight-up crazy. Crazy almost always leads to the lack of success.

Web services have many advantages over shipping software. You can continuously update the service, fix bugs and add new features. There are no long development cycles. Embracing this is a key to success. The first version (or several versions, probably) of any service you create is most likely going to suck. And that’s ok. Your service won’t scale to handle a lot of traffic. It will be missing a huge amount of functionality.

Web services have their adavantages and disadvantages. I think that growth and feedback are great things, but you don’t need to open the kimono before you are ready. Having hundreds of users say ‘where is feature x?’, where feature x is the basic idea behind your service, is not useful feedback. It frustrates the users because a baseline doesn’t exist, and you are frustrated because you ARE working on it. There is also no need to invite a million users over, have the flash crowd toast your server, then say to yourself “Hey, this thing doesn’t scale to a million users”. You already knew this, and all it does is hurt your perception in the marketplace, IMHO. FeedLounge is in alpha testing now to work out major bugs (Hey, this is broken!), get user feedback (What the fsck were you thinking here?), and make sure that we can fix the scaling bottlenecks as we open the valve. We are growing by community feedback in a few quick, large steps, not one giant leap.

A passionate user is one of your greatest assets. And I would argue that the only thing of real value a web service has is its users. They act as advertising for you, telling all their friends about your service. They are the best source of new feature ideas. And they are the best Q.A. testers you can get.

Agree, agree, agree and agree. Couldn’t have said it better. I couldn’t have asked for better alpha testers, and the amazingly large response for beta signups has blown apart my expectations. It is very apparent from our user comments that the world is ready for another web based feed reader, and we are positioned to deliver. Are you watching?

Great post Mark, and keep more like that coming!

Update: Russell Beattie added to the topic, defending 24HourLaundry:

The thing that annoys me most about the comments on 24H Laundry is this ass-backwards sentiment about how there are no unique ideas left out there.

Mark said that there are no unique ideas, and I agreed with him. Depends on your definition of unique, and how far it has to go. If it is just an idea, there are no unique ideas. There may be unique implementations, but I firmly believe that someone out there has thought of it before. They just may have not done something about it.
And I was not and will not jump on the 24HourLaundry sucks bandwagon. People do what they do because they think they are doing the best thing at the time. I can respect someone else’s decision to do that, but that is definetly not the way I would take it. Then again, I am not as popular as Marc Andreesen.

Update: Ross Mayfield posted this little gem:

- Part of the rationale for stealth is competition. But there are more leaks than plumbers, and getting ahead of your competitors matters less than getting in bed with your users.
- Part of the rationale for stealth is stardom. When media was broadcast, you countered lack of access with exclusivity. Now you need to be inclusive from the get go.

Just make sure when you invite your users to bed that there is a bed to sleep in. Inclusive from the get go, yes. Talking blue sky because you don’t have anything, no.

Popularity: 23%

WIth all the rumors about Google starting an online payment service, I wanted to share my simple idea of a payment service that would become ubiquitous on the web. While I fully expect something from Google to be simpler to use than Paypal, and scale much larger, I would love to see Google take on micropayments.

The basic idea is to support micropayments between millions/billions of buyers and sellers. To acheive this, payment processing would need to be revolutionized:

Simple identity

Using something simple like sxip or similar, so that login and other high costs parts of the transaction go away from a user interaction perspective.

Front $1 to every user of the system

Or, alternately, have the user put the money into their accounts in larger chunks ($10-$20 or more). This will be the first key to a massive micropayment system, acting as a clearinghouse until the bill amount reaches an amount that the transaction can be passed to normal payment processors (PayPal, Visa, banks, etc). If a company like Google fronted $1 to each user of the system, that amounts to a maximum of $6 billion global maximum, they should be able to cover that.

Reduce the cost of a transaction

Confirm and process the requests in realtime, with no transaction log. A transaction log of billions per day would be a cost to great to bear for the system to work. Figure out what the system cost of one transaction (debit to one user, credit to another) is, set some cost per transaction, and let the system run. Invidiual vendors would be able to charge whatever they wanted, and could batch up transactions until they meet some minimum that represents profit to them.

Fraud prevention

Since you are not logging transactions, some sort of fraud tracking/prevention program will have to come into place. Google could use their IP in ‘click fraud’ here as well, or you could push some of the liability to the vendor of the transaction, and allow them to choose to log the transaction or merely refund any fraud requests.

If Google actually does something like this, I believe it will be the biggest impact to the current web that they have acheived to date.

Update: Charlene has some comments in the same vein, as well as David.

Popularity: 23%

Looking at a few if/elif/elif statements in my code, I thought I should use a switch statement instead. Who knew Python didn’t have one? I didn’t until this morning.

I found a post by Simon Willison on the subject, good stuff.

I will probably go with a dictionary here, as most of my ’switches’ are known prior to runtime.

Popularity: 23%

Tired of carrying my Windows desktop to Windows-only consulting gigs, I broke down and bought a Dell Inspiron 9300, and as soon as it arrived, upgraded it to 2GB of RAM. This is my first laptop with 2GB, the Powerbook 17 having 1.5 and about 4 others had 1GB. Note: I am selling the 2 256MB DIMMs on ebay now starting at $5, if anyone is interested.

I choose the 1900×1200 screen because I am a sucker for real estate, and although the 9300 has the grace of a boat anchor, it will do the job well running a database, app server, and application all at the same time. I won’t miss the 20″ LCD that my desktop has with a screen like this, either. This reduces my need to have putty be a tabbed-terminal application, although I would pay for something that did that.

I guess I need to get another laptop stand to put on the left side of my desktop display, so I will have a 17, 20, 17 layout, using Synergy to go between them. And if I can wrangle the Honeywell keyboard back from Jason, I should be one happy camper.

The Powerbook will still be my primary machine for mail, app development, etc, as the user experience is hard to beat. Mail.app may not be the best mail client, but its spam filtering works.

I might need to consider just upgrading the desktop to Windows Server 2003, so it can host a shared copy of SQL Server for all the other machines.

Popularity: 21%

Decided that since I was rotating all of this software around, I might as well take the hit and upgrade to Tiger.

Backed up the Powerbook to external disk last night via rsync, then started the upgrade to Tiger while I ate lunch. Took about an hour to install Tiger, and now it looks like about 2-3 more hours for Mail to re-index so that I can operate again.

First impression: Man this thing is slow! I can only hope that it is just spotlight doing the first index thing that is making this thing crawl right now.

Looked at the widgets, they are nice eye candy. I like the addition of more colors in the widgets, makes the occasional trip to the dashboard brighten up that small part of my day.

I can’t say much else until this indexing thing is finished… Spotlight is estimating 119 more days… UGH!

Popularity: 19%

Update: Yes, Mac OS X users have to manually delete 44 files in one of the steps of the upgrade. I say live with it. I would rather have the DUN upgrade than wait for an automated upgrade to be tested and released in another 4 months.

Check it out here

I ran the update last night. When they say expect to spend 30 minutes, they are kidding, it’ll take an hour or more. I did everything over Bluetooth sync, and it all went just fine. After about an hour I had my phone back, and it still made calls and had my data in it, so I have that going for me.

I had previously been using the ’shadowmite’ DUN hack to use the phone as a Bluetooth modem, and refused to update to the last firmware release because of it. Now, Bluetooth DUN is fully supported, and it still works. It actually even has a cute little icon that changes from the normal Bluetooth icon to a laptop/bluetooth icon to tell you that it is connected to the computer. The 15 second power up/down seems to be gone, and you even get a warning if you are on a data connection and try to make a call.

Nice work, Palm and Sprint, even if it is a little bit late.

Popularity: 20%

It seems my body is adjusting to the 3-4 hours of sleep I am getting every night. Since we set a date for the alpha launch of FeedLounge, my days have been filled with getting the work done for my clients, and the evenings, nights, and weekends have been filled with non-stop coding, testing, bug fixing, and deployment. I even threw in my own wedding for good measure, just to see if I could keep up. Oh yeah, and let’s landscape the house and have the wedding there… You can see that my TODO list has been long for quite a while.

It seems as if my coffee maker is set up for retirement in the next few weeks, as I have been drinking 2 pots a day for the last 2 months. I may have to upgrade it to one of those fancy cup-at-a-time units. Any suggestions?

From a personal perspective, the alpha testing is going quite well. Our crack team of testers is finding the bugs, and we are trying to knock them down within 48 hours, although on occastion that has not been the case. The server seems extremely stable, and I have learned to start weening myself from my ‘command and control’ center of 6 terminal windows open to the server. I can imagine this is what it feels like when you get your first baby. You are afraid to move your eyes away for one second, lest something bad happen! Team morale is still quite high, and we are chipping away at the feature list for the beta. The outrageously overwhelming response from the community for an application like this is also pushing us just a bit more as well. For all the beta testers that signed up, hold tight, we are making progress. And to the alpha testers, thanks for keeping vigilant with the bug reports and feedback.

Popularity: 28%

Welcome to the second of a series of posts on the development. If you missed the first one, check it out here: FeedLounge development: the parser.

The feed validator

We have feeds being parsed, but we also wanted to help make the world a better place by allowing the end user to know whether or not the feed is valid. As Geof rightly points out, we MUST follow Postel’s Law when parsing feeds (”be conservative in what you do, be liberal in what you accept from others”). Why not give a quick heads-up to someone that might be able to help fix the problem?
So, in FeedLounge, there is a banner near the item content that tells the user the feed is invalid. The user is given a link to click on to see for themselves what is wrong with the feed, using the service from feedvalidator.org.

Hopefully someone will come along that knows the person publishing the feed, and helps nudge them to fix the problem, to make the world a better place for all feed reading entities. If this banner gets too annoying, it can be hidden down to a small icon, and the user can go on with her feed reading. And of course this setting is persistent, so the user does not continue to be annoyed.

Interesting stats around validation

Of all the feeds that FeedLounge is currently parsing, validating, and tracking, 34.5% have some issue, 22.5% are NOT valid, and 17.8% are completly broken (404, 304, no response at all, etc). That’s 1/5 to 1/3 of all feeds that FeedLounge might not allow the users to read if we were using a strict feed parser!
As a side note, a large portion of the invalid feeds so far are from some version of WordPress. There are a lot of users out there that haven’t felt that upgrading is important. Please upgrade!!! The security fixes alone are worth it. I hope in a future release that the WordPress team might use one common codebase for feed creation, rather than separate code for each feed format. Disclaimer: dotnot.org uses WordPress 1.5.1.2, and will not change to something else anytime soon. I like WordPress, just offering constructive criticism, and I just want to give the WP team a friendly nudge from time to time

Popularity: 36%

We have noted in our alpha invitations that we intend for FeedLounge (company, people and application) to be as open as we can possibly be. So along those lines, I will be posting here and on the FeedLounge Blog about architecture, features and development of FeedLounge, so that everyone can see inside the beast, so to speak.

Which feed parser should we use?

When are you building a web based feed reader like FeedLounge, having data to read is step one. Luckily, there are many feed parsers already out there, so the “build vs. buy” decision was fairly easy. Focusing on the development of the user experience of the feed reader, the feed parser part of the application is only a ‘necessary evil’ in the scheme of things. After checking out several possiblities, including using my own Java/SAX framework, we decided on feedparser, the canonical namesake of the feed parsing world. Built by Mark Pilgrim, and currently at version 3.3, this is probably the most forgiving feed parser on the planet. Had I gone with my own solution, I would have spent months and months creating something as good. And with a liberal open source license, I am allowed to use it in a commercial project like this.

feedparser features

• feed format support - v3.3 has impress support of 4 feed formats and 15 different versions of those formats. This probably would have taken a good chunk of time to come up with support for.
• encoding detection - Anyone who has done this understands the difficulty without any explanation.
• tidy support - Want clean HTML content as output? No problem, it’s in there
• translated access between specific terms - If you know channel instead of feed, these are the same thing in feedparser. Use the terms that you are comfortable with.
• relative url support - Useful to us since we are ripping the feed apart to store it. Having no relative URLs is a great relief.
• great documentation - Mark produces some of the best, most-useful documentation in the open source world. feedparser is no exception here. Terse, but covering what you need to know. Need to do 401 auth? Here. Wondering about E-Tag support? There.
• over 2000 unit tests - I may run into some arcane case not covered here, but the likelihood is not very high.
• HTML sanitizing - Extremely useful for a feed reader, to prevent bad things. You don’t want to let someone else’s JavaScript run inside your app. Debugging that would be a nightmare, and maliciousness is also a concern.
• date parsing - Support for every date format they came across. You get a simple date format, consistent from feed to feed.
• It just works!- The best is saved for last, as this point cannot be made often enough. In the months of development so far, feedparser has never been the spotlight of a single problem. The closest we have come to some kind of problem is not checking for the existence of some item before accessing it. feedparser has been a huge net positive on development, with an almost nil overhead. To have alpha testers say that some of the feeds that don’t open in nearly anthing else show up in FeedLounge, that wasn’t us, it was feedparser and its magic voodoo.

Mark, thanks a million. I know you have ‘gone dark’ in the blogging world, but you are still rocking mine.

Popularity: 34%

I used to hate logging into the WP admin screens to do anything. It was a user experience nightmare. But Alex pointed me to WordPress Administration Design: Tiger, and I think I can bear to make another post. This is one of the best admin interfaces I have used, and what is amazing is that:

• It’s all just CSS/image changes
• It still doesn’t use a tree control (a favorite in any developer’s toolbox)

Anyone, and I do mean anyone using WordPress without this plugin is just torturing themselves. Can we get this as the default admin interface in 1.5.2 or better, guys?

Popularity: 25%

I should have followed this earlier, but since we launched FeedLounge yesterday, I started checking the search engines for mentions of FeedLounge. Yes, before you ask, we were already watching PubSub, Feedster, Technorati, populicio.us, etc. But Google, Yahoo, and MSN are still pretty important.
So, 24 hours after the launch, the stats for a ‘feedlounge’ search are thus:

1. MSN - 636 hits
2. Yahoo - 124 hits
3. Google - 2 hits

2 hits with Google. WTF! Earlier yesterday, Alex had prepared and submitted a Google SiteMap, hoping that would help the situation.
Fast-forward to 34 hours post launch, and a full 24 hours after the SiteMap submission, the stats are now:

1. Google - 674 hits
2. MSN - 636 hits
3. Yahoo - 261 hits

That changes things a bit. Is there any equivalent to SiteMap for Yahoo? Looks like they could use a bit of help.

Popularity: 19%

Having been quiet in the back of the room for the last few months, Alex and I are proud to announce our new project to the world.

Welcome to the world of FeedLounge, a web-based news reader that was designed and built to act like a rich-client, but delivered in a web client package.

But Scott, why did you build it?

I’m so please that you asked
I was continually frustrated by the fact that my favorite news reader, NetNewsWire, only allowed me to read items on one machine. While that is not the fault of a rich client application like NNW, my attempt to read news from many machines led me to frustration. Since I do a lot of roaming work with my laptop, and have a couple of desktops in various locations, I constantly used several machines and news reading applications to read news throughout my work day. The problem with all of this is syncing. NNW does syncing with Bloglines, but not in a good way. Bloglines forces itself to be the master in the relationship, so the sync relationship is not a two way street. Thus, I co-created FeedLounge. While I long for the user experience that a rich client can provide, I believe web applications are the correct delivery mechanism for this type of application. I hope everyone that uses the application find it to be as useful as a rich client application.

Alex led the website development, user experience and UI design, while I focused on the backend application, database, etc. The application has been an awesome experience in building, and we are all excited to share it with you. The FeedLounge team is larger than just the two of us, and they will start speaking up as we go along. I will be covering the development of the backend, etc, since I am a geek like that.

Since Alex and I are both believers of being open and up front, we will both be talking about most aspects of feedlounge on our own blogs, as well as the FeedLounge blog. I can’t wait to ‘brain dump’ all of this info that has been pent up waiting for the Alpha. So pull up a chair, sit down, check out the website, and let us know what you think. Hopefully we can push through the alpha test and into beta to get the application into more hands.

Popularity: 50%

In one of my projects, I ran across this error in my server logs:

Google doesn’t give a whole lot of help on the subject, but this bit seemed to help for my problem: Instead of:

Try this:

Where ‘utf-8′ is actually the encoding listed in the left side of the error message. Your database should already be compiled to support it, or it wouldn’t be spitting this error at you

This was in MySQL 4.1.10, using Python 2.4.1 and MySQLdb 1.2.0, for those wondering.

Popularity: 26%