Could you at least guard against null in your own library code? That’d be great, mkay.

Popularity: 5%

Do we think that our code is just cleaner somehow? That we possibly couldn’t make an error that stupid? Guess what, if you think that, I can almost guarantee that you did

Popularity: 22%

If you happen to be using Twisted, and you are using the LineReceiver protocol as your base protocol, and you keep seeing socket errors, specifically error 32 ‘broken pipe’, it may not be your fault…

LineReceiver has a property MAX_LENGTH, defaulting to 16384 (16K), after which your socket connection gets tossed without error. WTF? At least give me an error that I should deal with!

Obviously, to get around the problem, you just need to redefine the MAX_LENGTH property in your subclass of LineReceiver.

I am torn on this one. I understand the Twisted programmers were trying to defend me from (myself | others), but I was hoping for at least an exception. Is that too much to ask for? That’s about 8 hours of my life I won’t get back.

Popularity: 74%

(Sat on this post for a while, but I still feel the same, even after sleeping on it)

Couldn’t stand my excellently non-responsive wireless ISP, GV.NET. They are so over-subscribed in my area that during the day, may throughput is about 20Kb/sec. I should just dial up at that point (at least my Bluetooth Treo 650 gives me 34Kb/sec)!

Since GV.NET never responds when I call them for outages, or any other reason, I decided to not take the abuse of a monopoly anymore (monopoly to me in the cheap broadband area). Since I cannot get either DSL or cable modem, and satellite is NOT a choice for ssh sessions (with its latency), I ordered a T-1 line, and it was installed last Friday. Although I only get 1.5Mb/s (symetrical, of course), my ping times are 10 times better than the wireless.

Is it expensive? Yes.
What is it worth to me? Approaching a body part.
What would I change if I had it to do over again? Get it as soon as I moved.
Is it worth the cost to not live in traffic hell? More than 10 times!

And in case you missed it, if you are living in the GrassValley area, and have the ability to purchase GV.NET wireless internet service, don’t do it! You have been warned. Until they can actually return phone calls, and tell you their policies up front (nothing like a $250 bill for downloading some ISOs one day), they are not a valid choice.

Popularity: 34%

Just a short piece to say: don’t do it!

I just re-imaged a machine after fighting for 3 days to do it. I know you can upgrade from 3.4 to 4.0 with a bit of a windy road, but it is impossible to go straight from 3.4 to 4.2

I recommend (in valid order for my situation):

1. Start over with a fresh 4.2 image
2. Go from 3.4 to 4.0, then 4.0 to 4.2
3. Stay in the 3 range, until you can do the first recommendation

Just my 2 cents.

Popularity: 40%

Say you are writing some Python code, code that used to talk to MySQL as a backend. Say you are using autocommit, because you believe transactions are for academic weenies.

Now say you want to execute a set of updates to the database. And you don’t know which ones that have been executed before. In Python/MySQL land, you just execute each one in turn with a simple try/catch, so you can continue past failures such as duplicate keys, etc.

Now, suppose that you convert to Postgres as your backend. Now, you catch the exception, and continue on. The very next statement that you execute gives you this:

ProgrammingError: current transaction is aborted, commands ignored until end of transaction block

Now what do you do? Well, Google tells me the problem is not unique to Python, and sent some poor Ruby-on-Rails dev to the madhouse. He sent himself there because this error comes on the SQL statement AFTER the errored statement. I tried just close()ing the cursor, and creating a new one. No dice. Hmm….

Well, the solution for those dying to know is to abort() at the connection level, and then get yourself a new cursor() to continue on. Apparently this is a Postgres thing, not a driver or a Python thing, and I must say that it is brain dead STUPID. There, I said it. Why in the name of all that is good and holy am I required to hand rollback() an autocommit transaction? That has to be one of the stupidest things I have ever heard of.

Do I need some sort of magic ‘autorollback’ transaction? Pixie dust? What? I always assumed (perhaps totally incorrectly, but almost every other database on the planet is on my side here) that autocommit meant each statement was totally separate, and NOT dependent upon the success/failure state of the previous? Am I wrong?

Anyhoo, if you are using Postgres and autocommit, remember to rollback on any exception, as silly as it sounds, it does indeed work. This concludes this evenings rant.

Popularity: 30%

One remark to the crazy types out there that think everything on the internet should be free: It isn’t, and it won’t ever be, so get over it already. Do you honestly thinking hosting and maintaining any web service/application is free? Think again. It just happens to be subsidized in some way as to make you pay somewhere you wouldn’t notice. Think about it for a bit. Every web service out there has to sustain itself somehow. I repeat, think about it.

Popularity: 22%

Note: This post was written a couple of months ago, and never posted, because I always wanted to add more to it. All testing was done while the FeedLounge alpha was on a single server, and FeedLounge has moved on from that. Seeing as I might be switching away from MySQL altogether, I thought I should post this for anyone else running into these types of issues…

I was thinking that the MySQL MyISAM engine’s table level locking was killing the FeedLounge application and user experience in the past week, since the feed update daemon does a large amount of writes to the database almost constantly. I wanted to test the InnoDB storage engine, to see if the row-level locking would offset the fact that I use GUIDs as primary keys, which make primary key inserts not in sort order. So, I started on my journey..

First step, get the data over to the test server. This step was fairly simple and straightforward, as I do a nightly database dump from the alpha server, and rsync it over to the test server, to have a geographically diverse near-online backup. This was just a matter of doing this again to have the latest. Total time: about 25 minutes.

Step two, configure the test server for InnoDB. This was the easiest step, just setting a few variables in the my.cnf file, then restarting mysql. Total time: 10 minutes.

Step three, load the data into the test database. I dropped the existing test database, and then wrote a quick sed script to change the engine type in the massive SQL backup file. I then started the load. A dump takes about 20 minutes now, and a load on the alpha box took about an hour, so I figured it would be in the one to two hour range. Boy was I wrong! The 1.9 GB SQL file ended up taking about 14 hours to load on the test box, but that wasn’t the worst thing.

On the alpha server, using MyISAM, the entire database took about 2.4GB of disk space, including the indices. Once the load was done on the test server, the InnoDB files totaled over 10.6 GB!!! You’ve got that right, the FeedLounge grew to take up almost 5x the space just by changing database storage engines! This obviously throws our disk space calculations out the window. The bin logs on the test box total 1.7GB, so the data seemed to be correct.

So, now I am over an entire day into this testing, but the main question is, can I live with the 5x growth in data size? Doesn’t that seem a little like overkill? Is this stated anywhere? Is this just another one of those ‘known things’ that you are supposed to know as a MySQL DBA?

We are now on InnoDB, and the lockinng/contention issues are gone. The only issue left is the speed of count() queries on InnoDB, which is a known issue, and even documented in Zawodny’s excellent book. I am working on refactoring the code to remove the counting queries, to free up the database to do real work.

Popularity: 71%

After looking at the source for Mochikit, Dojo, Prototype and others, I decided that FeedLounge could use a bit more object-orientation in the client side code.

I created a simple object structure, and wrote a few little test scripts that I ran in the browser. Since I subscribe to 1500+ feeds in FeedLounge, the feed loading/processing on the client side is getting quite slow, so I wanted to make sure that this refactoring was going to help and not hinder client-side performance.

While the test results show what I have built will help performance quite a bit (no more looping over DOM nodes, looking for the right one), the results were a bit surprising…

Given 2 objects, feed and tag, where a tag can contain feeds, I wrote a simple loop to create n feeds, and set an unread count on the feed, then add that feed to a single tag (the worst case in FeedLounge). I wanted to make sure that this solution would scale for the power user, so I started the test at n=5000.

At 5000 feeds to a tag, Firefox was running the code in less than 1 second, which seemed acceptable, so I bumped up n=50000. Firefox choked saying “Out of memory”, so I dropped it in half to n=25000.

The loop at 25000 took 3 seconds of full CPU to run, and the count items return was quite quick. On to test IE, since we support more than just one browser…

In testing IE, I was shocked to find it taking 3 minutes at 100% CPU to run the loop at n=25000. It did eventually finish, but while I can justify 3 seconds on initial home page load to my users, I don’t think they will stand for 3 minutes! I also noticed that memory consumption in IE was over 3 times as much as Firefox. For n=25000, the delta in Firefox had been 50MB. That seems like a lot, but I also didn’t see many people subscribing to 25000 feeds. In IE, the memory usage was 160MB!

Does anyone have any ideas in how to optimize JavaScript for IE? I am doing some extremely simple things here, nothing rocket science at all.

I am not too terribly worried, as the patch for power users of FeedLounge is to switch to Firefox

Popularity: 43%

I have been in the (un)fortunate position of negiotation several contracts lately, and I am amazed at the response I get during the negotiation phase of the contract phase.

It seems that a large portion of people refuse to change the text of a contract in negotiation, in favor of trying to convince you that the intent of the contract is NOT what the contract says. They seem to think that is good enough, and the intent will cover us both. The only problem I have with that is two-fold:

1. Why in the name of all that is good and holy do you have a f**king contract in the first place, if your intention is not to hold to every word. Why can’t we just handshake and call it good, like my dad and his dad used to do. Oh yeah, because people no longer hold to their word.
2. I am signing something that I do not agree to, that can be enforced by law.

When I ask about the first reason, I just get stupid looks, but when I approach the second issue, the answer is ALWAYS that the contract text only matters if things get bad. Guess what, that is the part of the contract that I care the most about!!! When things go south, the contract is the only thing I have to protect me and you, and you are unwilling to allow the protection in a mutual manner when you refuse to change the contract. Give me a freakin’ break.

I have had no less than 5 of such incidents in the last 4 months. I am soo lucky.

I think a good bit of the ACTUAL reason for these refusals to negotiate is the fear of having to go talk to the lawyer again. That is the only reasoning I can come up with that is anywhere near sanity.

Popularity: 37%

I just found out that WordPress happily ‘converts’ some characters in the posts that I type. The problem is that these are not the characters that I typed. When I type an apostrophe ('), WordPress happily converts this character into ′. THIS IS NOT THE CHARACTER THAT I TYPED, JACKASS!!!!

Apparently all of this is done on a per-request basis, in the wptexturize() function. BRILLIANT! Let’s just break xml feed validation on purpose! I love this software… (which is another character that I did not type).

I am being censored in my own blog! Is this a f$$king wiki, or blog software?

Popularity: 21%

Update: In working with Alex on trying to ’sell’ this cool product to him, I told him that I oould fax, and he asked a few questions like ‘Does it pull phone numbers from the address book?’. I told him that I would check, and indeed it does, it pulls EVERY phone number in from the address book, not just fax numbers! That would have been too easy for the user. Also, the Windows user experience is actually worse than the Mac, because you can send the fax at anytime, without being forced to put in a phone number, or worse, if you add the phone number, but do not click add to recepients, the fax is never sent either. Pure Genius!!!!

I just installed my 7310xi on the network (wired, not wireless), and everything seems to be working. I used it standalone as a fax/copier, and that was great. Then I moved on to installing the software on a PC, and that worked fine. I was able to print and fax, no problems.

I then moved onto installing it on my OS X laptop. I like that HP ‘gets’ Rendevous (mDNS), and the device immediately showed up in the list of available printers (everything should work this way, I am surprised that we are this far into the computer age and still not there yet). I then attempted to install the software on the provided CD. The CD had issues reading (it sounded like a DJ scratching a record, then stopping for a second, then doing it again after a spin-up.), but did install the software, at a snail’s pace. Next issue, the installer REQUIRES you to close ALL applications on the Mac. This is just insane! What kind of brain-dead nitwit decided that was a good idea? It even required me to close my Finder replacement Pathfinder. So, after EVERYTHING was closed, installation continued. It finally finished, and then stepped me through some very ugly dialogs that only an engineer would understand.

After all of this, I added the printer, and printed a test page. Great. Now I try to print it to the fax machine that was installed. The only problem there is that it keeps asking me for which modem I should use… I don’t need to use a modem, you piece of crap… Anyhoo, googled for answers, found almost nothing, although the standard ‘re-install the software’ theme keeps pooping up for any problems with HP and OS X related issues. I tried HP online chat support, and after fully describing the problem, they tell me that I can’t be helped, and I need to call the customer support line. WHAT??? How can a company with a decent product have such a terrible end-user experience, and then top it off with useless customer support? I wasn’t trying anything out of the ordinary. It says on the box the unit came in that Windows and Max are supported. I am calling customer support now, and I will update this post with my (mis)adventure.

Update: I call 1-800-HP-INVENT for tech support, go through an automated process that correctly identified my voice request for OfficeJet All-in-One support, then told me to say Macintosh if this is for Macintosh support. I then say ‘Macintosh’, and promptly get the phone answered by the Windows support team. I kindly tell her I am using a Macintosh, and she kindly transfers me, telling me I will hear a few beeps. Minor bump in the road, but so far so good. The phone is answered fairly promptly, and I start talking with Jodi(sp?). He begins asking me the standard info (serial number, OS and verison, machine make and model, network setup, etc. I then tell him that I cannot fax, because the OfficeJet is not in the list of choices for the modem to use. He does a quick search, finds nothing, but thinks that this machine may be one of ‘the ones’ that cannot fax over Wireless. What the? Jodi comes back, and walks me through the non-intuitve way you have to follow to send a fax. The solution is:

1. Attempt to print something.
2. DO NOT choose the Fax… button. That would be sane.
3. Choose the fax machine entry from the list of printers. This will change the Print button to a disabled ‘Send Fax Now’ button.
4. From the options drop down, currently at ‘Copies and Pages’, select ‘Fax Recipients’.
5. Enter the details and choose ‘Add to Recipients’. Voila! The Send Fax Now button is enabled.

I just love to live with counter-intuitive software. There are 2 reasons I switched to the Mac. First, Unix underneath, since I am a command line junkie. Second was my friends’ recommendations that everything ‘just works’ in an intuitive sort of way. HP does not understand this at all. They just figured the easiest way to port the Windows crap to the Mac platform, including all the memory hogging monitoring apps that are the HP signature.

The only upside is that Jodi, the last person I talked to, was intelligent, articulate, and patient. He hepled me through the issue, and even offered to call me back, and then actually did call me back! Thanks Jodi.

Popularity: 21%