Today is my two-year anniversary of daily blogging. I started blogging without a break on August 2, 2003. 731 days. Whew.

Of course, I have cheated a bit some times. Blogging a day ahead or behind when things like travel and international date lines got in my way. Blogging past midnight some nights and counting that as the “next day,” then being able to slide until just before midnight two days later. But I’ve got some 1200 or so posts in those 731 days, so it’s not like I haven’t made up for it and then some. And I did do quite a bit of blogging before then, I just missed a day every week or so. But 8/2/03 was when I decided to make it a habit.

I gotta get a life!

Well, the blog comment and trackback spam have held up well. I wish I could say as much for the referral spam blocking.

For most of the month it did a pretty good job of blocking spam. Some spammers blew through initially with a hundred or two spams before I could add their domains to the blacklist, which I knew would happen. After being added to the blacklist, some still got through, but were restricted to just a few spams per day. By the 20th of July, the top 25 referrers were still mostly legit. But then things fell apart.

The culprit was a German porn spammer, who came in with three different domains (tzz.de, axxs.de, and cxa.de). I added the domains to the blacklist, but they continued to blow through with new subdomains until 14 of the top 25 spots on the referrer list were jammed with their crap. Although they did not post the thousands of spam that they could have, they mucked up my referral list enough to make it butt-ugly again. At least I have the satisfaction of never having given these scumwads even the tiniest bit of benefit for all their troubles. They’ve made things miserable in terms of my reading my stats, but they have not gotten an ounce of Google Juice nor a single referral from me and they never, never will.

In the end, the 18 spammers in the top 25 left behind 2427 referral spams. Take away the German porn spammer and that number drops to 4 spammers leaving 492 spams.

So it seems that the AwStats referral patch, while useful to some degree, is far from perfect. The question now is, will more spammers follow in blowing through it, or were the last few weeks a fluke?

As for comment and trackback spam, I have found an interesting trend: it’s dying down. Where it used to be flooding in at the rate of thousands per day, it has subsided to a trickle of only a few hundred a day (most of them coming from a porn web host). Why they’ve so dramatically dropped off, I don’t know for sure; maybe they detect the new version of Movable Type, figure it’s no longer worth trying, and move along? Beats me. But I like it.

Am I the only one who absolutely hates web pages with black backgrounds and white or bright-colored text? Perhaps my eyes are really sensitive, but whenever I try to read text on such pages I quickly develop after-images from the contrast that give me a headache. And yet such pages seem to proliferate, increasing rather than decreasing in number.

Maybe it’s that the people who publish pages with such color schemes don’t read their own pages in finished form…

By the way, one other thing I found in looking at the stats of my blog site is that a lot of people are hotlinking to images from my site. “Hotlinking” refers to the practice of “borrowing” images on other people’s web site to adorn your own. This is not about downloading an image from somewhere else and uploading it to your own site. Instead, it is the practice of leaving the image on the other person’s web site, and by using simple HTML code, making the photo appear to be located on your own site–but really it’s located on the original site, and every time someone visits your site, their browser takes a covert side trip to the site with the photo, snags the image, and plops it into the hotlinker’s page, as if it were really located there.

Probably the hotlinkers do a Google image search, find a photo from my site there without ever visiting it, and then use the URL of the image where they like. In doing this, anyone can display a photo on their web page, even though the photo is hosted by someone else. But the Internet is not really free, as much as it may seem like it sometimes. To host an image–that is, to keep an image file on a web server hard disk–costs money. Similarly, to transmit data, like the image file, is called “bandwidth,” and also costs money. In other words, they’re paying the rent and transportation, which you’re hijacking to your benefit. While it may not be fully honest to grab someone else’s image and put it up on your site, at least you’re paying the rent and bus fare. Hotlinkers steal everything.

Some people hotlink to images because they feel they have no choice, like Blogspot bloggers who can’t upload images–or that is, they couldn’t before, but now they can but haven’t learned that yet. Others maybe can’t upload images from whatever service they’re using.

Either way, it comes down to this: if you use someone else’s URL in your image tag so they host the picture but it shows up on your site, then you’re stealing bandwidth. It means that in order for your page to look good, I have to pay my web host x number of dollars for hosting the image and using bandwidth so people who visit your site can see it. It’s like my using your phone line to make all the calls I like. But people do it casually probably because they don’t understand what they’re doing–or perhaps they do understand but just don’t care.

I tried to engage something called “hotlink prevention,” which would prevent such bandwidth theft, but as a result, many people with some browsers were not able to see the images I put up on my own blog, so that had to go. What I am left with is detecting which photos have been hotlinked, renaming each photo so affected, and then re-editing my own blog post which used the image so that the new name is recognized; that severs the link to the hijacker’s site while keeping the photo visible on my own site. Sometimes, if I am sufficiently annoyed by someone costing me a lot of bandwidth, I’ll change the name of the original and reflect that in my blog, but then I will also place a new image with the original file name–with a replacement image carrying a sharp graphic message telling the hotlinker to stop stealing bandwidth. They probably didn’t figure that (a) I would be able to see that they stole the image, or (b) I could change the image and in so doing, change their site.

Recently, one image in particular has been hotlinked a lot, one I grabbed from a Simpsons episode. At least three different forum/member sites had this image hotlinked; one of those sites had the image in the main index, meaning everyone who visited would activate the link, and a different web site had several members who each had the image in their profile pages, as if one had used it and the others picked it up from them. The image was getting hotlinked more than 100 times a day because of this; at 20K a pop, that’s 60 MB every month, just for that one photo. No thanks.

For those of you who have been following the perils of this blog lately, you know that the latest implementations of perl/mysql/cpanel have knocked out some of the Movable Type scripting that makes comment and entry posting-and-rebuilding work. The dreaded “500 – Internal Server Error” messages have plagued both commenters and myself alike. It’s all very technical, but let me see if I can lay it out so that maybe you can clear it up if you’re having these problems yourself.

First of all, the errors are not fatal. They only occur on a save-and-rebuild with making new blog entries and with making new comments. The entries and comments are saved, but the pages are not rebuilt. That can be solved by a manual rebuild. Everything gets saved and eventually appears on the site, it’s just that the rebuilding must be done manually.

The fix: in the perl scripts, downgrade the DBI to 1.47, and downgrade the DBD::Mysql to 2.9006. If you don’t run your own server, then ask your web host to do it for you. However, you must also tell them to reset the cpanel autoupdate for your site so that it does not automatically reload the upgrades every night. My web host did this once, and it worked for 12 hours or so, then reverted. I told them of this, and again, they fixed it. They did not tell me if it involved further settings changes, but it was likely something in the cpanel that continued the autoupdate. So stress to your web host that the cpanel autoupdate must be verified as turned off, and then wait 24 hours to make sure that took hold. I’m still not clear as to whether the problems will return yet again, but it is clear that they are fiddling with the right toggle which gets rid of the bugs. I’ll let you know when the problems have stayed away for more than a day.

This should end your MT blog problems for the time being. Word has it that the developer of DBI::Mysql has versions 3.0000_0 and 3.0001_0 out by now, but whether that will fix the Sql/MT bug or not, I cannot say. More news as it comes in.

Update: for four days in a row, my web host tried to disable the automatic upgrading of the site, and on each day the fix worked–for about half a day. The auto-update kept kicking in and the errors would return. Well, maybe five times lucky: my host has disabled the auto-update completely now, so let’s see how that plays out.

This site is always deluged with spam, but every once in a while, some very subtly weird spam comes along that just catches your attention. The most recent has been a series of trackback spam from fake blogs. As I’ve mentioned, trackbacks are when other blogs refer to and link to this one, and their site software “pings” mine, or sends a special message to mine alerting me to the reference. Spammers fake such pings in order to get a link on my site to theirs.

But these latest fake pings don’t make much sense. There has been almost one a day for the past several days; all are fake blog posts on fake blog sites; all have an address which would indicate a recent blog post. The address includes “/archives/2005/07/” and then a blog article file name–all looking rather authentic. In each case, the message contained in the trackback ping is very similar: “Excerpt: You can read about it also on my blog page,” “Excerpt: More info can be found on my page,” and “Excerpt: Look at my page for more info….”

Now, I would expect in each case for the bogus blog addresses to be a shell domain with a redirect: go to the address given, and you’re automatically redirected to a different domain with hideous porn spam. Instead, when I visited each site to find out what was going on, a 500 Internal Server Error was returned. For the entire site, not just for the one page. And it’s not a 404 error or a domain not found/doesn’t exist error either.

There is one possible explanation: by ironic chance, these spammers got foiled by the exact same perl scripting bug that’s been plaguing this blog for the past several days. In short, the redirect software which they use to transfer you to the hideous porn site is incompatible with the new perl software, causing the error messages. Which would be both funny and all too fitting.

It’s also possible that these spam are fake fakes–spam which doesn’t connect to a spammer. I’ve gotten these before. In the past, spammers have sent me referrer spam pointing back to the International Atomic Energy Agency, and spammers sent a lot of people referrer spam that claimed to come from the John Kerry web site. My guess is that these are either hoaxers, or they’re spammers testing out a software configuration and not leaving their own address in case something goes wrong. Even fake blogs have been the seeming origin of spam before–although in the past, these fake blogs showed a specific address ending in “/archives.html”, which is not a real location in any blog that I know of.

Nevertheless, it does make it all a bit interesting. For me, the main thing is, none of this gets onto my blog anymore, so it’s just a nuisance or a curiosity. Even my anti-referral patch for AwStats is working well–I’ve been getting almost no referral spam this month so far. A few are leaking through, but can only manage to get one or two hits on my site per day, like a few small leaks in a dam holding back a deluge.

Everyone, please be patient while Movable Type gets its stuff together. Recently, upgrades to perl and mysql have been upgraded on web hosts all around, and it appears that the new versions may not be compatible with Movable Type, this blog’s operating software. This means that certain events on this blog will not function properly, such as posting to comments. If you try, you may get a “500 – Internal Server Error” message. However, your comment will have been saved–it’s just that the Movable Type script that’s now incompatible with perl can’t go on to rebuild the post or give you any other message. It is also possible that this is worse with TypeKey than with unregistered commenters, at least for the time being. The malfunction also makes it harder–though not impossible, as you can tell–to post new entries for the blog.

I am trying to see if I can get my web host to downgrade the elements that are causing the problem, so the site can go back to the way it was before–but I have the feeling that the elements are server-wide and so cannot be changed just for me. Otherwise, I’ll have to just wait for Six Apart to fix the problem on their end and issue a patch or an upgrade (3.2 is due out soon).

So please be patient, and I’ll get things back to running smoothly again as soon as I can.

Update (July 4): my web host has downgraded at least part of the new software that might be causing the problems–but I’m still experiencing errors. There might be fewer errors than before, it’s hard to say. Still looking for a way out of this mess.

Today is the 700th day in a row that I’ve blogged, no breaks. August 2nd will be the 2-year anniversary for nonstop blogging.

On a side note, the iTunes Music Store has finally recognized the audio blog. Do a search in Podcasts in iTunes for “Blogd” and you’ll find it. As I mentioned before, it’s just a dorky little trial recording, so don’t expect to be blown away.

Now to tune up the RSS feed to get the details straight (and by this I’ll find out how fast the iTMS updates its listings via the RSS feed), and then record a real podcast. Or at least try to.

Well, it’s the end of the first day of the new month, meaning I’ve got a new start on the stats. Now I can see how the referral spam is working. It is supposed to use the MT-Blacklist to filter out spam as it arrives (there’s no getting rid of it once it’s in, not yet), but there’s a snag: I just upgraded to MT-Blacklist 2, which does not publish a blacklist. And there was no way to enter new strings into the existing blacklist file. Fortunately, someone posted a script which would allow MT-Blacklist 2 to export a blacklist. However, there’s a snag.

MT-Blacklist 2 is supposed to import your old blacklist and incorporate it into its own, but mine clearly did not do that; there are 400 or 500 entries fewer in the new list than in the old one, obviously all of my custom entries. And since the referral spam is a mix of old and new spammers, I need all the data from both lists. And MT-Blacklist’s import function is badly broken (don’t start it, it will never end). Worse, the new MT-B seems to lack the ability to add new strings more than one at a time–which means that re-entering the strings from the old list would be a very laborious task.

Feeling that most of the spam I’ll get will be from new addresses, I decided to archive the old list and use a newly-generated list. And so far, the results are a mix of good and bad.

The good: it does seem that the referral spam is being blocked. After one full day, only 8 of the top 25 referrers are spammers, and most of them are brand-new ones that slipped in only a few spam before I caught them. So the list looks like it may be a lot cleaner than before. Additionally, the total tallies from last month’s referrals was encouraging: 17 of the top 25 were spammers, and the spammers in the top 25 unleashed a total of 1,637 spam. That is opposed to May’s figures, where fully 24 of the top 25 were spammers, who unleashed 10,000 spam hits. On top of that, June’s figures include unblocked spam from the first five days, 867 spams from that time span alone. Meaning that there was less referral spam in the last 25 days than there was in the first 5. So clearly the AwStats patch is doing its job.

The bad: the patch is not doing a perfect job. One gay porn spammer, for instance, has been leaving one spam hit regularly every four hours or so–and they’re somehow getting past the filter, despite being on the blacklist. Which means that there is a way around it. And if there’s a way around it, then eventually it is likely that all spammers will find their way around it. This is the part I was talking about earlier when I said that it was a constant battle of putting up defenses and the spammers working to get around those defenses.

So we’ll have to wait and see how long this solution works. At least I’ve got pretty airtight control over the comment and trackback spam. (Knock on wood)

Hey, a milestone passed and I didn’t even notice. My 1300th post, four posts back (Blog Spam Solution). Yes, I know it’s numbered 1326, but there were 26 posts that were tests over the past 2-years-plus and got erased, but the numbers weren’t taken back.

Okay, 1300 is not a special number. But it’s round. And another one to complement it: 700, the number of days I’ll have blogged consecutively, one week from today.

I should provide a wrap-up on how the blog spam is being handled, in case this could help others with an onerous amount of spam on their sites. I seem to have found a solution that allows for most spam to be blocked or moderated. So far, despite thousands of attempts, not one spam comment or trackback has gotten through the defenses on this site since they were erected. It’s still early in the game, but I have a feeling that this solution will work. I should note that this applies only to people using Movable Type to administer their blogs.

First, a word about the different kinds of spam: first, there is blog comment spam. This is usually generated by marketers who have automated scripts that search out the comment-generating scripts used by blogs, and then use them to generate hundreds or even thousands of spam-laden comments on the blog. The spam is in the form of links either in the comment body or in the URL link allowed under the visitor’s name at the bottom of the comment. The links are not just to bring people from the blog to their advertising site, but just as much if not more to generate “Google Juice,” as more external links to their site will boost its all-important ranking in search engine result lists.

Second, there is TrackBack spam. A “TrackBack” is when someone refers to someone else’s blog post in a posting on their own site; their blog software will send a ‘ping’ to the other person’s site, informing them that their post was referenced in another blog. Often blog software will create lists of TrackBacks with a link back to the source. Spammers, again trying to get Google Juice wherever they can, send out fake pings so they can get free links again.

And finally, there is referral spam (also called “referrer spam,” often spelled “referer spam”). People who keep blogs like to know what attention they’re getting. Their web hosts usually provide statistical data, showing how many people visit their site, which pages they go to, and which sites are sending people to theirs via links. For example, if macsurfer.com creates a link on their page to one of my blog posts, people will follow that link to my site. Their arrival brings with them a string a data about them, including which site they came from via a link. This is a referral, which is counted by the stats on my site. I can see how many people followed the link from MacSurfer. Some people, happy to receive links, will obligingly set up top-referrer lists on their web pages, with links back to the nice people who sent visitors their way, as a way of saying ‘thanks.’ Spammers again take advantage of this by generating bogus automated ‘visits’ that carry falsified link data, making it look like hundreds of visitors came to the blog from the spam site, when in reality, no link and no visitors actually exist. But this faked referral spam will get them on the referrer’s list, and give them even more Google Juice.

Referral spam is loathed by site owners because it goes a long way to making site statistics useless. After all, if you want to know who is really visiting your site and you see thousands of hits from texas-holdem, porn, pharmaceutical and other commercial sites, you know the stats aren’t telling you the true story. With all that spam, it’s next to impossible to know exactly how much traffic is legitimate. Fortunately, a lot of referral spam originates from the same IP Address, and so even hundreds of spams will likely count as only a few “visitors” in the main part of your stats, making them a bit more reliable, though still altered and uncertain. But the list of referrers, the list of sites with links to yours, is utterly trashed.

Often times these tricks will not even get the spammers what they want; bloggers have more or less stopped posting top-referrer and trackback listings. But the ability to spam in massive quantities is so cheap that spammers don’t care if the return is minimal; it’s still worth their few bucks because it will likely get them enough to pay for the spamming and then some. And for the spammers, turning up the volume doesn’t cost much more, if anything at all, and so the volume of spam being sent is increasing more and more. As bloggers try to block the spam, spammers think of new ways around the defenses.

The current programs to block and filter spam usually work from blacklists. However, the blacklists only know what you’ve banned in the past, they can’t know what a new spammer looks like until you tell them unless the spammer is really obnoxious. So the new spammer, or the ever-changing new face of the spammer, is allowed through the defenses at first before the blog admin catches him and puts him on the blacklist. Once on the blacklist, future attempts to spam are automatically blocked. There are a few tricks that the new apps use to detect spam, and so the new solutions mean that very little, if indeed any, spam gets through to appear on the blog at all, even for a short time.

So, what are the solutions?

First: MT-Blacklist. This is a long-standing spam filter that I’ve used for more than a year now. MT-Blacklist maintains a list of text strings (either words, strings with wild cards, or URLs) that the user has designated as spamworthy. Every new comment and trackback ping that comes in to the blog gets checked against this list, and is blocked if a match is found. Furthermore–and this is what wowed me the first time I used it–it has the ability to “de-spam,” which means that even if new spam has gotten past the filters, if you add a text string from the offending spam messages, MT-Blacklist could access all the archived comments and purge all the spam from there as well. These two combined features–blocking and de-spamming–made controlling spam completely possible.

Unfortunately, the de-spamming feature is missing in the new version. Movable Type itself has a comment filter in version 3, which allows you to single out comments by a common email, name, or IP address, after which you can auto-check all of them and de-spam. This will be useful, but only if the email, name or IP in a huge chunk of first-time spam that gets through is identical in multiple comment spams. However, that is all too often not the case. If Movable Type could add a filter for strings inside the comment text itself, that would plug that hole and bring back full de-spamming power.

However, as the author of MT-Blacklist pointed out here himself in a recent comment, MT-Blacklist, especially when used in conjunction with SpamLookup, blocks so much of the spam that de-spamming may not ever be a problem task. Still, it would be nice to have around just in case.

The new MT-B has some handy new features which makes catching that spam possible. One of them is the ability to filter comments with too many URLs; a common practice among spammers is to leave a comment with dozens of links. Since no real commenters do this, it’s an excellent way to filter out the spammers. MT-B will also now block comments or trackbacks which are duplicates of prior entries, another good filter against spam. MT-B will also automatically update the blacklist from the master blacklist for the app, meaning you get protection even before the spam reaches your site.

The second element to block spam, as I mentioned above, is SpamLookup. A new MT plug-in, it does something similar to MT-Blacklist, using IP addresses of spammers from a master list to help block spam. This method is called DNSBL. This plug-in also uses a good number of other defenses, including multiple-URL blocks, domain blacklist checks, and your own blacklist of text strings to force-moderate or block a comment or trackback ping, similar to MT-Blacklist.

I have both MT-Blacklist and SpamLookup installed. I am not sure which one filters before the other, but I suspect MT-Blacklist does. It catches pretty much all the comment spam. SpamLookup then gets all the trackback spam. Together, they make it extremely unlikely that much spam will get through. And what does get through–maybe one out of every thousand or so spams–gets moderated, where I can give it the boot manually, adding it to the blacklist–which brings us to:

Third, the blog software itself, Movable Type 3.1. The upgrade from version 2 to 3 brought with it very important blog spam safeguards: moderated posts and comment registration. Before, even with MT-Blacklist doing the efficient job it did, the first-time spams I mentioned before would get through, and sometimes stay up on my site for hours before I could spot them and take them down, especially if they hit during the night time here in Japan. Post moderation means that no spam gets through to the blog at all; I might have to moderate it if it doesn’t get filtered, but at least it never gets posted publicly, which is a big thing if you want to de-incentivize spammers. In addition, the Typekey registration allows commenters to post comments without having to wait for moderation, putting the onus of spam control partially on that side of the equation, and allowing legitimate commenters to bypass the moderation in a way spammers can’t.

So these tools combined make it possible to block pretty much all of the comment and trackback spam. What remains is the referral spam. As yet, there is no fix-all for this problem, but it appears that a patch for the statistics program “AwStats” (my favorite) is doing a fairly good job at keeping most of the junk out. This patch will not de-spam your stats, but it will block any incoming referrer spam by using the MT-B blacklist file as a filter. The spammers may hit your site, but the hits don’t get counted in AwStats.

After having used it for a few weeks now, it seems to be doing its job. In May, before I got the patch, the top 25 spammers left more than 10,000 referral spams in my logs. In the first five days of June, the top 26 referrers were all spammers, and had hit me a total of 867 times between those top spammers, averaging out to about 175 new top-26 spams each day. Since the patch was installed, only 21 of the top 26 are spammers (5 legitimate referrers got back on the list!), and those spammers have a total of only 1434 hits–a total of 567 new hits over the course of 16 days, or just 35 new top-26 spams each day, meaning the major referral spam has been cut by about 80%. Some referral spam is getting through, but mostly it’s just the first-time stuff, before it gets added to the blacklist. Next month, with the patch in place from the beginning, I will be interested to see how little spam gets through at all.

The down side to the AwStats patch (aside from the fact that you need AwStats in the first place) is that it has very little documentation. Because of this, it was unclear to me how the patch would be applied. AwStats itself is a server-level app, meaning that I can’t patch the Perl script that makes AwStats run. However, it seems that only the “.conf” file for AwStats needs to be changed. Again, the project site is unclear, but seems to indicate that a tool called “GNU Patch” is needed to install it. How that works, I have no idea. All I know is that the patch must be worked into the AwStats .conf file, and that the pathway to the MT-B blacklist.txt file must also be added… somewhere.

Fortunately, my web host (might as well give Surpass Hosting a plug, seeing as how they did me a favor) support technician helpfully volunteered to install the patch, and apparently did the job right, as it seems to be working. Unless you know how to use the GNU patch tool and stuff like that, you might want to ask your web host for the same favor. The file that needs to be patched is titled “awstats.domain.com.conf” (replace “domain.com” with your domain name and extension), located in “tmp/awstats” on your home directory. The pathway to the MT-Blacklist file is probably “/home/youraccountname/public_html/blacklist.txt”, though you should check that first. MT-B v. 1.6 allowed you to see the pathway, but I can’t seem to find the pathway listing in version 2.

So, to sum up: using Movable Type 3.1, MT-Blacklist 2, SpamLookup, and a patch for AwStats, I’ve been able to solve–finally!–most of my spam headaches.

Now to wonder about what the spamming scumbags will come up with next…

It seems as though I reinforced my spam barriers just in time. Although I wouldn’t notice it because my filters keep all but a few bits of debris out of my sight, I can look at the logs and peek at what’s going on behind the dam I’ve erected. And it’s horrific. The spammers are now unleashing huge floods of spam upon this blog, and probably to the blogging world in general. In just over the past one day, more than 2000 attempted spam comments were shot in my direction. And trackback spam, once just a sometimes thing, has similarly exploded: more than 200 in the last day. Incredible. God help the poor wretch with a blog who doesn’t know how to install a spam filter. Think about it: in one month, you could find your blog choked with 60,000 comment spams and 6000 trackback spams. Good lord.

And now it’s no longer the porn or pharmaceutical industries that are leading the charge–the overwhelming offenders are the gambling sites, both poker and casinos. They’re making the porn and pill people look like jaywalkers.

Anyone out there with an independently-run blog (not Blogspot) with comments activated? How are you holding up?

Okay, I’ve fixed the error in the comments that kept giving people a “comment-pending template missing” message. It should now give you a message telling you that your comment is waiting for moderation. The comment system should be working fine now.

The new blog setup is also working fine. MT-Blacklist is blocking most of the comment spam coming in (about 500 spams daily!) and SpamLookup is blocking the Trackback spam (another couple hundred every day), while the blacklist patch to AwStats actually seems to be successful in blocking most of the referral spam, though it doesn’t stop the initial flood from beginning before I can add the keywords to the spam blacklist. So my spam moderation has dropped to just a few per day, and my statistics, while still flooded with spam, aren’t nearly as choked as they used to be. That’s livable. I shudder to think what it would be like if none of these spam filters were available–comments would have to be shut down completely.

Still, comment moderation, while it works and keeps spam out, is less than perfect due to the wait involved for the comment to appear. Using Typekey would solve that problem, but many don’t want to suffer the hassle needed to set up the account, which I can fully understand. So, if you post here even semi-frequently, feel free to ask me to set up a Typekey account for you–just leave a comment asking that. I’ll hand it over to you via email (I’ll give you an address to contact me at if I don’t know yours), giving you the username and password. Then you just click the “sign in” link next time you leave a comment, insert the username/password, have your browser remember the info, and from then on in it’s automatic. That should make commenting easier than before, and the same Typekey account can be used to comment in any other blog using Movable Type 3.1.

Any comments, suggestions or observations about the site?

What is it with non-commercial programmers/scripters and documentation? This is something I have noticed for years: people who write small scripts or programs that are not self-executing rarely write more than a few lines of vague instructions on how to use them, even if that much. These people apparently feel that telling other people how to use their creations is unnecessary. “Here’s a great script,” they tell you, “which will do all these great things.” But they don’t tell you where to put it, how to modify it, or anything else that’s necessary for using it. Sometimes these people don’t even make clear what their script even does beyond general descriptions.

Are they just yanking our chains, or do people who acquire the skill to script suddenly lose sight of the fact that most people don’t know how to script or use a command line interface, and can’t figure out what the hell they’re supposed to do with these cryptic files? Better not to publish on the Internet claiming you’ve just created something useful, for all the headaches you cause so many people trying to follow up on your promise of a cool product with needed features. I’m not trying to be ungrateful, these people put a lot of work into some good ideas to make a useful product. But if you spend that much time making the script and then bail on the documentation, then what’s the point?

And I’m not talking about full tech support here. I’m talking about sitting down for two friggin’ minutes and writing down the most basic of instructions. If the file is self-executing, and simply works when you double-click it, then fine, minimal documentation is OK (ironically, those programs usually have good documentation), but the stuff that needs to be changed and placed just so–these are the ones the developers seem to completely abandon once they finish coding.

Were this only a sometimes thing, it wouldn’t be so bad–but it seems that it’s the case nine times out of ten. Is it just me, or what?

Are you using Safari? If so, look at the search box on this page. Notice anything about it? Rounded corners, gray placeholder text, the circle-x ‘clear’ button once text has been inserted?

It turns out that Apple created a few new tag features just for Safari, intended for Dashboard widgets, which are based in Safari. The ones I’m using are for the <input> command, namely type=”search” and placeholder=”Go fish!”. The type=”search” attribute is the key one, though–that’s the one that rounds the corner and gives you the circle-x button. And it is invisible to other browsers, so they still see the regular search box–which means 91% of my visitors. So no loss, but a little cool tweak gained with the extra code. Heard about it on Ars Technica (though they don’t seem to like it), and more in detail from Bartelme Design. The “placeholder” attribute, by the way, adds the gray text inside the search box which disappears when you click to type.

So far I’ve applied it to the main page, the individual archive pages, and the search page. What I haven’t figured out yet is how to make the little magnifying-glass icon appear on the left so you can review recent searches…

Update: a helpful member at Ars Technica filled out the answer: in order to get the “recent searches” menu, you have to add both “autosave” and “results” attributes. For example: <input type=”search” autosave=”anyname” results=”5″ placeholder=”Search this site…”> and that would give you all you need for the full search box in Safari, recent-search magnifying glass icon and all. Cool! Now, everybody just switch to Safari, now that you have a reason.

Okay, I’ve figured out the comment system and have revamped things a bit with this whole Movable Type 3 upgrade.

I’ll continue with comment moderation, but there’s an easy way around it: register with TypeKey. It’s explained on this page, but essentially it’s a free registration system which will allow you to place comments here and any other Weblog using Movable Type 3 without moderation, meaning your comments appear immediately.

If you log into TypeKey and allow the cookie to stay, it works seamlessly. If you log out of TypeKey or if the cookie expires, then it’s a click-new page-click-and-back exercise, and probably takes less time than typing in your name and email anyway.

Without registration, comments are of course still open and I’ll usually be quick about approving them (my computer is usually on and the comments are emailed to me, causing Eudora to beep at me). Except, of course, when I’m asleep….

As a result of all this, the site should (knock on wood) be completely free of spam.

Additionally, the comments are now back to their normal appearance, and the search template has been rewritten so as not to look completely spazzy.

One note about the upgrade: MT-Blacklist apparently becomes far less functional, which is a pain in the neck. Under MT v. 2, MT-Blacklist will not only block new spam in comments and trackback which match blacklist criteria, but can also “de-spam,” which means that if a spammer got 500 spams into your blog before you could add him to the blacklist, the de-spamming ability would allow you to flush all that out of your blog with a few clicks of the mouse.

For some unexplained reason, MT-Blacklist 2 lacks the de-spamming ability, which is a major annoyance to me. It essentially makes the program half as strong as it used to be and makes it far less appealing. While MT v. 3 does allow for possible de-spamming of moderated comments and trackbacks in the control panel, it doesn’t work as easily as MT-Blacklist used to. I would have thought that an upgrade would enhance or add features, not chop them in half.

However, there might be a new plug-in that works as well: SpamLookup. I’ll let you know how it works. If it does, then maybe I’ll be freed from constant spam moderation. One can only hope.

Okay, the site itself has been upgraded to Movable Type 3.17. However, I’m still trying to get used to all the new stuff, so please be patient in the meantime. One point is that the comments are now being moderated, which means that when you leave a comment, it will not be posted immediately. Instead, it will be sent to me for approval, and will appear when I get to it. I’ll try to make that as soon as possible, but it might take several hours (especially if I’m sleeping!). Anyone out there know, for example, how to create registered commenters? I’ll learn how, but I gotta do some reading and experimenting, and I did sort of start this project late at night on the eve of a workday. So there’ll be a bit of a lag before stuff gets smoothed out.

Hopefully, the upgrade (along with MT-Blacklist 2 once I get around to installing it) will come along quickly. Anyone out there with experience with version three and you know some possible kinks in the system and things to look out for, let me know!

I’m finally getting off my duff and upgrading to Movable Type 3, but in the process there may be a few quirks. If the blog does not display or accept actions as it usually does, please be patient.

Thanks!

For a few months now, I have been somewhat mystified about how some particular people have been finding my blog via search engine. My blog’s search engine stats reveal how many people find my blog through Google, Yahoo and others, and they reveal what search terms led to my site. And far too many have been of a rather pornographic nature.

They include “seven of nine nude,” “nude harry potter,” “nude widget,” “busty american soldier,” “katherine harris nude” (yecchh), “nude taiko drumming,” “seven of nine sex porn nude,” “harry potter porn,” “kung fu porn,” “porn widgets,” “surprise porn,” “nasal porn” (wha??), “walter cronkite nude photo” (almost but not quite as much ‘yecchh’ as for Katherine Harris), “porn with popcorn,” “hurricane george and porn,” “japanese rice patties sex slavery,” and “sex on the 26th floor of goldman sachs.”.

So I wondered how on earth the search engines sent people my way for stuff like that? It bothered me for a while, until I figured out what was happening. At the bottom of each of my blog’s individual entries (where one blog entry and all comments are listed on a single page), I explain about the blog’s anti-spam software and how it blocks certain keywords–and then I list some of the words, which include “porn” and “nude.”

Meaning, of course, that these words now appear on the same page as every single blog entry that I have made, about 1300 of them, on a huge variety of subjects. Aha.

Though now I am, admittedly, a little bit curious as to what exactly is happening on the twenty-sixth floor of the Goldman Sachs building.

Last month marked the most excessive period of referral spam I’ve experienced so far for this site. (See here for an explanation of referrer spam.) The problem itself started a year and a half ago, in November 2003, with a grand total of 7 “hits” from a Paris Hilton site. It was #36 on my referrer list, with the referrers before it all legitimate, adding up to an aggregate 1167 hits from 35 referrers before the one spam showed up.

I didn’t even notice it.

December’s experience was almost identical: the Hilton spammer was #35, with another 7 hits, 1153 other hits coming from the prior, all-legit referrers.

But then Hilton advanced: in January 2004, one Hilton referral was at #18 with 13 hits, while another Hilton spammer was at #18 with 8 hits (interestingly, the RIAA seemed to spam my site just in front of the first Hilton site, coming in at #17 with 14 hits–certainly faked, I doubt the RIAA linked to me from their main page).

This went on for a few months, but the spammers stayed out of the top 10–until April 2004. Suddenly, the top three referrers were spammers with a total of 333 hits, with others beginning to litter the spots below. For a few months they died down, scattering several dozen hits in the lower rankings, until in July, someone mysteriously spammed me 1556 times. It was mysterious because the spammer made it look like the links were coming from the IAEA, and I know the International Atomic Energy Agency was not linking to me from their home page. The only thing I can figure is that it was a spammer testing out a new script before using it to get referrers to point to his site. I am assuming it was Burnham, but I may be wrong–there is a lot of scum out there.

Spam referrals continued to persist in the lower rankings, sometimes hitting as high as the #4 spot, but outrageous spam held off–until October, when the infamous Burnham Internet Sales Cockroaches hit full-force, dominating not only the top 5 spots, but also 25 of the top 27 (and probably more that I couldn’t pin down as Burnham), their top 25 spam referrals totaling a record 3758 hits. This prompted me to blog about them, reporting on their scumminess, and I am happy to report that Google still ranks that post of mine as #1 in a search for “Burnham Internet Sales,” with their own site ranking #2. Score one for the little guy! Try to spam me to get high Google rankings at my expense, you do so at your own risk!

Although Burnham stopped harassing me, other spammers started glomming on, and since late last year my stats have been clogged with the lowlifes. I tried everything I could to get around them. I tried IP Deny in my site’s control panel, but it didn’t work. I tried to create an .htaccess filter myself, and it worked, but only for about 24 hours, after which the spammers blew through it like tissue paper. I wasn’t publishing a “top referrer” list which the spammers hope they will appear on, nor was my stats page open to public view (it’s in a password-protected directory), so I wasn’t encouraging them in any way. But this is a blog, and blogs are spam magnets. Spammers care little if one site which they’re decimating isn’t getting them any profit, it costs them next to nothing to do, and they probably enjoy defecating over sites that deny them a return. So I got more and more spam, and there was no way to block it.

Here’s where I get to the “Hopeful, but Wary” part. Last month, as I mentioned at the top, was the most excessive month of referrer spam. 24 of the top 25 referrers were spammers, accounting for more than 10,000 fake hits. So I looked into it again, and found a possible solution. There is an AwStats (the program used to get statistics) patch on SourceForge which claims to “check comment spam blacklist for referer spam.” The information on the patch is incredibly sketchy; I thank the author for making it, but I would think they could spend more than 50 seconds to type the instructions. There is no explanation beyond the title as to what it even does. But someone on a forum claimed it “scrubbed” the referrer logs of all spam, which sounded great to me.

What was clear was that the patch used the same blacklist that MT-Blacklist used to delete comment and trackback spam. All I would have to do was add the referral spammers to the blacklist (easy enough to do with MT-Blacklist’s interface–and many of the spammers were already there).

It turns out that I had even seen this before and blogged on it, but had not used it because it required admin skills I frankly do not have. As I said, the instructions were sketchy at best, and I could not come close to deciphering them. But after May’s spam numbers drove me to search some more, I found someone who suggested the web host’s tech people install it instead of the site owner. Worth a shot, I thought, so I asked–and they did it.

At first, I was disappointed; I expected the existing spam to melt away, leaving nice, clean stats behind, but alas that was not to be. Before the patch was installed, the top 26 spots were filled by spammers who had hit me a total of 867 times, and that was just after 5 days. After the patch was installed, the spam was still there. Failure again, I thought. But half a day later I noticed something: the spammers I had added to the blacklist had not hit me in that time. Other spammers, not on my blacklist, had hit me again. So I added more spammers from the referral logs to the blacklist, and they stopped adding hits to the list as well.

If my understanding now is correct, the patch does not clean up old spam, but it stops new spam that you’ve blacklisted from getting in. And that will do quite nicely in the interim. I’ll just have to keep an eye on the stats (which I usually do anyway) and blacklist spammers as soon as they hit me. With luck, the flood will return to the relative trickle that I was experiencing a year ago.

I’m also checking out ways that AwStats might be tinkered with to get rid of older spam, too. Perhaps by removing the AwStats log file for the month, a new one will be generated automatically, this time fully applying the blacklist filter. I’m almost afraid to try it, though, for fear of screwing things up.

Still, hopeful news. But I’m not counting my chickens yet–spammers have blown through defenses before. Time will tell. I’ll keep you posted.