Faking Approval?
Something which has long been suspected among some vendors on the Internet is faked good reviews of an item. When something is sold on a site which allows customer reviews, what is to keep the seller from using varying IP addresses and user accounts to create a herd of sock puppets to give glowing, rave reviews of a product? The answer is, nothing. And the problem is that it is very difficult to find out if this has happened, as tracks can be covered pretty well and most people would not deem such an offense worthy of a detailed, difficult investigation. Even so, fake reviews are sometimes caught.
Right now, I am wondering if I wasn’t the victim of fake reviews for an iPhone app. After downloading only free apps for several weeks, I started buying a few cheap paid apps. The first was a good purchase–something called “Mileage Tracker,” an app which allows you to track what mileage your vehicle is getting by inputting odometer readings and information about each gas purchase.
But the second app I bought was a Japanese language study aid, a kanji flash card program called “Kanji Flip.” Priced at what I believed was a reasonable $3, it promised to drill me on kanji based on JLPT levels, keeping track of wrong answers and working me out on those more than other cards. As usual, I checked out the reviews, and they were all glowing–and maybe that should have tipped me off, as almost no app gets purely good reviews. Anyway, I bought it, and quickly discovered that I had bought a dog of an app.
First, let me spell out what’s wrong with the app; if you don’t need to know this, then skip the next five paragraphs.
The problem that immediately stands out is that the data is incomplete. Several flash cards have missing or incomplete Japanese readings (pronunciations) of the kanji. There are only 2230 cards in the set, it would have taken the author only short time to check for blanks, maybe an hour or two to check for missed data. See below just a half dozen examples of botched cards out of more than a dozen I found after reviewing only a few hundred cards–which means well over a hundred cards are seriously flawed, about 5%.
Furthermore, the app is totally non-configurable–in any flash card set, you want to have some control, some manner of setting aside unwanted cards or being tested in some way; this app has none of that. In fact, the only “feature”–marking your guesses right or wrong–is user-driven, not part of the app itself. It doesn’t test you, it asks you to test yourself.
Then there’s the mixing of the levels. You would think that the author, having categorized the kanji by their JLPT levels, would have kept them separate–but no, when you choose a higher level, it includes flash cards from all the lower levels–there is no way to keep out the easy cards.
But what is worst about the app is the repeats: as you continue using the app, it starts sending you more and more repeats of the same cards. I just tested this by trying out level 3, which has 986 cards. The first seventeen cards were unique–but then it tossed me three repeats from the first seventeen. Then, after three new cards, it gave me eleven repeats in a row. Then I got six new cards and eight repeats. Out of the fifty or so cards I tried, nearly half were repeats, and a couple were given three times. With nearly a thousand cards in the level, it would have taken far longer than it should have to get through the entire set.
Even worse, many of the cards which were repeated were ones I had tagged as having scored correctly on–in fact, the last four I tried were all previous “correct” cards. The author says that he gives you some “easy” cards to “encourage” you, but swamping you with cards you have already mastered is aggravating, not encouraging. But the only useful feature of this app is the claimed ability to track wrong guesses and give them back to you later. These two features cancel each other out; the author could easily have included no wrong answer tagging and disguised it with the “easy” card “encouragement”–there was no way to distinguish these two features at work from simply random card repeats.
Okay, so the app is a dog–so why all the rave reviews? It’s rare for an app to get more than four stars on the App Store, especially one with so many problems and limitations. Then I noticed that a lot of the reviewers gave simple, one- or two-sentence reviews, and had no history of other reviews–and a few actually gave only one or two other reviews, those being negative reviews of competing apps. Wanting to warn off people who might be getting the wrong idea, I wrote my own review.
Then I noticed something else happening: you can also rate reviews as being helpful or not helpful. Suddenly, most of the rave reviews, including some of the simplistic ones, started getting only “helpful” ratings–and my review suddenly got a whole bunch of “not helpful” reviews. And one five-star commenter actually edited their review to include a criticism of my review.
Like I said, there’s no way to check, but it all comes across as being very fishy. Still, all I lost was $3, and it re-taught me a good lesson: don’t always believe the customer reviews for a product, especially if they’re unusually glowing.
It’s definitely a weakness of the internet. In person, you can see who’s talking and judge for yourself their credibility; also, it’s generally hard to (in person) pretend to be someone else saying the same thing (like “this app rocks!”)
It’s one big reason that at least some power will continue to remain in the hands of influential reviewers. Lots of different categories of businesses had hoped that with the coming of the net, the power of the “chosen few” to really make or break your business would be lessened.
Open a restaurant, you better hope for a good review from that snooty writer from the local paper, or you might be in trouble. Selling a better widget, you better hope that the widget reviewer doesn’t “ask” for you to place too much advertising in hte reviewer’s publication, or else a review of your widget might never appear (or worse, be negative).
But instead, there are still individual reviewers that have power because of the trust factor. Once someone gets a name for themselves reviewing, people will (in theory) pay more attention to their reviews, because they know that person’s history and can judge their credibility.
What the net has mainly done is make it easier to aggregate reviews, but from generally reliable sources. Want to shop for a new car (like I did recently)? You can enter “Ford Escape Hybrid 2009 review” into Google and get several reviews from reasonably reputable people all in one search result.
Or if you want to see a movie review, you can go to RottenTomatoes.com and see aggregate reviews from the “top” critics, from the overall community, or from the “T-Meter” critics (which is weighted). You can even set up a list of YOUR critics, so if you notice that there’s a writer who consistently likes the films you like, you put ’em onto your list and can better notice movies that they like (which, presumably, you’ll like).
The other thing that the net does, in the long run, is offer a chance for the wisdom of the crowd to overwhelm the fake reviewers. If the review site (such as Apple’s review process for their apps) is set up properly, it will be hard for a single source (like the author/developer of an app) to automate the review process- and therefore they can’t jam the reviews with their own sock puppet BS.
Over time, as more and more genuine people write (and rate) reviews, the ones that are fake will slowly lose credibility.
Key to this, IMO, is to try and discern whether or not a review is real or fake- and then give it a “helpful” rating based not on whether or not you agree with the review, but on whether or not it’s really helpful. For example, if someone rags on a product but gives you helpful basic information and the product actually doesn’t fill THEIR needs but the info shows it might fill yours, then you should give it a “helpful” mark; it told you something you needed to know.
The other thing that I think will make a difference in the net in the long run is micropay. Nobody has yet figured out how to crack the credit card companies’ effective monopoly on quick electronic payment, but someday someone big is going to wake up and realize that micropay is the Holy Grail of making money on the net.
Thanks to the effects of the “long tail”, there is a vast amount of money to be made out there- it’s just in small, small amounts. For example, Luis’s web page is great. I love reading it and would pay- but not very much- to do so. If there were an effective micropay system set up, where I shell out a half-cent or a penny each time I load that day’s page, it wouldn’t cost me much at all- but if *everyone* had to do it, he’d make a fair amount of dough for his time.
I have a site (www.faafollies.com) that I’ve built up over the past couple of years. I recently went over a million page hits and regularly get 2,000 to 5,000 page views a day by anywhere from 1,000 to 1,500 readers. Right now I do it because it’s a labor of love (or, more correctly, a labor of loathing for my employer, but don’t get me started) but if I were getting paid even a bit for that, it’d make a big difference in my willingness and ability to keep the site going.
Micropay would help with the reviewing process because presumably, a site (like Rotten Tomatoes, or whatever) that does an excellent job of the review process would be able to charge a fair amount for what they do- and hence be free of the need to advertise.
Relatively few sites on the Internet manage to make money by charging for content. Almost everyone tried it and failed- AOL, the big news producers, etc. ESPN has managed to do it, and a few other sites, but other than porn (which is a whole ‘nother world in and of itself) for the most part you can’t make money by charging for your content.
The reason you can’t make money charging, though, is because the way to get money from them to you is too much of a hassle. You’ve got to charge too high an amount to break even, thanks to the credit card companies’ monopoly; since they have a “per-transaction” charge in addition to taking a percentage of each transaction, if you don’t charge at least that transaction fee you’re screwed.
Micropay. Micropay, when it comes, will revolutionize the Internet. And this totally wasn’t where I thought I was headed when I started this comment but the micropay thing has been in the back of my mind for months now.
About the only guys that I think could carry it off right now are Google… Microsoft could, but they’re too greedy and would try and emulate the credit card companies instead of truly being free thinking. Yahoo is a sinking ship. Apple maybe, but they’re greedy too and haven’t exactly been super-smooth lately (the iPhone ver 2.0 software sucks compared to earlier versions).
Micropay. Remember the concept, folks- you read it here first. That’ll be one-tenth of a cent.
This is the author of Kanji Flip. I did not even know this blog existed, but a friend of mine found it, and I thought I would reply to some of your comments.
The thrust of your post is about fake reviews. They are not fake, and in fact the one from the handicapped reviewer really made me happy. Not the “$3 paid for my app” kind of happy, but because I actually helped someone to learn kanji. So if you feel the comments are fake, well, there’s nothing I can do to dissuade you of that.
As for the program’s content flaw on some of the kanji, I will agree with your findings. It seems a couple of the “fake” customers/reviewers have contacted me with similar discrepancies. I checked the data, and they were right; as are you. My import parser seemed to have truncated some readings. Also, as I mentioned in the legal section, the data is from KanjiDic, so whatever flaws exist in Monash University’s vast effort, are beyond my control.
I have taken the past two days to re-write a new XML parser to update the readings & meanings. Mostly to alleviate a flaw I didn’t see and don’t want to exist.
As for the progress system, I am sorry you feel it is too repetative for you; it seems to me you are more advanced than the target audience of this app. Or perhaps you simply don’t need as much re-inforcement. There’s no way for me to know.
I will say this: the repetition system is not at all as you describe it. It’s based upon an algorithm I built from a couple of years of actual use by myself and others. It’s loosely based off the Leitner system, but there are lots of things I didn’t like about Leitner’s so I tweaked it till it taught me (and others) very efficiently. The algorithm was to be used in a similar learning program for Pimsleur to filter their 20,000 phrases, but that hasn’t happened quite yet. Again, if the system doesn’t work for you, I apologize, but it works great for all those reviewers.
(One of the reviewers who contacted me from Microsoft, loved the program but found the missing readings as you did. However, he simply contacted me at Feedback@KanjiFlip.com instead of lambasting my unprofessionalism and lack of effort. Also, there seems to be 423 missing readings in the KanjiDic file. I will try to fill in these gaps. Not to satisfy you, but to accomdate the people who actually contacted me like civil people, none of whom felt “wronged” or “robbed”.
Lastly, I don’t have time to surf the web, blog, or do much but create my “crappy” apps. Did I put my dogs into the About screen? Hell yeah, I did. Did it bother anyone but you? Nope. Maybe when you write your app for iTunes, you can be more professional and not put your dogs into the About screen. But for me it’s pretty “sweet”. Take care bud.
Andre
There’s one alternative Kanji app at around $1, but it only features the JLPT4 Kanji. I’m waiting for a good Kanji learning system for the iPhone.
Meanwhile, there’s at least the free wwwjdic interface. Free and not bad…
http://cslmedia.org/iphone/
Andre:
I am happy to hear that the reviews are not faked. All I can say is, that is simply what appears to be the case. Check out paid apps, and you will find that ratings of 4 1/2 stars are very rare, and usually richly deserved; people are not shy about giving a low rating for whatever reason, and usually do. The plain fact is, your app has flaws which, if the app were free, would be overlooked; if there were no cost to this app, I’d have given it a much higher rating. But you are asking people to pay money for it, and there are certain things that come with such a demand. One of them is the necessity of quality control.
In short, when a product with rather significant limitations and such a high error rate receives better ratings than apps which obviously have far more effort poured into them, and much more quality to them, it gives more than a little reasonable cause to doubt the validity of many of the reviews.
While you “agree with my findings,” you fail to take responsibility, blaming the errors on others. The thing is, you published this app and asked people to pay money for it, and that means you are responsible for its quality. Anyone who has been in business for any amount of time knows this basic fact, which is why you proofread what you make before you present it to the world. Know it as “QC,” or Quality Control. You put it out with your name on it, yet failed to do even a cursory check of the output, you cannot claim foul nor fob off responsibility upon others when complaints are made.
Have you ever heard of the “Checkers” speech? If not, then you should check up on the whole story–you have just recreated it in part if not in whole. In short, Richard Nixon, as the candidate for vice president in 1952, was accused of taking inappropriate gifts. He went on live TV–one of the first uses of the medium for such a political appeal–and denied taking any gifts but one: a cute puppy. He then defiantly defended accepting that gift, and threw it back in the face of his opponents, saying that his kids loved the dog and they weren’t going to give it up–making his opponents seem like jerks who would take puppies from little kids’ arms to score political points.
The fact is, I never criticized your dogs, nor would I–I know a lot of people like to see that kind of stuff, and don’t begrudge it. I never referred to the dogs nor were they the point of my criticism. The criticism was that of unprofessionalism, which I evidenced with two main facts: (1) lack of even fundamental proofreading, and (2) the use of cutesy words in functional areas of the app. And now there is a third: the style and tone of this comment, which I’ll get to in a minute.
The point is, I made a factual point and provided evidence, and in response, you brought up sweet little puppies which were never mentioned: the classic “Checkers” move, the “straw man” argument. There is an echo of this fallacious logic in your third paragraph, where you use a handicapped person writing a review as evidence that none of the reviews were faked.
Finally, there is the response itself. I do not fault you for it as an individual; hell, if met with the same accusation, I might well do the same. The problem is that you are now a business; you are offering a product for sale and are taking people’s money for it. That does not mean you don’t answer criticism, but it means that you can’t have such a thin skin about it. You answer charges and, even if you feel that someone has charged you unjustly, you take the high road. That’s part of professionalism. If you don’t want to be a professional, then fine.
But if you don’t want to be a professional, then you have no place coming to people whining that they have offended you with charges of unprofessionalism.