Data mining consumer transactions in May: trying to filter the signals from the noise.

One of our more interesting and sexy recent data assets has been developed through a mega-panel of consumer transactions across millions of consumers and thousands of merchants.

As with all of our data sets, we don’t have access to any personal information.  The data has proven useful to track spending across many merchants, as well as different categories of merchants and is allowing us to take our product in a number of different directions.  One thing we haven’t yet begun to do is to take a data-mining approach to the information whereby we proactively try to identify outliers across all of the merchants.

Ultimately, we believe this will be one of the most powerful ways we can drive insight from the data, but it’s a very time-consuming exercise as it increases the likelihood for a false positive as any bias or nuance or sampling error that occurs will by definition bubble a merchant up to the top.

That said, I thought it might be fun and interesting to do some quick sorts of the data and see what it looks like in May.  So the following is an analysis of what has bubbled up in May, along with some thoughts.  Strong caveat emptor – some of what shows up below is (as mentioned above) the result of bias, sampling error and nuance, and I have done only some rudimentary work to dig deeper.  Hopefully, you will find something provocative, as well as glean a little insight into some of the challenges we face turning raw data into insights.

The first filter I did was to rank companies by the growth in the number of transactions Y/Y in May.  I only included companies with a minimum threshold of transactions to mitigate sampling error.  Here are the results:

1.  Martin’s Food

2.  Alabama Power

3.  Conoco

4.  Fandango

5.  Bebe

6.  Microsoft

7.  Dollar General

8.  Family Dollar

9.  Boost Mobile

10. Access Group

11. Amazon Video

12. Casey’s

13. Intuit

14. AT&T U-verse

15. WinCo Foods

16. Transamerica

17. Playstation Network

18. Brookshire’s

19. Kwikshop

20. USAA

Some thoughts:

• Martin’s Food reflects the integration and name change of their Ukrop’s acquisition.
• Alabama Power simply reflects the fact that Alabama has grown disproportionately to other states in our panel, so is purely the result of panel bias.
• Conoco showed a big spike in March, which carried into April and May and is not representative of any increased demand for Conoco (a much smaller retail brand vs. Union 76) gas, but likely reflective of either name changes for some stations or else changes in what shows up in the fulfillment.
• Fandango likely reflects a fulfillment migration (and/or purchase migration) away from the cinemas and to Fandango.
• Bebe is a publicly traded retailer in the midst of a cyclical decline over the past few quarters.  As the smallest merchant on the list, I suspect the growth is largely due to sampling error, although I suppose there is a chance that there is some benefit from a very easy comp.
• Microsoft is clearly benefitting from the release of Windows 7 in October, which is the month in 2009 we saw the large step up in Microsoft transactions.
• Dollar General and Family Dollar are both likely benefitting from a mix-shift to the discounters in the current economic climate.
• Boost Mobile suggests continued growth in Sprint’s prepaid business.
• Access Group originates and services private student loans and apparently took some share in a period of tightening credit.
• Amazon Video likely has benefitted from the service being accessible from an ever-increasing number of consoles, blue-ray players, wi-fi enabled HDTVs, etc.
• Casey’s General Store’s strong May was confirmed in their monthly same-store report.
• Intuit would suggest more consumers are purchasing Quicken products.
• AT&T U-verse has been growing consistently as they add new markets and new content.
• WinCo Foods reflects the company’s expansion into Utah late last year.
• Transamerica has a relatively small number of transactions, and as a smaller division of a much larger company, it is difficult to assess what might be going on there or if the data is just spurious.
• Playstation Network is apparently benefitting from an increase in customers and content.
• USAA is a financial services company offering insurance and other products to soldiers, veterans and their families; they managed to grow throughout the crisis and appear to be continuing to do so.

Next, I ranked all the companies (again only those with a critical threshold of transactions) by the increase in average basket size (adjusting for clear outliers).  I also excluded gasoline retailers since they all saw increased ASPs as a result of the higher cost of fuel.

Here are the results:

1.  Movie Gallery

2.  Hollywood Video

3.  Cabela’s

4.  Disney

5.  Pavilion’s

6.  Apple

7.  Abercrombie & Fitch

8.  American Airlines

9.  Tivo

10.  37 Signals

11.  AT&T Wi-fi

12.  United Airlines

13.  AirTran

14.  Playstation Network

15.  Carter’s

16.  Metropolitan Life

17.  FedEx

18.  US Airways

19.  TGI Friday’s

20.  XM Satellite Radio

• Movie Gallery and Hollywood Video are brands of the Movie Gallery company that recently filed for bankruptcy.  The ASP spike is clearly a result of promotional activity as a result.
• Cabela’s seems to have some kind of nuance since there is nothing that would suggest a sharp ASP spike for the company – likely it has to do with an increase in monthly payments to Cabela’s branded credit cards.
• Century Theatres (a division of Cinemark) continues to benefit from higher spend on movies, driven by higher-priced 3D films increased penetration, as well as concessions where they also apparently have some pricing power.
• Disney was surprising since the ASP delta in prior months had been in a fairly tight range;  May’s ASP increase also coincides with a non-trivial decline in the number of transactions, so it appears that something is amiss.
• Pavilion’s is a grocer banner of Von’s (part of Safeway); in the grocery space there is often an inverse correlation between traffic and ticket which is apparently what we are seeing here as traffic is down.
• Apple – I think they introduced a new device in April, but I can’t seem to remember what it was….
• Delta, like other airlines, has constrained capacity in order to drive higher prices and we saw the largest jump yet in May (note – many purchases in May are for flights in future months with respect to RASM).
• Abercrombie & Fitch has seen direct sales become an increasingly larger part of the mix lately, and I suspect direct likely has a higher ASP than in-store; this would be exacerbated in our data since our data over-weights direct.
• American Airlines is similar to Delta.
• Tivo is difficult to interpret since the average monthly payment in general is much larger than reported ARPU, as one can pre-pay for various periods across a subscription.  I wouldn’t read anything into this number.
• 37 Signals is a privately held Web applications company that Jeff Bezos invested in.  I am not sure what is driving the ASP increase, but I love their Basecamp application.
• AT&T Wi-fi – did I mention that Apple had introduced a new product?
• United Airlines is similar to Delta as is AirTran.
• Playstation Network has the honor of being the only merchant to make both lists as the collective minds of our young generation become increasingly vacuous.
• Carter’s may have benefitted from the opening of their Web store in April.
• Metropolitan Life appears to be getting higher monthly premiums.
• Facebook’s ASPs have been highly volatile as there are a decent number of very high ($1000+) transactions.
• FedEx played out in their reported revenue per package for the quarter ended May 31^st as they benefitted from very easy comps.
• US Airways is similar to the other airlines.
• I’m not sure why TGI Friday’s ASP trended up in May; I’ve never actually been to a TGI Friday’s, but have always wanted to – perhaps I’ll do some field work.
• For XM Radio, the increase was in large part driven by the addition of the music royalty fee put in place in July, 2009.

So there it is – an analysis of a quick and dirty filter of our May consumer mega-panel transaction data.

As you can see, there is a healthy mix of bona fide trends, nuance, and noise.  In fact, much of the work that we do here at Majestic is spend a considerable amount of time cleaning, modeling, and analyzing in order to separate the signals from the noise; an exercise that becomes increasingly more necessary when taking a data-mining approach to the data since, by definition, all false positives will bubble up.

As we continue to do more work with the data, we will ultimately be launching a product that focuses on data mining across all merchants, as opposed to regular updates on a basket of companies in a particular category such as grocers.  We are certain that this approach will enable us to unearth some real gems for our clients or at the very lease prove to be provocative.

In July, I will post on June data, only the focus will be on negative trends as opposed to positive ones.

What Google search activity can tell us about iPad demand

Tomorrow, Apple will start accepting pre-orders for their highly anticipated iPad tablet.  How many iPads Apple will sell in this year and beyond is a subject that has proven quite controversial amongst investors, analysts and the general public and in my capacity as Consumer Technology analyst, I’ve been asked many times by clients how many units I think Apple might sell.  My answer has consistently been that I thought Apple could sell 5 million units in calendar year 2010, and I always added the following caveat: “this is just my opinion based upon my thoughts around the affordable entry price-point, the halo effect from the iPhone, and a general sense that the time has come for convergent media devices in a tablet form.  All that said, I have no hard data to support my thoughts and we look forward to ultimately tracking actual sales through the same proprietary data we use to track iPhone and iPod Touch sales”.

When I wrote my Apple update note last week, I wanted to include some notes on the iPad in addition to our estimates of CPU sales, iPod sales and iPhone sales which are all tracked by triangulating across various proprietary data sets that we’ve acquired.  I lamented the fact that we had no hard data to include as part of a forecast, since we are after all a data-driven research firm and decided to see how well searches on Google in the couple of month prior to release had been able to predict the first 9 months of sales of other consumer electronics’ devices.   I would like to thank my friend and fraternity brother Jack Krawczyk for making me aware of the ways to use Google trends as a real-time predictor of consumption by pointing me here and here some time ago.  I decided to look at the Kindle (Since Amazon doesn’t break out Kindle sales explicitly, I used our Credit Cards transaction-level data to estimate 450,000 units sold in the 9 months subsequent to its release in November, 2007), Zune, PS3, Xbox 360, Wii, DS, and iPhone and as it turned out there was a pretty good correlation between the searching activity and subsequent sales.

Google provides a very nice interface for tracking relative query volume across multiple search terms at their Insights for Search beta site that also lets you look by geography and/or category.  You can even download to a .csv file if you are logged into your Google account.  By looking at a group of searches around each product related to intent to buy the product or learn about the product (such as “buy Wii”), I created relative search query indexes for each product.  The correlation coefficient was 84% and based upon the number of searches for the iPad relative to the other products, the data would suggest iPad global unit sales of 6 million units in 2010.  Here is a chart illustrating the results:

So with Google search trends suggesting 6 million units will be sold in 2010, I feel a bit better to have some empirical data in line with my own personal estimate of 5 million units which is well ahead of most analysts’ estimates.  Time will tell – all I know for certain is that they will be selling 1 – it’s nearly midnight – let me get my credit card….

The Commoditization of Information–Is Data Like Flash Memory?

Several years ago I was having a conversation with one of our clients: a buy-side firm that takes the same creative approach to unearthing and analyzing unique data as we do, and has been doing so since well before Majestic was founded.  The particular person I was talking to is a brilliant computer scientist whom I have the utmost respect for, and who has demonstrated on numerous times a profound ability to understand (at least from a technology perspective) how the world is evolving.  What he mentioned to me took me aback as it struck at the core of what we were trying to build.  He said, “Don’t you think all or most of this data we are working so hard to gather and acquire will be readily available to anyone in the not-so-distant future?”  Given much of our value proposition is largely focused on the uniqueness and proprietary nature of our information, it is easy to see why his remark had me concerned.  Fast-forward to today, and there is ample evidence that he was correct in his thinking, and loads of data seem headed on a path to becoming a commodity, but it turns out that with respect to Majestic I was wrong to be so apprehensive.

Commodity businesses are challenging ones.  When all you are competing on is price, margins are difficult to maintain and price wars can wreak havoc on a company’s value.  Just look at the history of flash memory, Website hosting, or what is currently occurring with telecom service providers.  Once consumers focus solely on price it becomes a fight to the bottom.  Ultimately, commodity providers need to layer something on top of the commodity product to differentiate themselves.  For instance, consider the crowded space of solid-state drives (SSDs), which many people speculate will grow tremendously into notebooks and enterprise computers over the next couple of years.  While the NAND chips inside the SSD might be considered the commodity, the controller that manages the memory can make a world of performance difference and allow a company to gain a tremendous edge.  So the layers on top of the commodity become the differentiators.

Back in the early days of Majestic, we thought it was imperative to demand exclusivity around all of the data deals that we signed, and we routinely paid a high premium for such exclusivity borne from a fear that without the exclusivity we would lose a considerable amount of our unique value proposition.  Over time, however, it became apparent that the value we added on top of the data through our analysis, triangulation across multiple datasets (including those we gather internally), and ability to distill insight from and bring life and color to the data in a way that was most relevant to our clients became our true value proposition and was also something that other companies would be challenged to replicate.  We realized that it was our ability to answer our clients’ most challenging questions, identify large points of inflection early, and solve our clients’ problems through our triangulation across many datasets of both the proprietary and non-proprietary sort that is what we were being paid for by the majority of our clients.

We therefore decided to reserve exclusivity covenants only for special cases where we determined the data was just too unique and value-added not to do so.  But data that is truly unique or difficult to access is getting harder and harder to find, thus shifting the value from the data to the process and end-product.  Companies that make their primary business simply packaging, providing a front-end into, and reselling data and information may find themselves in trouble.

Data and information are freeing themselves and disseminating at a frantic pace, and the concept of paying large or even very small amounts of money for information is becoming less and less appetizing.  Just ask the newspaper publishers.  As the raw data itself becomes more and more readily available and easily accessibly across numerous platforms, the value will shift from being a provider or platform to simply deliver the data to being a platform whereby the maximum value is extracted from large amounts of data.  Let’s take a look at some examples that illustrate this point.

Consider XBRL.  XBRL is a markup language (a child of XML), meant to standardize the way businesses report data.  Importantly, it is an open standard, free of any license fees.  With respect to financial data, an SEC mandate went into effect nearly a year ago requiring companies to “tag” financial data in XBRL, and there are many related initiatives currently underway, including a bill in Congress that would require XBRL tagging for all bailout requests and expenditures, as well as a bill meant to accelerate acceptance and broader use of interactive data.  Ultimately, in the not-too-distant future, nearly every business metric for public companies will be readily and freely available to anyone whether they want to use the data for their own work or build an application on top of the data in much the same way so many programmers have created brilliant and useful apps for the iPhone and now for Android phones.  Here is a rudimentary example and starting block of what can ultimately be accomplished–a not-for-profit project called Freerisk.  A few other companies looking to build businesses around making data open, free, or highly affordable can be found here,  here, and even Amazon is in the mix, as can be seen here.

So are data businesses becoming commodity businesses?  Not just yet, but it is beginning to happen, and I believe the trends here will accelerate, and this will ultimately happen quickly.  When it does, companies that make their livelihood selling in raw form large amounts of data will see their margins severely compressed–as with flash memory, the consumer will focus only on price.  That is what happens in commodity businesses.  The value will transfer to business models that leverage creativity, analysis, and the ability to drive insight from information in scalable ways.  Just as Apple knew that its hardware edge in the mobile space would not be long-lived so it developed the app store by opening up its SDK in a way that shook up the industry and will allow it to maintain its edge into the foreseeable future, so too must information companies think of creative ways to protect their platforms.

Consider two titans in the information business, Thomson Reuters and Bloomberg.  Bloomberg recently took an amazing initiative that should have received a lot more publicity than it did by opening up its proprietary security codes to anyone.  The interface can be found here.  In the meantime, Reuters was recently questioned by the European Commission on terms of use relating to its own proprietary codes (the RIC codes), and S&P is currently being investigated by the EC as to whether its codes breach any European antitrust laws.  So while the competitors are questioned and investigated, Bloomberg, by opening the codes up to everyone, will have lots of people creating tremendous value–and saving untold amounts of time–for lots of other people, and the codes will become more and more pervasive as a result.  In fact, they will likely become the currency that will be a large one of the many ways Bloomberg will benefit from this decision.  I expect Bloomberg’s bold initiative will be a paradigm shifter for the space, as others will be forced to ultimately scramble in response.  But for now Bloomberg is the one that seems to “get it,” and it will be interesting to see what Bloomberg’s next move is.

As far as Majestic goes, we will continue our mission of unearthing new and novel data sets whose analysis can drive important investment and business decisions, and we will continue to increase our focus on knowing which datasets to triangulate across to answer different types of questions, and how best to present the insight.  As more and more data becomes available to us and to the public at large, we will understand better than anyone else how to combine it with other datasets to answer the questions that drive investment and business decisions.  A tremendous amount of future value will be in both identifying the signals and filtering out the noise, and we’ve invested over 7 years in doing just that.  Our triangulation, testing and filtering process is the analog to the controller of flash memory.  I used to listen to arguments that packaging and reselling the raw content again and again to multiple different parties was the only way to get true value from the business, and I worried that what we were focusing on might not effectively scale.  Ironically, what once troubled me as not effectively scalable I now think of as providing us the opportunity to become the market leader of firms that can add value in the burgeoning world of pervasive and ubiquitous data–a world that is coming faster than most people think.

Hans Christian Andersen and gold: Are the new gold bulls modern day weavers?

In Hans Christian Andersen’s wonderful tale, “The Emperor’s New Clothes,” a couple of weavers promise the Emperor a new suit of clothes made from the most magnificent cloth imaginable. Beautiful and elaborate, the clothes would also have the special trait of being invisible to anyone who was stupid or not fit to view its splendor.  The weavers continued to get paid more and more as they described to all of the Emperor’s officials who came by to check on the progress all of the intricate details and exquisite colors.  While the officials saw nothing, none spoke up for fear of being stupid or unfit. Eventually, the Emperor himself sung the glorious praises of his new clothes that he could not even see lest he expose himself as unfit to be Emperor.  Finally, during a great procession, the Emperor paraded himself naked through town with all the people extolling and cheering the virtues of his splendid new clothes until suddenly a little child shouted out, “But he hasn’t got anything on!” This led to whispers, and eventually all the people were saying, “He hasn’t got anything on!”

Lately, we’ve been reading a lot of bullish rhetoric about gold. Much smarter investors (and likely smarter people) than myself such as David Einhorn, Paul Tudor Jones, and John Paulson have each made outsized, well-publicized bets on the rise in the price of gold, and several hedge funds have even created share classes denominated in gold. The world’s largest gold producer, Barrick Gold Corp., recently announced it had eliminated all of its gold hedges in order to gain as much leverage as possible to the price of gold. Current broad sentiment on gold has been extremely bullish by almost any measure as can be seen here, and central banks have been big buyers as well.  All of this has me thinking.

For nearly 15 years I worked for the brilliant market prognosticator Dr. Marty Zweig, where I was Co-Director of Research and did considerable work on our various models that determined asset allocations, etc. One of our core models was our Sentiment Model, which was contrarian in nature and would look for cases of extreme sentiment in one direction as a great opportunity to exploit a move in the opposite direction. A right of passage for any young analyst starting at Zweig was reading and grading on a very bearish to very bullish scale a multitude of newsletters to identify periods when the so-called “experts” were all aligned in one direction (they were nearly always wrong). The approach is remarkably intuitive (bearish sentiment means lots of cash on the outside that could move in on any positive news and vice-versa) – to learn more you can buy his terrific book, “Winning on Wall Street”. The model worked particular well at the poles. It is difficult to argue that we are not at or rapidly approaching a pole with respect to bullish gold sentiment. Witness the following chart showing daily volumes for SPDR Gold Shares (and the SPDR Gold Trust as a result has accumulated more metal than Switzerland’s central bank):

But sentiment (and 5000 years of history, for that matter) aside, I still can’t rationalize why in today’s world so many people, hedge fund managers (I know, I know – they’re people too), central banks, etc., would want to own large amounts of gold. I’ve talked to some colleagues whose investing acumen I respect, have read some articles and have done some research, but I still find myself wondering.

Many talk about gold as an inflation hedge, and while empirical evidence can be interpreted both for and against its effectiveness as a hedge – a challenging exercise since the price of gold here remained fixed at $20.67 for about 100 years and then $35 from 1934 up until the 70s – I would suspect that in this age of more and more sophisticated contingent claims wouldn’t there be much more precise and less speculative ways to hedge inflation such as these.

The argument of gold as a currency is a little more interesting. Central banks have been buying gold of late and may indeed become net buyers in the near future as the dollar by comparison appears more and more to be an inferior holding. So while gold no longer backs currencies, it is still regularly used to diversify fiat currency holdings. But as gold prices, already quite high, continue higher wouldn’t buying gold become incrementally less appealing? And how does a country with a reserve the size of, say, China’s attempt to diversify in any meaningful way into a supply-constrained asset without impacting the asset to such a degree that it loses the majority of its appeal?

The supply constraint argument may be the most compelling. There are only about 160,000 tonnes of gold that have ever been removed from the earth and this number is expanding at only 1.6% per annum. Money, by contrast, is expanding at a much greater rate right now as money will begin to lose its purchasing power. The supply-side argument makes enough sense in as much as I can see that current gold supply is limited and growing at a far slower rate than money supply, but it’s the whole demand thing that is a bit ambiguous to me since most of the demand appears to be driven by the constrained supply in what becomes something of a virtuous circle: a dynamic being driven in part by a lot of the rhetoric currently extolling gold’s virtues.

This gives me a great idea. If the supply constraint is really the driving force, why don’t we create a vehicle that removes this whole sticky demand and utility thing altogether? We’ll call the units Supply Constrained Assets Minus Any Real Utility or SCAMARUs.

Initially, we’ll only create 160,000 units and the only characteristics these units need to have is that the float can only grow by 1.5% a year maximum and that they oughtn’t be hard since hard assets tend to have some real utility, and we want to keep these babies pure and devoid of any worthwhile use vs. the current hard assets that people currently use to hedge inflation. For instance, steel is quite hard and that comes in handy with respect to building cars, infrastructure, skyscrapers, bridges, appliances, etc. Gold is also hard and gold can be used for fine jewelry, and (pausing…), and, hmmm… well there’s NFL superstar Chris Johnson’s teeth and this gorgeous Vertu cell phone, and… come to think of it, maybe my SCAMARU idea is not so differentiated after all. Wait – I did think of one other thing… The Emperor’s New Clothes were weaved from gold!!! Where’s that little boy when you need him?

Q4 2009: Less bad is the new good! Unless you don’t happen to have a lot of money….

Last week, we put out a report on the lodging industry, and one of the themes that our analyst focused on was that things were getting “less bad”.  This has been a theme throughout our research lately across most of our sectors.  In fact, if I do a Google Desktop search across “less bad”, over half of the results show up in the last few months.

Less bad is an interesting concept.  For hundreds of years, there has been considerable philosophical debate on whether good and bad are relative or absolute concepts.  On the one side, you had the moral relativists such as Spinoza and Sartre, and on the other side, you had the ideas of absolutists, such as Kant.  Having once mistakenly thought that facing the consequences of returning home at 1am, smelling like cigars and wine after a night out with the boys without calling would excuse the relatively “less bad” transgression of coming home a couple of hours late from a tame business dinner, but again not calling, I can say that I now place myself firmly in the absolutist camp.  But what does this mean for investors and for the economy?

While there are a few companies that are actually benefitting from and performing quite well in this challenging environment, such as Priceline (PCLN) and Amazon (AMZN), most companies continue to struggle.  That said, the majority of these companies are doing “less bad” compared with a year ago.

Much of Majestic’s time has been spent identifying companies, and trends within companies, that are the best of the less bad (apologies to my wonderful high school English teacher and wordsmith, Ms. Sherman, for that last phrase).  More and more, we’ve been observing that the segments of companies we follow contributing most to the “less badness” (Ms. Sherman would now be sending me to the gallows) are the segments at the highest end; those that the wealthier Americans can afford.  Here are just a few examples:

From our lodging report, we can see that the luxury hotels, such as the Ritz Carlton, have seen the greatest improvement.  As illustrated below, the declines in Average Daily Room rates have improved the most in Q4 in the luxury segment:

ADR by Hotel Segment

With respect to online shopping across all categories, we’ve seen a tremendous shift in the percentage of overall consumption to higher income consumers.  Here is a chart of Hitwise representing this mix-shift in November ’09:

Online eCommerce Spending by Income

Recently reported November US new auto sales, which were flat in the aggregate year-over-year, showed a notable discrepancy in the performance of luxury brands vs. other brands:

• Mercedes  +19%
• Jaguar/Land Rover +20%
• Lexus +14%
• Porsche +18%
• Cadillac +10%
• BMW +3%
• Mini -32%
• GM down 2%
• Ford 0%
• Chrysler -25%
• Infiniti -26%
• Nissan +21%
• Hyundai +46%

Mirroring this trend in new vehicle sales, our proprietary CarMax (KMX) data shows luxury name plate volume in used vehicle sales up 18.5% in FQ3 ’10, while overall volume is up just 8%, despite luxury name plates selling at a 40% premium to the CarMax average sales price.

In the Homebuilding sector, the luxury builder, Toll Brothers (TOL), reported fiscal Q4 ’09 new contracts up 42% in units and a whopping 62% in dollars, while on average, the other builders (who sell considerably lower-priced homes) saw unit order growth down 1% over the same period.

Other evidence includes Tiffany & Co. (TIF), reporting better-than-expected third quarter results, along with Saks (SKS), which attributed much of the improved performance to its New York City flagship store, representing about 20% of the chain’s overall sales.

Here is a chart showing the dramatic recovery of luxury brands’ (Hermes, Chanel, LV, Prada, Dior and Gucci) transaction growth on eBay’s US platform:

Luxury Brand $ Transaction Growth on eBay

So, there is ample evidence that the highest-end has been doing considerably less bad of late.  Unfortunately, for the majority of Americans, things have not gotten less bad; they remain just plain bad, as is well presented in Nouriel Roubini’s piece in the Globe and Mail.

Unemployment continues to rise and evidence persists that it may remain high for a considerable period of time versus past recessions, due to outsourcing and other factors.

Indeed, when I think of my own skewed perspective, it strikes me that nearly all of my friends and colleagues who lost their jobs over the past 18 months (primarily in finance in New York City) have since found gainful employment, which for many, is at banks that have been bailed out and are now hiring again.  It’s worth noting that most of the unemployed, currently struggling to find jobs, contributed their tax dollars to rescue these banks, some of which are accruing record or near-record bonus payouts, which I would surmise contributed as much to Saks’ recent flagship improvement as tourists exploiting the strong Euro.  Moreover, to be fair, these same dollars have helped spur the recovery of many of the firms of Majestic’s clients and have helped us to enjoy a stronger year of performance than we had planned.

While I don’t consider myself wealthy, I do put myself in the category of those who are currently feeling more “less bad” than most other Americans right now.  And for those of us, like myself, who are going into the holiday season considerably less fearful and more willing to spend money than we were a year ago, we ought to remind ourselves of the words of the great philosopher and moral absolutist Immanuel Kant: Morality is not the doctrine of how we make ourselves happy, but how we may make ourselves worthy of happiness.

Lies, damned lies, and statistics (and Bill Belichick)

This is my first post on a blog that will discuss and explore issues surrounding Majestic Research, equity and market research of both the traditional and non-traditional sort, and data analysis for decision making in general.  Majestic Research was founded seven years ago by me and by Seth Goldstein in an effort to build the first equity research firm driven entirely by analysis layered on top of large amounts of proprietary empirical data from non-inside sources.

Much fuss is being made about Bill Belichick’s decision to go for a first down from his own 28-yard line late in the game on Sunday.  A decision that may have cost him the game (although there is no way of knowing what the result would have been had he punted).  Many talking heads, including radio announcers, sports journalists, current and former players and coaches, my cab driver yesterday morning, and Patriots fans everywhere have lambasted the decision with adjectives such as “idiotic,” “unthinkable,” “feeble-brained,” “uninformed,” and several unprintable on this blog being thrown around generously.  Of course, what every one of these talking heads fails to do is back up their claims with anything other than “common sense” or “anyone who knows football knows…” while every statistical analysis shows unequivocally that Bill Belichick made absolutely the right decision if the measure is the decision that gave him the highest likelihood of winning the game.  It isn’t even close.  In fact, the statistical analyses I have looked at, such as here, actually don’t tell the whole story, since the likelihood of the Colts scoring after a punt is even higher than what is stated, given they would be going for it on every fourth down as compared to a typical series from which the statistic is derived.  So his decision is even more correct than the fine folks at advancednflstats.com have determined.

Now, as a lifelong Steelers fan, I have no love for Bill Belichick.  He is a cheater, a poor sport, and a bully, and I will take a coach like Mike Tomlin over Bill Belichick every time. That said, I have always had more respect for his coaching ability and his willingness to think unconventionally and rely on statistics over conventional wisdom in an effort to win than for any other coach in the game.

As CEO of a firm whose (often unconventional and counter-to-consensus) analyses and insights on industries, companies, and products are driven entirely on analysis of data, I have tremendous empathy for him right now given all of the abuse being thrown his way.  From time to time, our analyses will lead to flawed conclusions; perhaps there was sampling error, or bias in the data, or the data is not a perfect proxy for what we are trying to measure.  I like to think that we get it right more than any other firm precisely because we remove the emotion and human bias and focus strictly on the data, but still, no data is perfect.  Most of our clients understand this and are more interested in trying to understand the “how” to explain the delta between our information and reality than anything else.  But we will always have a couple of clients who will ask how they can possibly trust us going forward, who claim, “Majestic was wrong” (if it wasn’t human error – I would counter that the data was wrong), or spew some adjectives and nouns at us that fall in the unprintable category mentioned above, or in very rare instances even cancel their subscription.  The thing is, just as I would expect Bill Belichick to make the same decision if he found himself in the same circumstance precisely because he made the “right” decision, and it just didn’t work out for him, I would expect our analysts to come to the same conclusion that may have led us astray in the past if the data looks the same, as long as we haven’t learned anything new about the data that would suggest otherwise.  As well, I would expect our salespeople to push the information just as hard as the last time.

If we are correct 80% to 90% of the time, we are doing an excellent job.  Casinos make a lot of money (although not as much as they used to, as can be seen in one of our excellent gaming reports) by only being right slightly over 50% of the time.  The thing is, many people are unable to make the distinction between a correct decision that doesn’t work out and an incorrect decision, and often times this failure will hurt them in the future.  In fact, when it comes to making investments, I would argue that nearly every successful trader does not succumb to this flawed way of thinking and that is precisely what makes her successful when compared to, perhaps, an equally analytical trader who does suffer from this gambler’s fallacy.

So, I commend Bill Belichick on making the clear correct decision in a situation that required some courage and chutzpah (and while my hatred of the Pats makes me glad it didn’t work out, the mathematician in me also makes me a little sad – okay not that sad), and if he was still married, I wonder what he would have said to his wife when he got home.  Because now that I think about it, back when I was covering consumer technology, there were certainly a couple of occasions when a company would report not exactly in line with what I was representing from our data and I would come home clearly upset.  When my wife asked me, “What is the matter?” I’m quite sure I answered, “I got it wrong.”