The Answer is in the Crowd

Back to Boeing

2009-10-09T15:16:00.017-05:00

In June, I wrote my first hypothetical Prediction Market case study, featuring Boeing and its beleaguered 787. Establishing a theme for subsequent blog posts, I posed the question: What might be different if this company used internal markets? The idea was to ask the same question about many organizations, exploring the myriad ways Prediction Markets can help solve today's business problems.

But today, this blog returns to Boeing for inspiration.

I don't mean to pick on Boeing, nor even fixate on them. And I certainly don't mean to ignore all the other companies that have Prediction Market-worthy news headlines. Indeed it's nearly impossible to flip through a single day's Wall Street Journal or New York Times without confronting an organization that needs improved decision intelligence.

But the latest development in the Boeing Dreamliner story provides a convenient segue to a topic I've wanted to address: the ROI of enterprise prediction markets.

From what I've seen in the marketplace (both first and second hand), Prediction Markets frequently face the same bottom line scrutiny as any other enterprise application, tool, or resource. How much value will it generate and when? Arguably, the business case should be extra tight when evaluating something new and unconventional. And prediction markets fit the bill. They are not yet widely adopted; they stem from newfangled trends such as crowd-sourcing; and most egregiously, they challenge traditional management orthodoxies.

So it's no surprise that discerning corporate decision makers want to know what they'll get for their investment. What is often surprising, however, is the way in which they approach the equation.

Here's what companies often say:
This seems very promising, but I'm not sure it's worth the cost. What kind of returns can I expect? What is the value of this new information?

Here's the cost-benefit equation those comments imply:
Value = benefit - cost
   or ...
ROI = benefit / cost
   where ...
   cost = price of Prediction Market solution + cost of internal time & resources
   benefit = value of the information (generated by the Prediction Market)
Not to get hung up on the math, but these simple equations are missing at least three variables:

the cost of not doing
the cost of alternatives
the multiplying factor of the company's management "skill"

The cost of not doing

Here, I'm simply flipping around one of the original questions. Instead of only asking what is the cost of doing something, sophisticated leaders also evaluate the cost of not doing something. In other words, what is the risk of passing on a particular opportunity, or ignoring a particular problem? Those risks should be considered, and considered as costs. When those risks manifest into missteps, the costs become clear and public. (In the case of Boeing, a $1 billion write down.) I've already written about this a bit in my last post, so I won't belabor the point here.

The cost of alternatives

Again, we return to the Boeing example. Wednesday, the WSJ reported two interesting developments about Boeing's commercial airplanes division. First, that production delays were not only plaguing the Dreamliner 787, but also the latest version of the 747. Second, that the company has taken substantial measures to get a better handle on its production process. This includes building a high-tech war room to more closely monitor the Dreamliner's progress:

Vital to Boeing's plan for keeping the 787 on track as it starts building the 850 planes on order is a space center-style control room -- officially called the Production Integration Center. One of the hub's wide glass walls overlooks the Dreamliner final assembly line, where the plane's body and wings come together. On the opposite wall, 24 big screens display information including overseas shipments of parts, urgent technical questions and even earthquakes around the globe, which could misalign factory equipment and cause delays. Suppliers as far afield as Australia, Italy, Japan and Russia can call in through translators and show Boeing engineers in the center close-up images of their components using high-definition handheld video cameras. Robert Noble, Boeing's vice president of supplier management who runs the 24-hour center, says immediate, multimedia communications have eliminated the problem of often unclear email exchanges between distant engineers who work on opposite ends of the clock. "That takes days out of problem resolution," he says.

Additionally, the company has brought in-house previously outsourced parts of the manufacturing process, purchasing factories or taking large stakes in suppliers. And finally, Boeing says it has added several project engineers to oversee key areas of the production line directly.

One doesn't need to be a rocket scientist (or a jet engineer) to recognize that these efforts are major financial investments. From acquisitions up the supply chain, to an expansion of middle management, to the technology, staff, systems, and secure pipelines that power the communications hub ... Boeing has accepted a very high price for improved decision intelligence. Which, logically, means they have placed a very high value on course correcting the Dreamliner program.

Do they need to make all these investments? As an outsider, I can't answer that question. But one thing is clear. The costs of alternatives to Prediction Markets (assuming that increased communication vehicles, added SME's, and tighter ownership are alternative ways of giving leaders better information) is high. Very high.

Given that, my simple suggestion is that companies factor in those costs when evaluating Prediction Market solutions. Once leaders decide that improved intelligence is a must have, not a nice to have - the actual costs of a Prediction Market are likely to pale in comparison to most alternatives.

One can't help wonder whether collective intelligence tools like Prediction Markets could have prevented the Dreamliner delays in the first place, saving the company billions in write offs, reputation losses, and expensive fixes.

The company's management "skill"

My final recommendation for the ROI equation focuses on the value of the market data. Prediction Markets produce collective forecasts and other assessments, delivering them in the form of dynamic market prices. Additionally, markets generate rich transactional data, which can help companies understand the who, where, and why behind the collective predictions. When markets are successful, this data is both unique and valuable to corporate decision makers. How valuable? In my experience, that depends on two things. First, the value of avoiding missteps and miscalculations like those discussed above. This should be the first calculation. And second, the actions of management in possession of the data. This latter variable is simple yet often overlooked in the ROI calculation. Unique information in the hands of passive managers may ultimately result in little value. Intelligence in the hands of aggressive, knee-jerk type managers may result in some value or may destroy value. Intelligence in the hands of the most skilled company managers can create significant value. Just like any other piece of data that companies capture, value is neither guaranteed nor quantifiable until someone acts on it.
In conclusion, enterprise prediction markets should be held to ROI evaluations, as long as both costs and benefits are fully loaded. In the case of Boeing, it seems a fair assumption that a relatively minor investment in collective intelligence could produce meaningful ROI. Especially when weighed against the costs of not doing anything and the costs of alternative solutions.

The Cost of Not Doing

2009-10-07T11:10:00.018-05:00

In my quest to write hypothetical case studies on companies & organizations that need Prediction Markets, I've admittedly missed many easy opportunities. Just a quick mental scan of this summer's headlines produced several ripe candidates: The Yahoo!-Microsoft Deal, post-bankruptcy General Motors, the "closing" of Guantanamo Bay, the Cash for Clunkers program, etc. All are business or political endeavors whose fate is (or was) unknown, but whose future could have been predicted, potentially, by the proverbial crowd. If only the crowd had a Prediction Market, that is.

A Prediction Market is a crowd-sourcing tool that organizations can use for improved decision intelligence. Employees play the market ... executives get better information. These internal markets can be (and are being) used for forecasting, new product development, capital investments, and increasingly, project management.

But instead of jumping on these summer headlines and blogging up a storm, I got caught up with my own project management woes. Too many things to do, not enough time, yadda yadda.

In my case, there's no real cost to delaying my next blog post. But in the case of major initiatives in Corporate America or our government, there is plenty at stake. The costs of missed deadlines, inaccurate sales forecasts, budget shortfalls, marketing flops, rejected legislation, or failed mergers are significant, and scary. Which is why the relatively simple and inexpensive prediction market solution is so compelling.

As I've written before, Prediction Markets can't solve everything. But they can provide information that decision makers just can't get anywhere else. They can ask Yahoo! and MSFT employees for individual prognoses on the partnership, then aggregate it into actionable data. They can ask GM dealers across the country how many customers will return cars under the 60-day guarantee, ensuring the benefits exceed the costs. They can ask government officials from disparate branches the likelihood of Gitmo closing on schedule, uncovering hurdles and loopholes.

When the answers to these big questions can't be found among the usual experts or with the usual tools ... wise companies are beginning to trust the wisdom of the crowd. They're experimenting with prediction markets, wiki's, open innovation models, and the like. They're trusting the insights and ideas of their various stakeholders, not just their executives.

But many companies aren't. Many companies look at Prediction Markets and similar cutting-edge tools as unnecessary costs without guaranteed returns. This sounds interesting, but what's the ROI? That, the quintessential business question, can't be shunned. But it only takes a brief contemplation of the costs associated with big business blunders to entertain a new form of the question:

What is the cost of not doing?

This Blog is Moving and Expanding

2009-09-27T16:45:00.005-05:00

This blog is in the process of moving to a new location. Why? It's joining a broader site focused on my consulting business. I offer services in Prediction Markets as well as Strategic Marketing and Product Development.

Here's what's happening:

The Answer is in the Crowd blog will be moving to WordPress.
The WordPress site is an umbrella site for my business, Dawn T Keller Consulting LLC.
You can access the new site at: http://www.dawntkellerconsulting.com
To go directly to the prediction markets blog page, click here.
Until the blog actually makes the migration, it will simply link you back to this blogspot site.

See you on the other side!

Background on This Blog(ger)

2009-07-03T10:39:00.003-05:00

Before I post my next Prediction Markets case study, I thought I'd offer an overview of this blog, its objective, and its author.

WHO I AM:

I'm a Marketing professional who got hooked on Prediction Markets (and the Wisdom of Crowds) in late 2005 while working for Best Buy Co.

I spent 3+ years studying, testing, deploying, and evangelizing internal Prediction Markets within Best Buy (in partnership with my boss, Jeff Severts, who initiated the original WOC experiments at BBY).
We were fortunate to have our work featured in publications such as the Wall Street Journal, New York Times, as well as Gary Hamel's book The Future of Management.

In addition to my endeavors in Enterprise Prediction Markets, I have 15 years experience in Marketing and general business management. To me, it's been a career of cool roles, teams, and projects:

Led new service development team for The Geek Squad
Co-led overhaul of Best Buy's extended service plan program
Co-developed initial customer/marketing strategy for Best Buy China.
Developed sponsorship sales marketing collateral for NBA franchise.
Led internal Marketing/Promotion/Creative team for Fox Television affiliates in Florida.
Co-developed integrated multi-media advertising vehicle for television station in 1998.
Initially trained in television station general management, including advertising sales, news production, marketing, and programming.

I have an MBA from Columbia Business School and a BA from Princeton University. That's not necessarily relevant to this blog, except for the following: Columbia educated me as a business generalist; Princeton taught me how to think critically and write effectively.

WHAT THIS BLOG IS ABOUT:

Taking a 'What If' approach, I'm offering up hypothetical case studies of prediction markets addressing current business problems. In other words, how might things be different if internal markets were more widely utilized?

Material will be ripped from the headlines, focusing on current events in the world of business, government, and/or social enterprise.

The objectives of this approach are twofold:

Build awareness and interest in prediction markets among a broader set of business leaders
Make more tangible/relatable an otherwise theoretical management concept.

As such, it is not my current intention to make this a general Prediction Markets blog, but rather a very specific and hopefully unique one. There are plenty of intelligent and experienced subject matter experts from the world of economics, technology, law, and finance who can (or already do) blog/publish on other aspects of Prediction Markets. I’d rather expand the digital conversation than just echo it.

WHY I AM BLOGGING ABOUT PREDICTION MARKETS:

To start, I have the passion. I believe there's a gold mine of untapped collective intelligence inside most organizations.

I am fascinated by the way in which web 2.0 technologies are democratizing organizational decision making and turning old management orthodoxies on their head.
Accordingly, I don't believe that Prediction Markets (or other CI vehicles) are a fit for every enterprise. My experience at Best Buy convinced me that corporate culture and management philosophy play a significant role. We had the right environment.
Though I’m mostly writing about internal markets, I strongly believe that an integrated suite of web 2.0 vehicles can truly transform an organization’s decision intelligence, employee engagement, innovation, and productivity. I don’t know of a company that’s farther along in this transformation than Best Buy (but if there are others, I’d love to hear about it.).

Finally, I think there is a need in the marketplace. While the number of voices in the Prediction Markets camp is growing, I don’t believe there are enough people talking publicly and consistently about enterprise applications. While this is happening in private conversations between service providers and potential customers, I’d like to invite a broader constituency to the discussion.

WHOM I HOPE TO REACH WITH THIS BLOG:

That brings me to my intended audience.

This blog is not geared toward the existing Prediction Market intelligentsia. While I would be honored to have fellow enthusiasts read and critique this blog, I’m not writing this for them, specifically.
I’m writing for a general business audience, decision makers in particular. The discussion will be geared toward people who may have never heard of Prediction Markets, may be minimally aware and curious, may be knowledgeable but still skeptical, or who may be interested but need help engaging others within their organization.

Thanks for your interest in this topic and this blog.

- dkeller 7.2.09

Case Study from the Headlines: BOEING

2009-06-27T18:06:00.008-05:00

James McNerney, Chairman/CEO

Boeing Corporation

Dear Mr. McNerney,

Do you sometimes feel like the last to know? Like people won’t tell you what's really going on? Like transparency is demanded of you, but not always provided to you?

This week’s headlines give me an inkling that you might. As I read the news about the latest unexpected delay of your Dreamliner jet, I tried to put myself in your shoes. If only you had known this would happen … earlier. Perhaps you could have avoided the bad press, the analysts’ chastisement, and the customers’ ire. Perhaps you and your team could have even avoided the delay all together.

Such information deprivation, if the WSJ’s assumption is true, seems unfitting for a CEO. But then again, you're not new to this gig. You know this is a common ailment in the corporate upper-echelon. Communication woes can be the byproduct of the very systems and norms that make big companies work

But like many textbook management problems, this one has a solution set. As the cliché goes, the answer is often in the room. Or in this case, allow me to suggest that the answer is in the crowd.

Respectfully yours,

-A Prediction Markets evangelist

I contemplated starting a Prediction Markets blog quite a while ago. But like the beleaguered Boeing 787, it never seemed to get off the ground. Turns out, all I needed was a hook. Unfortunately for Mr. McNerney, his current predicament fit the bill.

Actually, the real catalyst was my former boss and Prediction Markets partner at Best Buy Co., Jeff Severts. During the 3+ years that we tinkered, tested, and touted internal corporate markets together, Jeff would often cite Boeing as the perfect candidate for a Prediction Markets case study. Following the Dreamliner saga in the news, Jeff was quick to recognize that Jim McNerney likely wasn't getting the information he needed to make effective decisions, announcements, or promises.

At Best Buy, we (and our senior leaders) often felt the same pain. That certainly didn't make us unique. What was unique was the candor with which our executives acknowledged this challenge. This organizational self-awareness, combined with our corporate culture of valuing all employee insights, made us fertile ground for incubating a Prediction Market. So we did.

But this blog is not about Best Buy.

This blog is a "WHAT IF" exercise ... taking headlines from the world of business, government, social enterprise or economics ... and imagining how things might be different if Prediction Markets and other Collective Intelligence mechanisms were widely adopted.

Prediction Markets are fundamentally about tapping the wisdom of the crowd and unearthing information that is otherwise hidden. In these speculative markets, participants trade assets representing unknown future outcomes, essentially 'betting' on events they deem most likely. Rewarded for accuracy, traders are motivated to act on information rather than pure speculation. As the market dynamically aggregates all this information, asset prices can be interpreted as probabilities or expected values. Such collective forecasts have often proven to be more accurate than traditional predictive indicators. This has fueled their growing visibility and popularity in recent years. More on PM's can be found here.

Used within organizations, Prediction Markets are uniquely able to correct information flow inefficiencies caused by bureaucracy, time, human nature, politics and hierarchy.

And so, back to Boeing.

Assuming that at least some, if not all, of these inefficiencies are at play within Boeing, let’s imagine how Prediction Markets could potentially change the course of events in this developing story.

WHAT IF … Boeing used Prediction Markets in its project management toolkit?

Problem:

Six major delays over a 6-year period for Boeing’s highly anticipated Dreamliner 787 Jet.
Problem exacerbated by several executive statements (some as recent as last week) promising, or otherwise suggesting that flight testing was on schedule and imminent.
Latest announcement, along with lack of a revised ETA, sends Boeing’s stock down 12%.
Meanwhile, its leadership is taking heat from the press and Wall Street regarding its ability to effectively manage and communicate.
Delays have been blamed on factors such as outsourcing, parts shortages, vendor miscommunications, design flaws, labor strikes, and installation errors.
With this series of missed delivery dates, Boeing loses credibility with key stakeholders and stands to lose significant profitability.

Hypothesis: Project management of the 787 lacks a sufficient information flow mechanism. This results in delays and/or errors in what’s being communicated up the chain of command, and subsequently to external stakeholders.

Proposal: Boeing should launch an enterprise prediction market, with specific contracts focused on delivery metrics of the 787. The following is a high level summary:

What type of prediction market?

The Boeing case is a good application for project management-based markets – which help monitor whether a project will be on time, on budget, and/or to spec. Boeing could translate any number of questions concerning the project’s health into contracts, such as: Will a particular milestone be reached? Will this budget be exceeded? Will we pass a certain inspection? As traders take positions in these stocks, the magnitude of their confidence or skepticism will influence how much they wager. Once the future event is eventually measured, they will profit or lose accordingly.

Who would trade in this market?

As with any enterprise market, management can decide whom to invite. After that, it’s all voluntary. On the most conservative side, Boeing could limit trading to certain divisions, departments or ranks. A more typical approach would be to include all employees. And at the most ambitious end, McNerney and team could incorporate supply chain partners or even customers into its trader base – acknowledging that in today’s globally networked economy, critical insights about its business exist outside the payroll.

Let’s assume the hypothetical Boeing market is open to all employees. Any individual - whether in engineering, supply chain, marketing, IT, finance, sales, or project management, and located anywhere around the globe – can weigh in on the 787 based on insights unique to their role, experience, and knowledge. All trades are anonymous, which provides the freedom to share information and honesty through the market without concern for retribution or lost face.

When would Boeing employees make trades?

Prediction markets are designed to compel trading when people believe a stock is mispriced. Since prices represent probabilities, traders act when they don’t agree with the current market prognosis. Imagine a contract called: 787 Flight Testing Will Be Ready by June 30. A supply chain employee might be the first to know that a critical parts shipment is missing and short the stock on this information. A technician might hear lunchroom chatter about a difficult instrument installation and sell her shares. An engineer may be part of a small team that has finally solved a nagging design flaw and buys up the stock on new-found confidence. A union member may short the stock, knowing that a strike is looming. A project manager may buy shares, bullish after a successful meeting with an offshore IT provider. The market will reflect all of this in real time. Would a biweekly project status report do the same? Is there another way Mr. McNerney could access or synthesize all of this disparate information and sentiment?

Why would employees trade?

Prediction markets work because traders are individually motivated to generate collective accuracy. Each time a trader has reason to believe a stock is mispriced, he or she sees a profit opportunity. The engineer privy to the design correction can quickly “buy low” before his accomplishment is widely known, confident he’ll be “selling high” before too long. The union member can sell all his shares, avoiding a major loss when the strike blows the current production schedule. These profit incentives are what make markets more promising predictors than surveys, status reports, or meetings – where timing, motives, and voices may be sub-optimal. In our Boeing example, employees wouldn’t be trading real money, but rather an internal currency that may convert to modest cash prizes, giveaways or soft benefits. Fortunately, prediction markets don’t only rely on material incentives. Organizations like Google and Best Buy have discovered the power of reputational incentives (a leader-board tracks most profitable traders) in their markets. Additionally, prediction markets are inherently democratic institutions. For many, having a voice in the system can be motivation enough.

How would Boeing executives use the market?

By tracking the various prices and price changes of these stocks, management would know exactly what employees think and feel at any given moment. For example, if the stock: 787 Flight Testing Will Be Ready by June 30 is trading at only $25 on June 10, management knows their insiders are largely doubtful the milestone will be reached. If the Will We Pass Inspection stock suddenly dives from $70 to $30, they’ll have reason to make a few phone calls. Such quantifiable confidence measures are not easy to come by, especially in real time. In this manner, the market can serve as an unparalleled early warning signal.

Intended Result: Once in possession of these unique insights, Mr. McNerney and his team could do what they do with any other form of information – take action. By adopting the prediction market as one form of decision intelligence, they could more clearly see discrepancies and investigate them … identify pitfalls and avoid them … uncover communication gaps and correct them. In a project as complex, costly, and public as the 787 Dreamliner, there are myriad ways in which better and faster information could be invaluable. How management chooses to act on the information will ultimately determine the value of the market. But the upside is significant. Improving key decisions in project management or external communications could go a long way toward restoring confidence among Boeing’s customers and investors.

As Mr. McNerney surely knows, the answers aren’t always in the room. It depends on who’s in the room! But in an organization with thousands of insightful employees worldwide, there’s a good chance the answers are somewhere in the crowd. This simple but powerful truth is why Prediction Markets could potentially change the course of events for Boeing and its Dreamliner jet. By enabling the company to swiftly and accurately tap its own collective knowledge, markets create an intelligence engine with infinite application and shelf life.

-Dawn Keller, 6.27.09