Data is often peddled as the new oil, an enormous untapped asset with the power to transform the world. Like oil, data has spawned a lucrative industry of Silicon Valley giants like Google and Netflix, whose success is seemingly predicated on collecting, generating and weaponizing data.
But too often, data is hawked like snake oil. Software companies peddle data as the panacea to all problems, promising the only way to survive in the digital age is to buy tools for collecting data, then buy more tools for using it. The reality is that most of these tools don’t deliver value in large, complex enterprises. Across sectors, organizations are struggling to realize the potential of their data warehouses, predictive analytics, machine learning, and other tools.
As President of Palantir, a technology company that builds software for institutions with complex data environments, I’m no stranger to the snake-oil promise of big data. I’ve watched some of the world’s most preeminent institutions distracted by the battles of digital transformation, only to lose the war against upstart competitors. I’ve seen transformation projects that took years to yield results, if they did at all.
Done poorly, digital transformation is an exercise in buying tools and services. Done well, it’s a means of leveraging your unique strengths to evolve your business. I hope to help you invest your time and money more wisely by sharing two of Palantir’s guiding principles for effective transformation: solve the right problem, and follow the right path.
Solve the right problem
What does it mean to solve the wrong problem? Let’s look deeper into Google and Netflix. Lots of vendors are pitching AI as the way to replicate their success in your organization. But Google and Netflix make money on predictions. Their core businesses are driven by a single model: predicting what you want to click on next. They make one decision, over and over, and they dominate because they record each outcome with great precision. The first result on a Google search, or the first show that pops up on your Netflix homepage, is their best guess for what you want to see. If you click on the third result instead, they learn from that, and the next time you might click on the second result, and so on.
It’s easy to take Google and Netflix’s example as signal to invest in prediction. But for most companies, prediction doesn’t solve the fundamental business problems at play, and what works for Google and Netflix won’t necessarily work for an automotive company or a freight operator (or an industrial manufacturer, or a naval force, and so on).
Freight rail operators can predict which wheel is likely to break, and subscription businesses can predict which customers are likely to churn. But what about maintaining a fleet so it stays on the rails, or creating a product that customers want to keep paying for? Prediction might help optimize one slice of the piece — say, keeping one wheel in motion, or keeping one customer on the books. But investing in decisions will help optimize the whole business model.
Consider the subscription business. The business can build a machine learning algorithm that tells us which customers are likely to churn, but at that point, it’s too late to make a real difference. If the company can predict it, it’s because the customer is already dissatisfied, which they’ve indicated by consuming less content or turning off auto-renew. The business could save them, at least temporarily, with a costly last-minute intervention. But what if they used data to improve their offerings so they don’t want to leave in the first place? (Netflix doesn’t use algorithms to decide what original content to invest in…)
Or consider predictive maintenance, the holy grail for manufacturers and operators of complex machinery. A freight operator can predict which wheel on which train in its railroad network is likely to break, but intervention is expensive and impact is minimal. What if the operator focused on using their data to improve how wheels are ordered, installed and operated instead?
Ultimately, unsatisfied customers and creaking wheels aren’t the fundamental problems businesses need to solve. They’re merely symptoms of the real problems, the ones that keep executives awake at night. Snake oil salesmen prey on these fears, promising simple fixes for complicated problems. And it’s true that algorithms and AI have real value in many organizations. They can be powerful tools that augment our most valuable assets: people and knowledge.
But business is not about making predictions. It’s about making decisions, and business leaders aren’t making just one decision at a time. They’re making dozens or hundreds of interconnected decisions every day to move the needle on problems fundamental to the success or failure of their business. Ensuring business leaders focus their energies on the right questions is vital to any digital transformation.
Follow the right path
Once we’re focused on the right problem, the question becomes how to solve it. Organizations often attempt to mimic Silicon Valley upstarts in their digital transformations, but what works for a young tech company may not work for a decades-old organization.
Fortunately, what makes transformation hard is also what makes it valuable. An organization with decades of accumulated human knowledge and market leadership starts from a position of relative strength. The key to maintaining that position is bringing in technology in a way that pushes the organization forward while honoring its evolutionary journey.
It’s tempting to lead with technology, to dive into the data and come up with an “insight” (e.g., customers who turn off auto-renew frequently cancel their subscriptions altogether). But if we start out by digging for insights to build models on, we gloss over the critical part: how we actually deploy the insight, learn from it, and refine it. We also risk a lot of correlation/causation errors when we search for the miracle insight instead of doing the hard, slow work of understanding the system. Building a model that predicts a single point of failure in a complex system doesn’t account for all of the upstream decisions that led to that outcome. Perhaps more insidiously, it treats all the inherent problems in that system as a given.
Effective transformation starts with humans. Organizations run on naturally occurring decision loops. When we start a new customer relationship, the first thing we do is seek out the humans at the center of these decision loops. These are the assembly line workers, maintenance managers, sales reps, biologists, and so on who actually do the work of carrying out an organization’s mission. We ask them: What decisions do you make? With what information? Under which constraints? As we begin to understand the system, we can begin to optimize it.
But the focus on the operator doesn’t stop where technology starts. Too often, “data” (and big data, AI, ML, etc.) is carved off as a separate, highly specialized part of an organization. This is a failure to honor the organization’s reality, and in fact distorts it further by adding another layer of bureaucracy. We’ve accepted that real-world businesses are built on decision loops, not single decision points, with humans necessarily at the center. The best transformation brings technology in to augment them.
At Palantir, we’ve long believed that the most powerful technology marries the complementary skills of humans and computational systems. We commonly hear that workers in today’s global economy will have to be retrained to interact with tomorrow’s technology. This framing assumes that technology is inevitable, and that the onus on workers is to adjust, but experience doesn’t bear this out. Look at Toyota’s Production System, built on the concept of “automation with a human touch” and consistently the source of the automotive industry’s best profit margins, including over upstarts like Tesla, whose Elon Musk has brought humans back into a once-automated assembly line. To quote Musk: “Yes, excessive automation at Tesla was a mistake. To be precise, my mistake. Humans are underrated.” We can view his mea culpa as evidence of the primacy of human decision-making — an illustration of his power to change in response to learning.
Knowledge and understanding are what differentiates an organization from its competitors. Data is only an approximation for knowledge. It’s necessary, but not sufficient; we have to curate data into knowledge and synthesize knowledge into understanding, and for that we need human judgment. Technology that complements the existing skills and knowledge of many different kinds of workers without dramatic re-training or adaption generates the fastest and most sustainable results. Half of Palantir Foundry users in heavy industry are blue collar workers on the factory floor. They use Foundry to make decisions informed by data, and Foundry records their decisions. Their back-office colleagues analyze, interpret, and, where appropriate, model the results to make continuous adjustments that improve the whole system.
Conclusion
Snake oil salesmanship has conditioned us to believe that modern businesses need big data tools to compete. But I hope I’ve convinced you that treating technology as a goal in itself won’t position you for long-term success. AI and ML aren’t silver bullets — the true source of alpha in today’s market is your rate of learning, not your technology. You win when you can evaluate your decisions and make changes based on their outcomes faster than your competitors can. Data and technology will give you an edge in this battle, but only if they support the people making decisions, rather than impeding them.
Seek out the right problem, and solve it with the right approach, starting with your humans and the decisions they make. Invest your technology dollars in a system that improves over time as people work together to create knowledge, make decisions, and learn from them. These decision loops, the flow of human knowledge, are the engine that powers your business. Don’t look to technology to replace this engine — look to technology to supercharge it.
