Paradoxes of Insight
How do you generate insight while not getting trapped in its dynamics?
“The whole problem with the world is that fools and fanatics are always so certain of themselves, and wiser people so full of doubts.”
― Bertrand Russell
The game of innovation is a battle for insights that can be acted on quickly and effectively in a marketplace with constantly evolving requirements. It’s a journey of mastery requiring high energy for disciplined experimentation, savvy interpretation and novel reframing of possibilities.
In what follows, I explore some inherent paradoxes that leaders need to be aware of in order to realize the benefits of insight while not getting trapped in its complex dynamics. This article and its forthcoming companion, Insights of Paradox, form an integrated whole to stimulate appetite for creating breakthroughs in both thinking and action.
Let’s start with a quick look at the phenomenon of paradox itself.
Paradox as Statement
A terse statement that is seemingly contradictory, yet apparently true, is a paradox. For example, in design we sometimes hear “Less is more” which is rather puzzling until you realize that avoiding extraneous detail results in greater impact. Or in business we are sometimes exhorted to “Do more with less” which sounds nuts at first blush but means “get creative and work out how to get the job done with fewer resources.”
Paradox as Situation
Another take on paradox is that it can be a self-contradictory situation that at first seems true. For example, in the spirit of two-way communication and openness to listening, your new boss trumpets that “My door is always open!” yet that office door is always closed whenever you actually visit. You knock and are greeted with a gruff, “Can’t you see that my door is closed and that I’m busy?” Worse, if you foolishly point out the contradiction, you may have the paradox reiterated with “Yes, it is always open. Now please close it on the way out. I have important work to do.”
Organizational life is filled with such gems which you need to navigate deftly in order to avoid any career-limiting moves. Early in my career as VP of Research and Development at TeleLearning Systems, Inc., I was strongly encouraged by the CEO and Chairman to leverage risk in service of breakthrough innovation.
“I want you on cutting-edge projects where you take risks at the 90% level,” he urged me.
“You realize, of course, that means that out of a random sample of 10 such projects, then 9 would fail?” I warned.
“Yes, I do. Just make sure that you select and work on the winner first!”
As any entrepreneur will relate, and the dismal statistics of failed startups and abandoned enterprise change efforts will support, it is rarely obvious ahead of time exactly which projects will be outright winners, even after some clever pivoting.
Little wonder that insight is a highly valued phenomenon, especially in addressing the demands of complex problem-solving and action-oriented design thinking. Unfortunately, however, we have little insight about insight itself.
Even after more than a hundred years of empirical research in psychology and several millennia of philosophical wonder, the mechanisms that underlie insight continue to be shrouded in mystery. The trio of intuition, imagination, and insight appear to be related gifts bestowed on a lucky, creative few and, even then, only at limited times and under special circumstances.
Genuine insight causes you to see the world differently. New possibilities open for action-taking and positive growth. In essence, insights shift your reality, but often in unexpected ways.
There are more things in heaven and earth, Horatio,
Than are dreamt of in your philosophy.
- Hamlet (1.5.167–8), Hamlet to Horatio
Virtually all managerial positions in the digital economy increasingly rely on data-driven decision making. Cutting-edge enterprises use analytics as the basis for creating value through managing key analytical processes and using the results of these processes as the basis for making informed, evidence-based decisions.
Learning the process of reframing a business question as a data question and reasoning about what data might be of assistance is just the start, of course. One also needs to master methods for obtaining such data, integrating and scrubbing the data, and performing the analysis. Finally one then engages in deriving and communicating “insights” from the analysis to various choice stakeholders.
There are, however, very important domains that do not lend themselves to such an approach. For example, despite all the critical success factors (CSFs), relevant key performance indicators (KPIs) and cool related metrics you might generate, effectively leading teams (perhaps especially of business analysts and data scientists!) still requires a wise and intuitive human touch, driven by empathy and nuanced emotional intelligence.
Imagine being the CEO of a rapidly growing company selling advanced tools in the burgeoning space for enterprise analytics but realizing that your executive team, while good at executing current strategy, is not very creative, much less innovative, when it comes to future strategy.
Your team tends to think small, they believe tried and true incremental steps forward will suffice, and they tend to act only within their established comfort zone as executives.
But as CEO, you realize all too clearly that you need a serious leap to a new level of performance if you are to remain competitive in a marketplace filled with up-start competitors with plentiful VC money, masterful in applied AI and all shooting bullets with your company name on them.
It is rather ironic that the very tools you build for computational insight are powerless for giving you insight about generating the breakthrough needed to address the growing existential threat to your business.
How in the world do you evoke, perhaps provoke, a revolution in both strategy and action in your executive team?
Let’s leave that complex situation with its disturbing twist alone for now and, instead, focus on exploring a rather basic question.
What is insight?
Insight is the phenomenon of apprehending or seeing into a situation, event or system with such penetration that you understand its fundamental workings or essence. With insight, you suddenly perceive a pattern that was previously hidden, unavailable or even invisible.
Insight is driven by the precipitation of informative connection between the previously unconnected.
The phenomenon of a sudden “a-ha” moment can be thought of as the result of crossing a threshold for cognitive reorganization. Before reaching the threshold, there is nothing in conscious awareness but once the dots all snap into place with a thread of unexpected connection to create a discernible pattern, there is a release of energy, often both relieving and joyful.
The cognitive system then enters an extended refractory period where it begins energizing once again but is unable to reorganize for a period. It is as though the new insight has a gravitational pull on the newly organized ideational elements that locks the emergent pattern, and the system that contains it, in place.
An insight occurs in a complex cognitive system (e.g., a person, a team, or even a whole organization) when it reaches a tipping point for the sudden, self-organizing reconfiguration of constituent elements. The system shifts from perplexity to emergent clarity. What was previously disjointed falls into a distinct pattern for new action.
Complex cognitive systems may exhibit multiple stable states and threshold responses to energizing “stimulants” such as curiosity, puzzlement and especially paradox. Once a system crosses a threshold for insight, it exhibits a new stable state that may be more powerful (e.g. high performance) or that may require new management and leadership choices (e.g. strategic redirection or even reinvention).
This capacity to reframe disparate events or ideas into meaningful wholes, and thereby act more effectively on the basis of that emergent insight, is vital for any professional, team or organization intent on staying relevant, creative and being innovative in both the modern workplace and the new digital economy.
The Expansion Paradox of Insight
In gaining insight about the dynamics of some deep mystery, the scope of that mystery may actually expand. It is as though the closer you get, the further away complete understanding becomes. Answers ignite yet more questions.
The more I understand what Heidegger is aiming at the more I see how much still escapes me.
— Leo Strauss, “Existentialism,” lecture given in February 1956
The more you come to appreciate, the less confident you become about totality of your understanding. What previously seemed finite, obvious, and concrete, becomes infinite, mysterious, and intangible. The safe shallows turn into murky depths and the fully graspable becomes fascinatingly evanescent.
Some undergraduates want to be the all-knowing professor. A few undergraduates eventually do become the professor but by then the all-knowing-ness evaporates. In teaching about leadership and innovation at Stanford for more than 20 years, I came to learn far more from my students than they did from me.
A zen master appreciates all too well that the title is honorific and misleading. Similarly, CEOs sometimes laugh that people think they are the most powerful person in an organization.
The Temporal Paradox of Insight
Certain insights are experienced as realizations. You suddenly see into reality and its underlying structure so deeply that you cannot go back to a previous way of viewing the world. Once you have the insight, it feels completely silly or even unnatural to go back to the original way of seeing.
For example, if you suddenly have the insight that a corporate board member, well known for philanthropy, is actually a thief, then seeing this scoundrel’s past good works in the world suddenly becomes unavailable or, at the very least, difficult to hold in the same high regard as previously.
The above might be thought of as a temporal paradox. Here the paradox of insight is that as it reveals a new present, it also conceals an old past.
Another example: Once you realize how a magic trick is done—through being shown (insight via education or even revelation) or working it out logically (insight via intellect: focused attention and problem-solving)—it is no longer magical. You could pretend to still be fascinated but the experience and its meaning have forever been changed.
Where the magician led you to (mis)interpret reality as having the possibility of magic, the insight now leads you to reinterpret reality by disappearing the possibility. The magician becomes just another person with ordinary ability.
Once you see the reality, the illusion is gone. Or even more generally, and profoundly, once you experience reality directly, all illusion disappears (insight via enlightenment).
The Blindness Paradox of Insight
With insight there also comes a paradox of blindness. Insight is an occurrence of re-configuration whereby the previously unforeseeable sometimes comes to appear as inevitable. The new perception comes to dominate and sometimes blind you to yet other possible insights.
Leaders and their organizations sometimes fall in love with an insight that generates higher performance but they may needlessly miss the “next big thing” because they are stuck in a new way of thinking and acting that obscures or even blinds them to an even better framework.
Recall that transitioning through the refractory trough after an initial insight takes some recovery time to re-energize and become available (open) to further new energizing inputs. With enough yet further energizing input, the system ascends again to cross a new threshold for another, possibly even more powerful, insight.
Take for example the famous 9 dots problem so often seen in books and trainings on “creative problem solving.” As you probably well know, the task is to join all 9 dots with only 4 straight lines while never taking your pencil off the paper.
Many people are initially stumped until they are shown or given broad hints leading to the insight of drawing a line beyond the limits of the invisible square they had predictably, but needlessly, imposed around the 9 dots.
Such “thinking outside the box” was celebrated in the famous phrase that ricocheted throughout corporate America.
When executive students attending my Stanford classes on Leadership and Creativity proudly rushed to show me they already knew the answer to this old chestnut, I would then ask them to do the task again with only 3 lines.
The point here is that the insight of thinking “outside the box” usually blinds people to continuing to push the envelope for further and even deeper insights.
Moreover, “the box” is not the invisible square that most people think it is. The problem with this formulation is that it provides next to no power for the new assignment of doing the task with only 3 lines!
Try it now and see if you can do it. (One solution is given at the very end of this article.)
If and when you complete the task with only 3 lines, then do it with only 2 lines and finally show me the one line solution! (Contact me if you can’t stand not knowing.)
A similar phenomenon occurs with the introduction of a proven “best practice” into a corporate culture, often based on some new insight about process and people. Across time, managers usually tweak this best practice to fit their particular circumstances but it is very rare for them to continue to look for yet more and deeper insight for a “better-than-best-practice” or what in SWIFT we call a next practice.
Notice how unwitting acceptance of the word “best” creates pride but also the unexamined presumption of global maximum for all time. That is until a new and unexpected competitor has the insight of breaking the illusion and suddenly arrives on the scene to disrupt your marketplace and threaten your pre-eminence with an “out-of-the-box” solution.
Witness the advent of social media technologies completely transforming the territory of traditional fields such as advertising and PR. So much so, that agencies were forced to become data-driven powerhouses of media analytics and evidence-based insights beyond the wildest imaginings of 3-cocktail lunch Madmen. Mere in-depth demographics, and even advanced psychographics, became overshadowed by the inexorable force of real-time algorithms running on digital machines fueled with computational science.
Transforming data into insight is the holy grail of analytics yet the numbers rarely, if ever, speak for themselves. To be frank, extracting actionable business insights from data is part science and part art.
The scientific or analytic approach to insight uses data science to focus on systematically converting structured and unstructured data into informative patterns and meaningful knowledge via sophisticated machine algorithms and statistical methods. Predictive modeling, artificial intelligence (AI) and machine learning play a major role in extracting such information and insight, especially for so-called “big data” sets that are simply too large or complex to be dealt with by traditional data-processing application software.
Types and Levels of Insight
Algorithmic insight harvested from such computational extraction and scientific analysis can, and increasingly does, inform action-taking for maximizing product engagement, building brand loyalty, generating leads, driving traffic, and ultimately making good business decisions in general.
Transformative insight, however, is another beast altogether. It is an interpretive endeavor that requires human creativity and intellectual ingenuity for generating never before entertained perspectives and radical reframing of possibilities. Transformative insight not only connects the dots in radically new, more powerful ways, it simultaneously opens a space of possibility for taking action not previously available or even imaginable.
The analytic approach is suited for a limited portion of the data “iceberg” viz., events, trends and structure —but it falters for shifting the fundamental models or frameworks within which it is conceived.
While analytics may provide a data-science super-highway to algorithmic insight, and can, perhaps, contribute tangentially to transformative insight, it cannot ensure mechanical certainty for any breaking out of the pre-established computational “box” (i.e., model or paradigm) in which it is framed. Breaking free of the models that generate the structure in which the detected trends occur entails engagement at a wholly different level of insight.
Big data analytics has proven extremely fruitful in many contexts in helping to generate systematic insights but what does a leader do when data is sparse, contradictory or even non-existent?
Recall the CEO trapped in the irony of his own world-class analytics tools being next to useless for creating the transformative insight needed for a breakthrough with this senior team trapped in obsolete insights for effective organizational growth. Sadly, the standard, brute force way to escape the entrenched, limited way of thinking, and move the company to being competitive at the next level, is to bring in a whole new executive team! This wrenching “solution” usually entails severe damage to trust and large costs to institutional memory, not to mention the heavy emotional toll on all involved.
Beyond computation and algorithmic insight, creative breakthroughs rest upon attaining novel insight where previously there was only mystery and associated perplexity. This is where human intuition, imagination, and breakthrough creativity are not only still important but become especially vital for generating transformative insight.
In the complex, modern mall of human thinking and action for technical and social innovation, insight comes in different types and at various levels. Leaders should never allow achieving an insight to paralyze further inquiry for even better connections and even more suggestive reconfigurations. Intellectual straight-jackets are toxic for your sustained success and no insight, however powerful, can be taken as final.
Human creativity is a matter of seeing possibility beyond the fresh options made available by any particular insight, no matter how broad or deep the analytics may be.
Perhaps that is a meta-insight (an insight about insight) worthy of discussion and exploration at your next analytics scrum sprint or larger executive team meeting for game-changing innovation. You can be sure your competitors are busy having that very conversation.
This article is the third in a series. Next up: The Insight of Paradox.
Previous article: The Power of Invisibility.
The MILO Process is an experiential off-site designed to trigger multiple levels of transformative insight for executive teams grappling with thorny organizational challenges surrounding leadership, innovation and marketplace performance. Using an immersive, design studio format, we explore issues in depth to confront the fundamental changes needed to dynamite logjams and create breakthrough results.
The MILO Process employs frameworks and principles from SWIFT—a leading-edge technology for creativity and innovation containing both strategic methods and tactical tools for generating insight. To learn more about how to create the radical shift to ultra high performance, please contact me for a confidential briefing.
