The Technium

The Shirky Principle


[Translations: Japanese]

“Institutions will try to preserve the problem to which they are the solution.” — Clay Shirky

I think this observation is brilliant. It reminds me of the clarity of the Peter Principle, which says that a person in an organization will be promoted to the level of their incompetence. At which point their past achievements will prevent them from being fired, but their incompetence at this new level will prevent them from being promoted again, so they stagnate in their incompetence.

The Shirky Principle declares that complex solutions (like a company, or an industry) can become so dedicated to the problem they are the solution to, that often they inadvertently perpetuate the problem.

Unions, or example. Unions were a brilliant solution to the problem of capital management which tended to exploit uncapitalized workers. But over time as capital increased in complexity, unions complexified as well, until unions needed management. The two became one system — union/management. So now the problem with unions is that they are locked into the old framework, the old system. They inadvertently perpetuate the continuation of the problem (management) they are the solution to because as long as unions exists, companies feel they need management to offset them, and so the two became co-dependent. In effect problems and solutions tend become a single system.

In his brilliant, classic book The Innovator’s Dilemma, Clay Christensen demonstrates how disruptive technologies almost always arise from the margins of an industry, where they start out as insignificant, or toy, solutions. Honda’s hobbyist electric bicycles were no threat to the big four automobile companies, until electric bikes become motorcycles and motorcycles became small efficient cars. Cheap crumby dot matrix printers were no threat to big offset printing companies until dot matrix became injet printers and injects became the HP Indigo 5000 on-demand printers. In each case, the solutions were marginal, barely working, at first, and therefore ignored. I think what Clay Shirky is pointing out is that many problems, too, are marginal at first, and therefore ignored. Established industries like to focus on established problems.

Shirky made his quote in a recent talk, a bit from his upcoming book Cognitive Surplus. Shirky also referred to a similar idea in a recent blog posting about the ways in which media companies and the media industry are often constitutionally incapable of changing because they are still solving the last problem.

yin-yang1.jpg

In a strong sense we are defined by the problems we are solving. Yin/Yang, problem/solution, both sides form one unit. Because of the Shirky Principle, which says that every entity tends to prolong the problem it is solving, progress sometimes demands that we let go of problems. We can then look to marginal solutions and ask ourselves, what marginal problem is this solving that might be a more appreciated problem later on?




Comments
  • Faré

    Your story is not incorrect, but misses the key notion of incentive: the establishment has not just the knowledge of the old problem, but the incentive to perpetuate it. The budget of an administration grows proportionally with the problem, as opposed to the budget of a free market company that grows proportionally to its success at solving problems.

    Also, unions were never the workers’ friends. They have always existed to parasite workers, and have never benefited any worker.

    • http://www.tumblr.com/blog/his-divine-shadow His Shadow

      “They have always existed to parasite workers, and have never benefited any worker.”

      This is egregiously a-historical histrionic nonsense. Your ignorance of unions does not make them useless.

  • Pierce Presley

    Everything outside of the union example was fine, though I think the thought predates Shirky’s comment by a fair amount. One of the reasons people banged on about “exit strategies” in our recent conflicts is that absent one (or with one that command ignores), the brass tends to see continuing the mission as an end in itself. Non-profits can be a special case, as some are set up with very specific goals in mind that are incorporated into their founding documents.

    All that being said, the union example is bursting at the seams with fail. Unions did not come into existence to solve a problem, but to correct an imbalance of power between labor and management. The difference is crucial; as another commenter pointed out, the lack of unions in certain industries or states or countries has not resulted in a withering away of management or exploitation, quite the contrary. That’s because management exists mainly as a way to maximize profit, and part of its toolkit is getting the most out of workers while giving the least in return. At different times, companies have done this with strike busters, scabs, company stores, child labor and other tactics and practices that are now outlawed.

    It is unrealistic in the extreme to imagine that if unions were to cease operations that management would go away and leave us in some worker’s paradise where they neither had to surrender earnings to be managed nor to be represented.

  • Facetime

    I think the union example applies in the public sector, where public unions, like police and teachers unions, stand in the way of progess, whether it’s reforming drug laws or providing alternative methods of education, such as vouchers or charter schools.

    This is one of the many reasons that public unions don’t make sense. Their “management” (the government and taxpayers) is not concerned with profits but with seeing services provided efficiently. There is no history of exploitation of labor in the public sector as there is in the private sector. Politicians make insane concessions to unions on pensions and benefits, which is why states like CA and NJ are in awful shape. The stronger the public employees unions, the bigger the mess in the state.

  • matt mills

    Shirky’s principle was first stated by Max Weber in his work on bureaucracies, in that their first task after being established is self preservation. Or maybe earlier, but that’s the first I know of. The whole of his work on protestant work ethic and capitalism is actually worth a read, if you’re into how institutions function.

  • Bruce

    I came here from Techdirt, expecting to agree completely with the coining of the “Shirky Principle” which I think is pretty brilliant. That said, I was shocked by how bad an example the union one is (not to mention the affirmations of that example in the comments).

    Not to turn this discussion into a stereotypically lame online flame war over the value of unions, I would just make two points. First, it’s one thing to claim that unions are not valuable now, it’s quite another to claim they have “never” been valuable. That statement displays a lack of historical awareness that is simply mind-boggling. Second, when talking about unions (and workers’ interests generally) try to take step back from your own immediate context. Not everywhere is Silicon Valley and real-life unions today are not Hoffa-era Teamsters.

  • Adam Shostack

    Hi Kevin,

    This reminds me a lot of John Gall’s excellent little book “Systemantics” in which he lays out how systems for solving things tend to perpetuate the problem and make it harder to solve..

  • Suzanne Lainson

    I’m seeing this in less institutionalized examples. I’ve grown weary of endless blogs talking about the death of big media and major record labels. It’s an old story now and I think some writers don’t want to let it go because they have nothing else to say.

    Similarly, the anti-copyright blogs seem to be going over the same ground week after week. I’m inclined to think they need copyright to be the bogeyman so they can set themselves up as the anti-establishment voice.

    The same in politics. Politicians, bloggers, and commentators need someone to complain about, so they don’t necessarily want their adversaries to go away.

  • Lloyd Morgan

    I’m glad David Levinson noticed this, too.

    The Shirky Principle is a modern remake (rehash?) of an Upton Sinclair quote from 1935 (in ‘I, Candidate For Governor: And How I Got Licked’):

    “It is difficult to get a man to understand something, when his salary depends upon his not understanding it!”

    (Bottom of p109, Google books.)

    • http://www.kk.org Kevin Kelly

      @Lloyd Morgan: Thanks, that is my favorite alternative so far. Not quite the same as Shirky’s, unless you alter it to say “It is difficult to get an organization or institution to understand something, when their income depends upon not understanding it!” Even then, I prefer Shirky’s formulation.

  • Richard Careaga

    We’ve all run into people who are so in love with problems that they can never bring themselves to accept a solution. (Often seen in passive-aggressive corporate culture.) It’s a nice turn to generalize this to institutions.

  • Metatone

    Exactly, that’s why in non-union industries we see that both management in general and management exploitation of under capitalised workers is withering away.

  • gmoke

    In the enviro field, I see something similar in that people are so wedded to a problem orientation that they can’t begin to get into a solution orientation.

    All of this may be a symptom of a lack of systems thinking. When you see things as discrete issues rather than as part of a complex system, you keep on running from one discrete issue to another while the system becomes more and more unbalanced and chaotic taking more and more discrete issues into critical failure.

  • John Robb

    This is just a repackaging of Caroll Quigley’s brilliant thesis in:

    The Evolution of Civilizations: An Introduction to Historical Analysis

  • David (Skipper) Everling

    Human bodies can be likened to institutions and Clay Shirky’s Principle, at the very least in an abstract sense if not literally.

    We are complex entities, made up of smaller complex systems, arising from trillions of microscopic cells. Our foremost systematic goal is self-preservation. The human condition, then, is like the institutional problem; ego-centrism.

    I find an especially apt comparison to your last paragraph, on the Shirky Principle, “progress sometimes demands that we let go of problems.” When humans let go of the problem of self-perpetuation it means they are willing to sacrifice their lives for the greater good. We define our progress as a society by the moments when people emerged from the margins and publicly shed concern for the self. Gandhi’s hunger strike, the symbolic protest at Tien’anmen Square, the legendary selfless martyrs of diverse religions all over the world. Historic turning points in the progress of human societies are defined by the acts of those who can set aside the institutionalized problem of maintaining the system; body and mind.

    “Every entity tends to prolong the problem it is solving”. Perhaps we can apply the Shirky Principle toward improving our selves, as well as our institutions.

  • John Carl

    A scary example struck me a while ago about all the money that goes into cancer research. What are they researching, anyway?

    Seems mostly like, how to extract bundles of cash from dying cancer victims. A huge industry!

  • Ivo Quartiroli

    Another way to look at this is that through technology we create problems in order to sell solutions.

  • Sundog

    The Shirky Principle seems all too apt regarding the “drug war” in Mexico. I’ve seen the illicit US-Mexico drug trade valued at $15 to $40 Billion per year. But these enormous sums don’t include drug war spending by both nations via law enforcement, incarceration, military, border control and other institutions. Such spending obviously has consequences for the incentives of these actors.

    Then there’s that other drug war. See Alfred McCoy, “Afghanistan as a Drug War”.

    http://www.tomdispatch.com/post/175225/tomgram%3A_alfred_mccoy%2C_afghanistan_as_a_drug_war__/

    I very much recommend this article, as it takes the whole systems approach mentioned by commenter gmokes above and situates the problem in its historical context. Also, US experience in Afghanistan will probably end up being applied in Mexico, just as Iraq experience is now being applied by personnel from the Naval Postgraduate School in nearby Salinas.

    William S. Lind, “Counter-Insurgency in Salinas?”
    http://www.counterpunch.org/lind11242009.html

  • David Levinson

    This really evokes Upton Sinclair’s Law … “If a man’s paycheck depends on his not understanding something, you can rely upon his not understanding it.” (the wording of the quote varies from source to source). Replace understanding with solving, man with institution and you have Shirky’s Principle.

    • http://www.movablemedia.com aboer

      David, glad you brought this up. Saved me from googling it.

  • John

    I saw a similar concept to Clay’s statement that struck me in a similar way.

    “Once you became an idea’s defender, you had a harder time changing your mind about it.”

    The above quote is from Michael Burry where he is talking about resisting discussing ideas with other people. He likes to work on problems in isolation because of the freedom he gets with his thinking.

    It’s an interesting problem because it is the focus on a problem that allows the refinement. However the trick is to know when enough is enough and it’s time to move on.

    Here is the article about Burry:
    http://www.vanityfair.com/business/features/2010/04/wall-street-excerpt-201004?printable=true

  • joel garreau

    i will never forget working with a very large foundation devoted to health care. the crystal moment came when i realized that if they ever did solve the problems, there would be a lot of people who wouldn’t be able to afford their expensive mortgages.

  • AnthonyC

    I didn’t know this principle had a name.

    I’ve often thought that every inventor’s ultimate goal should be to put himself out of business. That, after all, is the natural endpoint of a development process- a product so reliable, and so effective, that no further work is needed short of occasional replacement or repair. This, of course, wouldn’t sit well with board of directors, however good it might be for society.

  • example

    But over time as capital increased in complexity, unions complexified as well, until unions needed management. The two became one system — union/management. So now the problem with unions is that they are locked into the old framework, the old system. They inadvertently perpetuate the continuation of the problem (management) they are the solution to because as long as unions exists, companies feel they need management to offset them, and so the two became co-dependent. In effect problems and solutions tend become a single system.

    Surely you’re not suggesting that without unions, there would be no management? That literally makes zero sense. One only needs to look at any of the many non-unionized companies, and you’ll find plenty of management.

    The idea that you wouldn’t is just bizarre.

    The purpose of unions isn’t to eliminate management, it’s to allow for collective negotiation. With a union/management framework, workers can collectively bargain. Without a union, each individual worker is pretty much powerless to management’s dictates.

  • Remco Van Der Schaaf

    So, so, so, so many non-profits fixate on this exactly. Unable to let go of the thing they were set up to address, and which is by some distance no longer a problem. Rare is the non-profit that can just let go of what was, and embrace what will be, or even what is…

  • Fred

    Exactly, that’s why in non-union industries we see that both management in general and management exploitation of under capitalised workers is withering away.

    We do?? Hahaha. On what planet?

  • jaycbird

    Has no one here ever seen the classic Alec Guinness film THE MAN IN THE WHITE SUIT?!

    I saw it when I was only 13, and it turned me into a cynic…

  • MAP

    It’s not ethics unless it hurts.

  • Brett Boessen

    Not to pile on, but another thinker who came well before Shirky and was addressing these same principles about institutions is Alasdair McIntyre in After Virtue.

    MacIntyre divides the objectives of institutions into internal and external goods, arguing that institutions arise to sustain particular practices, and to the extent that they pursue these “internal” goods, they thrive.

    However, institutions inevitably pursue external goods as well (as opposed to their internal goods), in an attempt to sustain themselves. When they do this, they turn away from the “problem they are trying to solve”: to perpetuate the practices for which they were established.

    I enjoy Shirky’s work a great deal myself, but to call this the Shirky principle takes the history of thought a bit for granted.

  • Michael E. Rubin

    AnthonyC wrote: “I’ve often thought that every inventor’s ultimate goal should be to put himself out of business.”

    Totally agreed.

    I used to work as the head of social media for a media agency. When going on client pitches, we would talk about the natural cycle of social media adoption. It started with education and then implementation.

    The kicker? If seen through its natural evolution, the cycle would end with us being “out of a job.” That is, the client would be self-reliant enough in social media that they would be able to take the operation in-house.

    In essence, doing our job meant putting ourselves out of business.

    We got a LOT of double-takes with that. But I think clients — even the ones who didn’t give us their business — really appreciated the honesty. It must have been refreshing to hear an agency say this. I know if I was in their shoes, I would feel that way.

  • djeak

    We do?? Hahaha. On what planet?

    Fred: sarcasm.

  • Wayne

    Military arms manufacturers are the “hidden hand” behind many of the world’s wars. Weapons manufacturers WANT war because then they can sell weapons to BOTH sides. They orchestrate covert operations behind the scenes to cause and perpetuate war.

    Genuine world peace would be a disaster for these companies because it would eliminate their reason for existing.

  • baji kulkarni

    Does it remind you of Jim Carse’s ‘ Infinite Games’ ?

    • http://www.kk.org Kevin Kelly

      @Baji: “Does it remind you of Jim Carse’s ‘ Infinite Games’”

      Yes, it does.

  • Steve Lewis

    In medicine its called Munchausen syndrome by proxy, or manufactured or induced illness. It was first diagnosed by a pediatrician, who observed that some mothers intentionally harm their children in order to get attention and praise.

    Munchausen has corrolaries in the workplace! Harvard Business Review recently came out with the article Munchausen at Work. Employees create fictitious organizational problems in order to solve them and receive praise for it! See http://bexhuff.com/2007/12/munchausen-syndrome-in-washington-and-on-wall-street

  • Roland Hjerppe

    Stafford Beer made a similar observation in his “Managing Modern Complexity” about “esoteric boxes” which, when. strung together, form larger organizations. These esoteric boxes are not closed systems but they are more often than not ultrastable, resistant to. change, and are “designed for survival.”

    In the background we have Le Chatelier’s principle “The principle is named after Henry Louis Le Chatelier and Karl Ferdinand Braun who discovered it independently. It can be summarized as:

    If a chemical system at equilibrium experiences a change in concentration, temperature, volume, or partial pressure, then the equilibrium shifts to counteract the imposed change and a new equilibrium is established.”
    from Wikipedia

  • George Howard

    I greatly admire Mr. Shirky’s work, but you realize, of course, that this “rule” is sort of the definition of a bureaucrat?

    best,

    George

  • Reader

    It’s kind of like the economy right now – even with an absurdly inefficient system, it doesn’t take anywhere near the number of people in the US workforce to run a country with 300 million people in it. The goal is to introduce inefficiency into the system in order to provide jobs.

  • Britt Blaser

    Early in the Iraq war, some general observed that we’d not make progress until we stopped admiring the problem. I love that phrase.

  • Scott

    This is part of the problem with stage gate analysis, as well. Instead of becoming go/no-go decision points, they become walls that a group MUST get over, because they don’t want their project to go away. Someone without a project becomes ripe for redeployment. We can make our cultures more innovative when we allow both the destruction and creation of projects. This is how capitalism is supposed to work; if companies are kept afloat artificially, then those resources: capital, human, intellect, are not freed up for new projects.

  • Steve Witham

    Suzanne Lainson proposes that politicians don’t necessarily want their adversaries to go away.

    In one sense that’s clearly true: a candidate needs to contrast himself to something. But you can imagine a candidate saying, well, things are in pretty good hands now, I’m no longer necessary.

    But what Kevin Kelly suggested about unions and management works for politicians and especially major parties or alignments like conservative and liberal: each opposes the other, not because they literally need an opposition, but because the opposition is trying to do those bad things.

    This self-perpetuating polarization is a feature of democracy, designed to find ways to muddle through differences. If there were no differences, resolution wouldn’t be necessary. The funny thing is that democracy’s solution is to treat opposition as the basic reality. It requires there to be opposing candidates to choose between, and then elected representatives eventually vote on hotly contested legislation.

    It’s not that yes/no decisions are unnatural, but that rather than lots of decisions all over the place, democracy funnels things into a loggerheads model: a small number of disagreeing groups up in arms over a small number of decisions. And working on that model perpetuates the actual loggerheads situation.\

    Is there a split between organizations set up assuming a problem will be permanent, vs. orgs that assume it won’t be? If so, are the ones that assume permanence more likely to preserve the problem?

  • Ambrose

    > Unions, or example.

    Should probably be “Unions, for example”?

  • martin

    this is, by the way, a brilliant condensation of the theory of social systems:

    systems define themselves through drawing borders against, and managing interaction with, “the outside”. this my have been a “solution” once.

    but over time each system tends to become auto-reflexive: primarily reacting to itself and the inner-systemic neighborhood, preserving its own structures.

    nearly all institutions of the whole “Western World” seem to have reached an overly complex state of self-development.

  • Peter Bachman

    The 14th book of Bokonon entitled “what can a thoughtful man hope for mankind on earth, given the experience of the past million years?” seems to apply here.

  • Faré

    Your story is not incorrect, but misses the key notion of incentive: the establishment has not just the knowledge of the old problem, but the incentive to perpetuate it. The budget of an administration grows proportionally with the problem, as opposed to the budget of a free market company that grows proportionally to its success at solving problems.

    Also, unions were never the workers’ friends. They have always existed to parasite workers, and have never benefited any worker.

  • Guest

    Exactly the problem with the Microfinance industry too. It started a vicious circle which the Institutions never wanted to un-loop. Perhaps, making leaders redundant (not irrelevant) in the long run is the plausible way out.

  • http://www.technostall.com/ Chankey Pathak

    Good observation, nice post!

  • http://www.thepomoblog.com Terry Heaton

    This principle is also true of social movements.

  • Chris Graves

    The union example above doesn’t hold as a principle – although doubtless there will be examples which do support your case.  In general unions are necessary in the current environment because there is still a tendency for some managers to cut corners, become lazy and prioritize their own needs above that of the company or their reports.  In such an environment Unions provide a useful check and balance.

  • Kidd Redd

    A great thesis undermined by the union example. I would offer the example of traditional ad agencies, who have a status quo to maintain in old media, because that’s where most of their revenues derive. 

  • http://twitter.com/jconti John Conti

    Er, um. There is no doubt this is a reality, often hard to see. However we have some common sense expressions that may indicate this is not so original: “hammer just lookin for a nail”, or maybe “generals need wars”?

    I think it may be over-reaching to assume causality here. Not that this certainly isn’t true in many cases, however, I often think disruptive technology is more about discovering hidden demand than any prescient engineering (Disruptive Technology or Hidden Demand?). For every electric bike there have been other, less successful, electric flyswatters and page turners.

    Great stuff as always.

    Thank you,
    John

  • John Joyce

    The tense of the verb to solve is the key to seeing why the Shirky principle is good only as far as it goes, i.e. the infinitive “solving” rather than the past tense “solved”. Thus we see that what is being addressed is an attempt to solve a problem which inherently admits of alternatives one of which may work and eradicate the problem. Up to that point of solution the “entity” is the de facto standard of care.

  • http://www.facebook.com/people/Larry-Kachimba/100000892449329 Larry Kachimba

    Professional political activists who all work on one issue or another, without ever coming close to solving them – as indeed over the same several decades since this model of politics arose, corrupt politics undermined democracy and made all policy inexorably more dysfunctional. http://www.counterpunch.org/2013/03/15/the-progressive-movement-is-a-pr-front-for-rich-democrats/

  • oliver

    Hey, Shirky had found out this one principle?
    This is one of many principles, which John Gall mentioned already in 1978.
    35 years later, and from the about 30 principles tha gAll found out, Shirky found out one… and should thatone be named after Shirky?

    Peopls should read some of the older books,they have more wisdom in each, than there is in one year of TED talks (even I enjoy looking them).

    About John Galls findings:

    http://www.laetusinpraesens.org/docs/systfail.php

  • Gaston

    A ham fisted semi-reworking of the Hegelian dialetic and completely naive overview of the history and aims of the trade union movement in one post.