Let’s posit for a moment that technological progress, in general terms, leads to a sort of density of knowledge such that one need not fully grasp underlying processes to make effective use of them.  This is sort of a holy grail in software development, the creation of standalone “black box” modules whose inputs and outputs are the only pieces of information of any relevance to the programmer.  That we haven’t reached this state quite yet is telling, but that doesn’t mean it’s impossible to reach.  

Presently I am examining past developments and have come to the conclusion that human technological progress really is about building successive layers of black boxes.  Occasionally it becomes necessary to open them up to check their assumptions, and we have people for that.  Mostly, though, we build things like economies and markets around the inputs and outputs of these black boxes.  

Take agriculture, for example.  Most of us in the West know very little about how the food in our supermarkets end up there.  We usually have some vague idea about farming, which evokes some pretty strong traditional imagery, but unless we buy directly from farmers we know, most of these images are charmingly antique when applied to US-based agricultural production, or hopelessly naive when applied to, say, Mexico.  I am suggesting that modern agriculture, as practiced by the US and plenty of other countries, is a black box, a process we know and need know nothing about beyond its outputs (someone else looks after the inputs).  And yet we manage to harness that output in a number of ways.  Some of these ways are black boxes themselves.  

Arguably we should know what’s in the boxes.  But sometimes it’s easier just to ignore the truly uncomfortable questions and pretend that the output is really all that is important.  After all, while we seem to understand what goes into the manufacture of hot dogs, we nevertheless hate to be reminded of the fact, nor do we ever feel inclined to watch the process.  And so it is with agriculture.  We are occasionally shocked to learn of the appalling conditions meat producers have inflicted on livestock in the name of keeping food costs low, but we rarely act on that outrage.  Sometimes we lament the lot of the field worker, often an undocumented immigrant of some sort, but only until we remember that the workers, being “illegal” immigrants, are somehow depriving Americans of their jobs.  No, far better that the whole thing should remain a mysterious process, lest we feel guilt for the child workers and trafficked persons and indentured peasants whose work ensures that we don’t pay more than $3 per pound of chicken.  

These processes seem to be everywhere.  Granted, they aren’t all composed of every social ill we can stuff in.  Some of them are quite opaque, esoteric, or arcane.  But the effect is a social order that is increasing in complexity and has far surpassed any person’s ability to learn every necessary aspect of it.  I suppose one could argue that society has been pretty complex for a while, though I’m certain we are entering a phase in which the number of people who are required to understand even significant portions of it will fall.  

But there are things we can’t predict, at least not well.  If we manage to build enough of these black boxes, we get something entirely new, a substrate that emerges from the components of the previous layer, but is more than the sum of its parts.  What this looks like is anyone’s guess.  

That doesn’t mean we couldn’t try, though.  What we need is a sort of conceptual framework for thinking about emerging complexity.  The first step is to look at other instances of emergence.  We have some of these.  For instance, everything about the formation of planets and stars is representative of emergence.  Chaos, what we assume was before the present order, is a different level of complexity than planets and stars.  The formation of solid bodies, in fact, while perhaps predictable in some ways, isn’t possible if some other conditions aren’t met.  Thus it would be impossible for an outside observer, assuming such a thing could exist, to predict that the swirls and streams of particles and energy racing outward from the Big Bang would eventually coalesce, not just into objects, but would also arrange themselves in self-similar ways when they did.  But this isn’t something we control.  We need to look at thing we control.

Another example, and one that we do control, is the Internet.  Given the collection of technologies represented by previous generations of computing, I doubt anyone could really have predicted the Internet.  Or, if they did, nobody could have predicted the things that in turn run over the Internet, such as World of Warcraft or Facebook.  This is an example of emergence that is useful for us, because it most definitely models the black box theory.  Using various Internet and web resources doesn’t require one to understand precisely how packet switched networks operate, and yet they underlie the connection.  Nor are we required to understand how random access memory, magnetic disk storage, or silicon-based microprocessors work.  (One could even argue that the modern computer is emergent from the parts, since one might not be able to predict, for instance, Windows XP, by looking at just the components.)  

The next step is to figure out what these have in common.  But that means also that we have to establish, in fairly expressive detail, the chains of causation that would have allowed us to predict the next step.  Reverse engineering, in effect.

While difficult and tedious, reverse engineering is nothing compared to the next step, though, which is to  survey all of the sufficiently developed and nascent technologies currently in existence.  This is broad.  We would, of course, include the existing mainstays, such as agriculture, finance, and the like, but we would need to be especially keen to suss out anything remotely promising without falling into the pseudoscience or magic trap.  In fact, this may be quite a difficult undertaking.

Only once we do that could we begin speculating on what emerges from this soup.  To date, most predictions tend to run on linear augmentations of existing technologies, with the occasional combinatorial gymnastics to consider new paradigms.  And to date, most predictions have been wrong in ways that are almost humorous in hindsight.  I’m not interested in individual towers being built on the modern substrate of human technological advancement.  I am interested in what the next substrate looks like.  Is that even possible?  Let’s discuss it and see.