According to Farmer, there are two kinds of complexity: inherent
and apparent. Inherent complexity is the "true" complexity of chaotic
systems. It leads to dark unpredictability. The other kind of complexity
is the flip side of chaos -- apparent complexity obscuring exploitable
Farmer draws a square in the air. Going up the square increases apparent
complexity; going across the square increases inherent complexity.
"Physics normally works down here," Farmer says, pointing to the bottom
corner of low complexity for both sorts, home of the easy problems. "Out
there," pointing to the opposite upper corner, "it's all hard. But we
are now sliding up to here, where it gets interesting -- where the apparent
complexity is high, but the true complexity is still low. Up here
complex problems have something in them you can predict. And those are
exactly the ones we are looking for in the stock market."
With crude computer tools that take advantage of the flip side of
chaos, the Prediction Company hopes to knock off the easy problems in
"We are using every method we can find," says partner Norman Packard, a
former Chaos Cabalist. The idea is to throw proven pattern-finding
strategies of any stripe at the data and "keep pounding on them" to
optimize the algorithms. Find the merest hint of a pattern, and then
exploit the daylights out of it. The mindset here is that of a
gambler's: any advantage is an advantage.
Farmer and Packard's motivating faith that chaos possesses a flip side
firm enough to bank on is based on their own experience. Nothing
overcomes doubts like the tangible money they won from their Las Vegas
roulette wheel experiments. It seems dumb not to take advantage of these
patterns. As the chronicler of their high-rolling adventure exclaims in
the book The Eudaemonic Pie, "Why would anyone play roulette without
wearing a computer in his shoe?"
In addition to experience, Farmer and Packard place a lot of faith in
the well-respected theories they invented during their years in chaos
research. Now they are testing their wildest, most controversial theory
yet. They believe, against the unbelief of most economists, that certain
regions of otherwise complicated phenomenon can be predicted accurately.
Packard calls these areas "pockets of predictability" or "local
predictability." In other words, the distribution of unpredictability is
not uniform throughout systems. Most of the time, most of a complex
system may not be forecastable, but some small part of it may be for
short times. In hindsight, Packard believes local predictability is what
allowed the Santa Cruz Cabal to make money forecasting the approximate
path of a roulette ball.
If there are pockets of predictability, they will surely be buried under
a haystack of gross unpredictability. The signal of local predictability
can be masked by a swirling mess of noise from a thousand other
variables. The Prediction Company's six rocket scientists use a mixture
of old and new, hi-tech and low-tech search techniques to scan this
combinatorial haystack. Their software examines the mathematically
high-dimensional space of financial data and searches for local
regions -- any local region -- that might match low-dimensional patterns they
can predict. They search the financial cosmos for hints of order, any
They do this in real time, or what might be called hyperreal time. Just
as the simulated bouncing roulette ball in the shoe-computer comes to
rest before the real ball does, the Prediction Company's simulated
financial patterns are played out faster than they happen on Wall
Street. They reenact a simplified portion of the stock market in a
computer. When they detect the beginnings of a wave of unfolding local
order, they simulate it faster than real life and then bet on where they
think the wave will approximately end.
David Berreby, writing in the March 1993 Discover, puts the search for
pockets of predictability in terms of a lovely metaphor: "Looking at
market chaos is like looking at a raging white-water river filled with
wildly tossing waves and unpredictably swirling eddies. But suddenly, in
one part of the river, you spot a familiar swirl of current, and for the
next five or ten seconds you know the direction the water will move in
that section of the river."
Sure, you can't predict where the water will go a half-mile downstream,
but for five seconds -- or five hours on Wall Street -- you can predict the
unfolding show. That's all you really need to be useful (or rich). Find
any pattern and exploit it. The Prediction Company's algorithms grab a
fleeting bit of order and exploit this ephemeral archetype to make
money. Farmer and Packard emphasize that while economists are obliged by
their profession to unearth the cause of such patterns, gamblers are not
bound so. The exact reason why a pattern forms is not important for the
Prediction Company's purposes. In inductive models -- the kind the
Prediction Company constructs -- the abstracted causes of events are not
needed, just as they aren't needed for an outfielder's internalized
ballistic notions, or for a dog to catch a tossed stick.
Rather than worry about the dim relationships between causes and effects
in these massively swarmy systems crowded with circular causality,
Farmer says, "The key question to ask in beating the stock market is,
what patterns should you pay attention to?" Which ones disguise order?
Learning to recognize order, not causes, is the key.
Before a model is used to bet with, Farmer and Packard test it with
backcasting. In backcasting techniques (commonly used by professional
futurists) a model is built withholding the most recent data from the
human managing the model. Once the system finds order in past data, say
from the 1980s, it is fed the record of the last several years. If it
can accurately predict the 1993 outcome, based on what it found in the
1980s, then the pattern seeker has won its wings. Farmer: "The system
makes twenty models. We run them each through a sieve of diagnostic
statistics. Then the six of us will get together to select the one to
run live." Each round of model-building may take days on the Company's
computers. But once local order is detected, a prediction based on it
can be spun in milliseconds.
For the final step -- running it live with bundles of real money in its
fists -- one of the Ph.D.'s still has to hit the "enter" button. This act
thrusts the algorithm into the big-league world of very fast,
mind-boggling big bucks. Cut loose from theory, running on automatic,
the fleshed out algorithm can only hear the murmurs of its creators:
"Trade, sucker, trade!"
"If we can earn 5 percent better than what the market does, then our
investors will make money," Packard says. Packard clarifies that number
by explaining that they can predict 55 percent of market moves, that is,
5 percent more than by random guessing, but that when they do guess
right their result can be 200 percent better. The fat-cat Wall Street
financial backers who invest in the Prediction Company (currently
O'Connor & Associates) get exclusive use of the algorithms in exchange
for payments according to the performance of the predictions. "We have
competitors," Packard states with a smile. "I know of four other
companies with the same thing in mind" -- capturing patterns in chaos with
nonlinear dynamics and predicting from them. "Two of them are up and
going. Some involve friends."
One competitor trading real money is Citibank. Since 1990, British
mathematician Andrew Colin has been evolving trading algorithms. His
forecasting program randomly generates several hundred hypotheses of
which parameters influence currency data, and then tests the hundred
against the last five years of data. The most likely influences are sent
to a computer neural net which juggles the weight of each influence to
better fit the data, rewarding the best combinations in order to produce
better guesses. The neural net system keeps feeding the results back in
so that the system can hone its guess in a type of learning. When a
model fits the past data, it is sent out into the future. In 1992 the
Economist said, "After two years of experiments, Dr. Colin reckons his
computer can make returns of 25 percent a year on its notional dealing
capital....That is several times more than most human traders hope to
make." Midland Bank in London has eight rocket scientists working on
prediction machinery. In their scheme, computers breed algorithms.
However, just as at the Prediction Company, humans evaluate them before
"hitting the return button." They were trading real money by late
A question investors like to ask Farmer is how can he prove you can make
money in markets with the advantage of only a small bit of information.
As an "existence proof" Farmer points to the people such as George Soros
earning millions year after year trading currencies and whatnot on Wall
Street. Successful traders, sniffs Farmer "are pooh-poohed by the
academics as being extremely lucky -- but the evidence goes the other way."
Human traders unconsciously learn how to spot patterns of local
predictability streaking through the ocean of random data. The traders
make millions of dollars because they detect patterns (which they cannot
articulate), then make an internal model (which they are unconscious
of), in order to make predictions (which they are rewarded or punished
for, sharpening the feedback loop). They have no more idea of what their
model or theory is than of how they catch fly balls. They just do. Yet
both kinds of models were empirically constructed in the same inductive
Ptolemaic way. And that's how the Prediction Company employs computers
to build models of high-flying stocks -- from the data up.
Says Farmer, "If we are successful on a broad basis in what we are
doing, it will demonstrate that machines are better forecasters than
people, and that algorithms are better economists than Milton Friedman.
Already, traders are hesitant about this stuff. They feel threatened by
The hard part is keeping it simple. Says Farmer, "The more complex the
problem is, the simpler the models that you end up having to use. It's
easy to fit the data perfectly, but if you do that you invariably end up
just fitting to the flukes. The key is to generalize."
Prediction machinery is ultimately theory-making machinery -- devices for
generating abstractions and generalizations. Prediction machinery chews
on the mess of seemingly random chicken-scratched data produced by
complex and living things. If there is a sufficiently large stream of
data over time, the device can discern a small bit of pattern. Slowly
the technology shapes an internal ad-hoc model of how the data might be
produced. The apparatus shuns "overfitting" the pattern on specific data
and leans to the fuzzy fit of a somewhat imprecise generalization. Once
it has a general fit -- a theory -- it can make a prediction. In fact
prediction is the whole point of theories. "Prediction is the most
useful, the most tangible and, in many respects, the most important
consequence of having a scientific theory," Farmer declares.
Manufacturing a theory is a creative act that human minds excel in,
although, ironically we have no theory of how we do it. Farmer calls
this mysterious general-pattern-finding ability "intuition." It's the
exact technology "lucky" Wall Street traders use.
Prediction machinery is found in biology, too. As David Liddle, the
director of a hi-tech think tank called Interval, says, "Dogs don't do
math," yet dogs can be trained to predictively calculate the path of a
Frisbee and catch it precisely. Intelligence and smartness in general is
fundamentally prediction machinery. In the same way, all adaptation and
evolution are milder and more thinly spread apparatus for anticipation
Farmer confessed to a private gathering of business CEOs, "Predicting
markets is not my long-term goal. Frankly, I'm the kind of guy who has a
hard time opening to the financial page of the Wall Street Journal." For
an unrepentant ex-hippie, that's no surprise. Farmer sees himself
working for five years on the problem of predicting the stock market,
scoring big time, and then moving on to more interesting problems -- such
as real artificial life, artificial evolution, and artificial
intelligence. Financial forecasting, like roulette, is just another hard
problem. "We are interested in this because our dream is to produce
prediction machinery that will allow us to predict lots of different
things" -- weather, global climate, epidemics -- "anything generating a lot of
data we don't understand well."
"Ultimately," says Farmer, "we hope to imbue computers with a crude form
By late 1993, Farmer and Company publicly reported success in predicting
markets with "computerized intuition" while trading real money. Their
agreement with their investors prohibits them from talking about
specific performance, as much as Farmer is dying to. He did say, though,
that in a few years they should have enough data to prove "by scientific
standards" that their trading success is not a statistical fluke: "We
really have found statistically significant patterns in financial data.
There really are pockets of predictability out there."