Ask HN: Why isn't all stock trading done by algorithm?

[Peradine]

I can't understand how a human, even with heaps of intelligence and education, is better at predicting which stocks will go up/down than an algorithm that can look at all the data in the entire market. What skill do stock traders use that can't be replaced by computation?

[greenpizza13]

Computers trade most effectively with technical indicators. These are things like price history, volume, etc. These trades are very effective in the short term, where trades can happen in terms of seconds or milliseconds. This is where computers excel and humans fall short. This sort of trading is based mostly on breadth. If you look up the fundamental law of active portfolio management, you'll see that breadth is less effective (exponentially less so) than skill is.

In the case of the long term, however, trading is done with fundamental indicators. These things can be more or less intangible and have to do with market events, people, and other indicators of company value that are hard to translate into math. Using fundamental indicators for portfolio management is what humans are better at, and these pay off in the long term (see Warren Buffet). These trades are done with skill, which, as I stated earlier, is exponentially more effective at creating gains than breadth.

In short, it takes a huge amount of breath to get the gains required by a relatively small amount of skill. Computers are better by far at breadth, while humans are better (for now) at skill. This, I think, is why humans still trade.

[wbsgrepit]

Not to mention the spectrum between intuition/innuendo and flat out insider trading knowledge that is cornerstone to the market -- trading algorithms don't do well against this human factor.

[spott]

Information not in the market.

Stock traders trade on rumor, fact and everything in between. They (can) look at who the company is run by, and how much they trust him, and look at the people in upper management. They can look at and understand news.

Part of the difficulty of the stock market is that it isn't a closed system: people make decisions to sell stocks based on the fact that they are poor this month. Until computers can understand and process all the external data, people will need to be in the trading loop.

The big exception to this is the trading the noise: high frequency traders trade against each other, closing the spread in bid/ask prices. However, they are essentially trading against other computers at that timescale, which makes them pretty amenable to algorithms.

[talmand]

You reminded me of this scene from Trading Places:

Randolph Duke: Exactly why do you think the price of pork bellies is going to keep going down, William?

Billy Ray Valentine: Okay, pork belly prices have been dropping all morning, which means that everybody is waiting for it to hit rock bottom, so they can buy low. Which means that the people who own the pork belly contracts are saying, "Hey, we're losing all our damn money, and Christmas is around the corner, and I ain't gonna have no money to buy my son the G.I. Joe with the kung-fu grip! And my wife ain't gonna f... my wife ain't gonna make love to me if I got no money!" So they're panicking right now, they're screaming "SELL! SELL!" to get out before the price keeps dropping. They're panicking out there right now, I can feel it.

Randolph Duke: [on the ticker machine, the price keeps dropping] He's right, Mortimer! My God, look at it!

[hsk]

You're overestimating the effectiveness of computers in taking in data and finding patterns. If you throw data naively at an algorithm, you'll get garbage. It's especially difficult to make sense out of trading data because of the sheer size and percentage of noise.

For any given trading strategy, a ton of thought, testing, and domain knowledge goes into creating the algorithm. It is not a black box that writes itself.

That said, computers are far more effective at certain tasks, especially latency sensitive simple calculations, just as calculators are far better at doing arithmetic.

[ar7hur]

I agree. And the OP is also underestimating the effectiveness of insider trading.

[chollida1]

Well the truth is that for some firms it is for about 99.9% of all trades.

Renaissance Technologies founder Jim Simmons is famous for saying they didn't override the algorithm.

In practice most HFT firms do a mix of both. The algos will do the vast bulk of the trading but you have human traders monitoring algos to do clean ups for cases where the algo gets "stuck". What defines "stuck" really depends on the sophistication of the algo and the firm itself.

Some algo's, such as internalizers's for crossing bought flow are so simple that there probably doesn't need to be much over site at all.

Market making is very similar, with the exception of a flash crash where they might pull out, market making algos should just run themselves.

[SEJeff]

Well for starters, someone has to write the algorithms. Not everyone has vast amounts of computing power to find the right inputs for the right algorithms or genius level quants to write the math which coders turn into strategies. There are so many small niches in the market that a human can still make a reasonable living if they find an area not traded by others aka "making a market". Now when it comes to competing with the machines, you pretty much nailed it.

Source: I work in electronic trading and have the past ~7+ years.

[Coding_Cat]

Would you happen to know some good reads (either on- or off-line) for someone interested in computer trading with a technical background (currently doing a Msc. In Computational Science != Computer Science)?

I have been interested in doing a bit of electronic trading as a hobby (and hopefully a small profit, but I know better than to hope for now) and possibly a back-up career path. But while I've found enough information about trading in general I found it rather hard to find anything dealing with automated trading specifically beyond very simple models.

[mjwhansen]

There are a lot of different schools of thought about approaching the stock market. One of them is the "chart" approach, where you (or an algorithm) try to discern the future movements of a company's stock price based on the past performance. Sometimes this works. Mostly, though, it doesn't.

There's also a big difference between "trading" and "investing." Trading is what you've described -- buying shares in the morning and hoping they'll go up in the afternoon so you can sell them later. Investing is buying shares of the company to become a part owner and hold them for years or decades, not days.

If you looked at NFLX's chart in 2012, you could "discern" that the share price would continue to hover around that price, maybe go up a little, maybe go down a little. And you could have bought it in September 2012 for $8 a share and sold for a nice $1 profit in October 2012 for $9 a share (split adjusted). But what the chart wouldn't have told you -- and would never have been able to tell you -- is that it would skyrocket in 2013 and up to its current split-adjusted price of $110 a share. The thing is, the chart never would have told you about this. And even a "pure" numerical analysis like could be done by a computer -- P/E ratios, cash flows, etc -- would not have predicted that growth. You could do DCFs all day every day in 2012 and never predict Netflix's rise. There are a lot of things that go into a company's rise that aren't numerical, like the quality of management, market moat, market growth, etc. And, of course, you had to buy it, and hold it for years, in order to see that return.

(In the interests of full disclosure, I should probably note that I'm a bit biased in this. I work for The Motley Fool, which advocates for long-term buy and hold investing, and produce a podcast for growth investors called Rule Breaker Investing.)

[kspaans]

Good point about trading versus investing. When index investing advocates say that it's hard to beat the market over the long term, I've always wondered what the difference is with traders. Do their returns beat the S&P500? Does an individual who trades forex as their day job do better than stuffing their money in an index fund for 40 years? Lately I've learned that larger traders can provide value apart from buy&hold, e.g. market making, finding the right price for something, shorting bubbles.

[zeckalpha]

Computerized trading uses more than just historical price for input.

[leroy_masochist]

Former investment banker here. This is a question I had when I hit the desk, and it's an important question.

The short answer to your question is: because most of the volume traded on exchanges is large blocks of stock being bought and sold by institutional investors, and you need humans to make these deals happen.

Longer explanation as follows:

On the trading floor [0], you have two groups of people: the sales team and the traders.

The sales team gets paid when they make markets, i.e., connect buyers and sellers. Specifically, the financial institution takes a fee that's a very small % of the overall transaction volume, and some of that goes into the sales team's bonus pool. The more stock trades flow through the firm (specifically, their business unit), the more they get paid.

The traders, on the other hand, gets paid to do two things, which are really the same thing: a) not put too much of the firm's capital at risk and b) set the firm up to make money by buying securities low and selling them high. Every trader has a "P&L" (profit & loss) number, which is the total amount of money they've made or lost for the firm since the start of the fiscal year. They get paid a bonus based on this number. They tend to know exactly what their number is at any given time.

So, there is actually a lot of tension on the trading floor between the sales team and the traders, because the sales team wants a lot of volume to go through their business unit, and any given trader wants to maximize her P&L.

Real world example might be: the sales dude gets a call from a hedge fund saying, "we want to sell $100mm of our shares in Alphabet at $720". He then shouts over to the trader (who sits close to him) to tell her about the call and she thinks for a couple seconds [1] and then says, "you need to make a market for 80mm of those shares at that price, I'm only taking 20mm."

In other words, the trader is saying that she'll only tie up $20mm of the firm's capital on this particular trade [2]. The sales person might come back and say, "c'mon, they took that $50mm of Microsoft stock you were trying to get rid of last quarter, we as a firm owe them a favor" to which the trader might respond, "OK, we'll take $30mm tops". So then the sales person will get on the phone and start calling everyone (other mutual / hedge funds, pension funds, etc) who might be interested in buying Alphabet at $720. Maybe the sales person makes it happen; maybe they don't. In any case, they need to figure out whether they can get 70 million dollars worth Alphabet stock pre-sold to other people in the market at $720 before they get back to the hedge fund trying to sell it with a response as to whether they can make the trade.

All of this involves MASSIVE HUMAN FACTORS. I'm sure we will one day be able to train AI to work through various constructs of "we owe them a favor" but right now you still need humans to get big trades like this done. And again, big trades like this constitute the majority of the overall volume in the market. So, that's why trades don't run entirely on algorithms...yet.

[0] I was in banking, not S&T, but have a decent understanding of how this works.

[1] Being able to make decisions of this magnitude in a couple of seconds (and have them be good ones) is one of the two skills you need to have as a trader; the other one is not letting the outcome of the last trade (good or bad) affect your thinking on the next trade.

[2] There is potential for both upside and downside in a decision like this; if the stock appreciates, the firm can profit by selling the stock at a higher price than it paid, but the reverse is also true. This is also an example of why "proprietary trading" is such a blurry line. In order to make markets for big trades, firms usually have to put their own capital at risk, even for a few minutes. At what point are they trading for their own profit vs. temporarily assuming risk in order to broker a deal between two counterparties? Go read Matt Levine's archived columns at Bloomberg if you find stuff like this interesting.

[kcbanner]

So once the pre-sell is agreed on are orders made on the open market? Or do these deals take place outside of that?

[harmegido]

This isn't trading though - it's playing middle man. You forgot to mention that the trader offering to buy $20mm of stock is sitting there looking at a screen telling him that real market makers are posting bids of $721 on the Nasdaq, etc. All the trader has to do is take the flow and offload the risk without moving the market such that he takes a loss.

I'm sure it's largely regulation that prevents these hedge funds from going straight to the market themselves and bypassing these "traders".

It's the same thing as exchanging currency in an airport. You see these big signs with large disparities between "Buying at" and "Selling at" for each currency pair. Very easy to make money like that as a currency swapper when you have the nearly-guaranteed hedge that is the massive FX spot market.

[mark-r]

"buying securities high and selling them low" - I assume that's a typo? Usually it's the other way around, unless you're trying to make a joke.

[ComteDeLaFere]

I learned more from this post than from all of the finance classes I took in school combined.

[anthony_barker]

What is an algorithm in your definition?

If it includes where and how to place the trades (a smart order router) I would say 99% of trades are managed by an Algo. Also there are the "dumb" VWAP, POV, TWAP algos which represent the bulk of "smart" buyside money as internally most firms use vwap as benchmark.

The bulk of retail orders in the US are on the other side of an algo from citadel, knight or citi. And often buying 100 shares of MSFT at market only really needs a decent SOR to provide BestEx.

Block trading still often gets fed to VWAP algos unless the stock is illiquid.

Finally the most interesting execution algos (implementation shortfall algos) are hard to explain and only statistically outperform.

If you are talking about actual investing - the first question is which asset class in which to park your money. If you can get an algo to do that (like Renaissance) you will be rich.

[brohee]

Because ultimately, stock prices move not based on "the market" but on things happening in the real world?

How does an algorithm interpret e.g. an Apple Keynote? By the time Twitter sentiment analysis (if such thing is really useful) gives results, an human trader already took a position...

[rezistik]

Humans are better at recognizing satire as well, April Fools jokes have famously caused some algorithms to react poorly.

[zrail]

Why do I need an algorithm to find a good business, buy shares, and sit on them forever?

[Mikeb85]

There are limits to algorithms. Stock prices are affected by human behaviour, which can't always be predicted.

Not to mention the various macro variables, like wars, weather, crime, etc... Plus, what time frame should the algorithms trade on? They're very good for predicting the very short term, I haven't seen much evidence that they're good for predicting longer time spans.

While algos eat up arbitrage and electronic brokers replace human ones, humans are still very good at other forms of trading... Not to mention, a large part of market activity isn't even trading - it's long term investing and collecting dividends.

[jackgavigan]

It's impossible to create an algorithm that encompasses all the factors that contribute to a company's stock price (or, perhaps more accurately, all the factors that a given investor believes will affect the stock price). There's also an implementation gap between the model a stock analyst can create in Excel and its implementation as an algorithm. The former is within the reach of far more people than the latter.

Finally, don't forget that somebody has to design and write the algorithms.

[chad_strategic]

This is a pretty good question... Not sure if it will be solved here.

I used to be a value investor around 2000-2008. A value investor would be something like Buffet or Peter Lynch. However I did make a lot money in Sept. 08 because I determined the market was over valued.

What I didn't forsee, was how much the dollars the Federal reserve would print and inflate the economy.

Regardless, after that I built my own algorithm, because I no longer believe in the structure of the market. I would rater trust numbers. Meaning there are to many analyst pumping stocks, federal reserve, insider trading, spoofing trades, ETFs, deratives, and financial warfare it's hard to make a true value investment. Yes, I have read the buffet / Grahm books, but those are over ~60 old.

I think it is Virtu (electronic trading / hedge fund) that hasn't had a day where they lost money since early 2009? I know Goldman and JP Morgan 90% of the time trade every day for a profit. So a lot of the market is already trading electronically. I think zerohedge.com has estimated the 70% of the market trades on electronically and that article was few years ago.

It's funny, because I have devised methods using social media / programming to manipulate the price of stocks. If I can think of ways to do that I'm sure sure Wall St. already is doing it.

Anyways here my algorithm it tracks over 500 stocks: http://www.strategic-options.com/trade/

[nether]

Your site charges $600/year. That's 6% of a $10,000 portfolio. Do you think it can make a 6% gain in a year after taxes and inflation to cover its costs for that size portfolio? 6% gain is about the long term average of the S&P500, after inflation, so you're proposing doubling the market gain.

[dragontamer]

Skill? I'm putting up trades as an investment. My money can't just sit around in a bank account forever, it needs to be invested to make actual gains.

All the hedge-fund managers that claim to have algorithmic trading have extremely high expense ratios. So its cheaper if I made trades myself.

I mean, its only $7 to execute a trade off of Scottrade or E-Trade. While buying a mutual fund with algorithmic trading will cost you like 50 to 200 basis points per year.

Yeah, its cheaper to trade in the raw or to just buy SPY or Vanguard funds (which are passively invested without algorithms)

[simo7]

The amount of trading done by machines decreases as your investing timeframe increases.

So at the extreme (high frequency trading), all trading is indeed done by machines.

[lujim]

In addition to an algorithm not having all available information on the market there is also the fact that any niche or inefficiency in the market can be duplicated by others until it is negated.

If you discovered that the market always goes up on Tuesday and drops on Wednesday that only works until everyone else discovers the same thing and starts selling on Tuesday and buying on Wednesday.

[oaktowner]

Most stock brokerages spend lots of times gathering data from the companies themselves -- the management group, the customers, industry analysts, etc. They are not making investment (BUY/HOLD/SELL) recommendations based on the past fluctuations of the market, rather on their "expert" appraisal of the company's worth versus its current stock price.

[blazespin]

You mean the computers that can't even understand the sentence "We saw grand canyon flying to chicago."

[john61]

I recommend to read Warren Buffet or his teacher Benjamin Graham. Buffet uses a computer, but only for Bridge playing.

[tmaly]

market conditions are always changing as are regulations. In 2008, they implemented an emergency short sale rule that banned short selling for a select set of stocks. An algorithm would not know what that means, humans have to step in to ensure stocks are not shorted during these times.

[Peradine]

Forgive my ignorance, but would it not be quite easy to tell the algorithm not to short certain stocks?