The sad thing about any thread including the term "High-Frequency Trading" is that we won't be able to talk about the technology, and instead will need to relitigate automated trading.
I think we'd have been better off if the title of this thread had been "Online Algorithms in Automated Trading Systems". As far as I understand, that title has exactly the same meaning, but doesn't set off a crowd of HN commenters who have never placed a limit order jumping in to crow about _Flash Boys_.
I'm a little frustrated, since this is a technical topic that happens to be very relevant to me right now (I'm not a trader, FWIW).
The High-Frequency Trading Arms Race: Frequent Batch Auctions as a Market Design Response[1][2] is a must-read. The tldr; is that a continuous mechanism (sequential trade processing) only looks efficient in time-space, but is actually inefficient in volume-space (most of the volume trades at stale prices). The solution is discrete batch auctions (batch trade processing).
I'd share the same sentiment. I was really disappointed yesterday when the only comments (after 5 hours of nothing) was a seemingly snark question on the evil of HFT.
And to everyone who feel the need to voice their disapproval of HFT: if this was an article about improving the penetration of M1 Abrams, are you gonna be voicing your disapproval about ... something? Why/ why not?
The first section of this which relates HFT to one-pass algorithms is kind of useless. Online estimation algorithms are much more broadly applicable than that, and I'd expect that their relevance to HFT would be immediately obvious. I've never worked in HFT, but I've probably written the online variance algorithm at 3 of 4 companies I've worked. If you care about metrics such as "how quickly are each of my customers sending me requests?", then this stuff is relevant to you.
But more to the point, is there anyone in these fields who doesn't know this stuff? At a basic level, the things that you can do a really good job estimating in an online way are closely related to sufficient statistics, which every undergrad stats course should cover. And hopefully anyone actually involved in writing HFT systems has way more stats background than that -- right?
I'm a systems programmer with a pretty primitive intuitive sense for maths. I "agree" with you, in the sense that the algorithms this article mentioned were intuitively obvious to me and thus seemed like they couldn't be sophisticated. What's the state of the art for online estimators?
I've been playing with simulations of market maker algorithms from the literature, and simple Bayesian stuff and MLEs are all I've really seen, which makes me think that the interesting stuff isn't in the literature.
a) there is a pernicious and wide spread myth about the elite status of people working in HFT. My experience is that it is largely the same as any other technical profession. That is to say, it is no more or less likely for people in HFT to know about online estimation than any where else.
b) Given the reactionary view that many HFT participants have to information sharing, widely known/documented algorithms are often thought of as secret sauce and are frequently not talked about.
c) I agree with your sentiment specifically to one-pass algorithms being not terribly novel but am very happy that people with HFT backgrounds are publishing anything at all as it will hopefully help to remove the backwards thinking around secrecy so prevalent in the industry.
The particular emphasis on one-pass algorithms might seem self-evident, but the thread running through all of this is an engineering challenge that is very relevant to HFT. If your algorithm takes 60ms to evaluate your current position on each tick, by the time you are finished, the market has moved to the next zip code. Thus, to keep up with the market, you might find yourself rewriting the network stack for better response time.
Before HFT, there was only a few market makers, which meant the bid-ask was high (if you wanted to buy or sell right now, you'd pay more), liquidity was low (it'd be harder for you to get in or out of a large position and in hard times it would take longer for the market to recover - See 80s crash vs flash crash).
With HFT, there's a lot more competition over the bid ask spread. Trading firms are essentially in an increasingly expensive fight over pennies, which is good for everybody else, since it's cheaper prices, more liquidity, and a more efficient market.
The people it's bad for is the old-school style brokers (which are effectively gone now) and large investmant banks who have a fundamental view on the market (they need to be smarter about executing since the market is more efficient and reacts much more quickly to their buying and selling).
We see crazy competition on latency because these firms can no longer compete on price. HFT has taken the spreads down from $.25 to $.01 (putting a lot of market-makers out of business, saving a lot of money for anyone who is actually trying to buy or sell shares). But since the price isn't allowed to go below $.01 (sub-penny rule) the firms are now all in this wasteful zero-sum race to get that $.01.
I find this question soooo frustrating. I mean, we live in a free ( relatively speaking ) capitalist society. Why does there have to be use to society for anything, unless the issue in question has negative externalities on others ?
While you will hear many negatives here I believe there are few positive outcomes.
The needs of HFT are such that technology barriers are pushed to the limits. Some of the technologies developed in the process make it back to the general public.
You have the wrong question. The question is: What is the optimal market structure? Trying to strike a balance between the benefits of competition and the troubles of market fragmentation. Fragmented markets = HFT + competition. Unified markets = monopoly power + liquidity. You choose...
Unfortunately, there are a couple types of HFT. At it's core, it just means trading using a computer to execute more/faster trades. That seems justifiable in making the markets more efficient. Where most people including myself have issues is when they get a pass in certain markets to frontrun normal sellers and eat some of their margins simply because their latency was low enough. An earlier HN post that I can't find had a discussion on how this type of behavior can be countered by sending fake buy offers without intent to actually buy, but for whatever reason, that is illegal.
There is no benefit. HFT does not lower spreads in reality, only nominally. They put in orders but back away when a matching order comes in. The myth is that HFT improves liquidity. It may look like that. But those orders aren't real and can't be fulfilled.
This is again how scientificism goes. We think machines improve everything. "If only we trade faster, that will help the markets!" It's not hard to see this is BS.
Anyone who thinks HFT is something needed should go read Flash Boys. HFT is basically front running with some variations. There is a small amount that is necessary arbitration that will have to be done at some speed, but by and large it is not performing any sort of a service, it is taking advantage of the people trying to buy and trade stock by being a manipulative middle man.
There are three sentences in this comment. The first two are wrong, and the third is hard to understand, but if I understand it correctly, it too is false.
1. _Flash Boys_ has been panned by technical experts and makes very little sense to people with a basic understanding of how the markets work. I can link to technical reviews if you'd like to see them. I'd be interested in any technical reviews of the book that defend it; I haven't seen any. As a fan of Michael Lewis: this book was surprisingly bad.
2. Front-running is an agency problem. It's what happens when your broker knows you're about to move a lot of stock and, acting on that insider knowledge, jumps in front of your trade. Market-making isn't front running. To make a colorable argument that HFTs are frunt-runners, describe a hypothetical sequence of trades in which that happens ("BOB BUY 10 @ $10, CAROL SELL 10 @ $10.01, &c"). If HN history is any guide, that exercise will quickly deflate the notion that there's front-running (or even any unfairness) happening.
3. HFTs were preceded in the markets by the human specialist system, in which people deliberately increased the spreads to scalp money from trades. HFTs fight over pennies. Human specialists rigged the stock markets so that every share included a premium denominated in dimes and quarters paid directly to the specialists.
Mistakes aside, the weirdest thing about Flash Boys, to my mind, is that it's cast as a David vs. Goliath narrative - with the role of "Goliath" filled by the awful HFT shops and "David" by Goldman Sachs and J.P. Morgan. When "David" is orders of magnitude larger than "Goliath" and tends to swing the industry by the tail, it's an odd framing.
[+] [-] tptacek|11 years ago|reply
I think we'd have been better off if the title of this thread had been "Online Algorithms in Automated Trading Systems". As far as I understand, that title has exactly the same meaning, but doesn't set off a crowd of HN commenters who have never placed a limit order jumping in to crow about _Flash Boys_.
I'm a little frustrated, since this is a technical topic that happens to be very relevant to me right now (I'm not a trader, FWIW).
[+] [-] kylebrown|11 years ago|reply
1. slides: http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre... 2. paper: http://faculty.chicagobooth.edu/eric.budish/research/HFT-Fre...
[+] [-] NhanH|11 years ago|reply
And to everyone who feel the need to voice their disapproval of HFT: if this was an article about improving the penetration of M1 Abrams, are you gonna be voicing your disapproval about ... something? Why/ why not?
[+] [-] bladecatcher|11 years ago|reply
[+] [-] CyberDildonics|11 years ago|reply
[+] [-] abeppu|11 years ago|reply
But more to the point, is there anyone in these fields who doesn't know this stuff? At a basic level, the things that you can do a really good job estimating in an online way are closely related to sufficient statistics, which every undergrad stats course should cover. And hopefully anyone actually involved in writing HFT systems has way more stats background than that -- right?
[+] [-] tptacek|11 years ago|reply
I've been playing with simulations of market maker algorithms from the literature, and simple Bayesian stuff and MLEs are all I've really seen, which makes me think that the interesting stuff isn't in the literature.
[+] [-] kasey_junk|11 years ago|reply
b) Given the reactionary view that many HFT participants have to information sharing, widely known/documented algorithms are often thought of as secret sauce and are frequently not talked about.
c) I agree with your sentiment specifically to one-pass algorithms being not terribly novel but am very happy that people with HFT backgrounds are publishing anything at all as it will hopefully help to remove the backwards thinking around secrecy so prevalent in the industry.
[+] [-] wglb|11 years ago|reply
[+] [-] expertonkappa|11 years ago|reply
[+] [-] galaktor|11 years ago|reply
http://webcache.googleusercontent.com/search?q=cache:WqysSpS...
[+] [-] crazypyro|11 years ago|reply
[+] [-] amelius|11 years ago|reply
Links are also welcome.
[+] [-] reverend_gonzo|11 years ago|reply
With HFT, there's a lot more competition over the bid ask spread. Trading firms are essentially in an increasingly expensive fight over pennies, which is good for everybody else, since it's cheaper prices, more liquidity, and a more efficient market.
The people it's bad for is the old-school style brokers (which are effectively gone now) and large investmant banks who have a fundamental view on the market (they need to be smarter about executing since the market is more efficient and reacts much more quickly to their buying and selling).
[+] [-] Adlai|11 years ago|reply
The second link specifically talks about social utility.
[+] [-] res0nat0r|11 years ago|reply
http://en.wikipedia.org/wiki/High-frequency_trading
[+] [-] lmm|11 years ago|reply
[+] [-] gadders|11 years ago|reply
https://scottlocklin.wordpress.com/2011/03/01/the-arguments-...
https://scottlocklin.wordpress.com/2014/04/04/michael-lewis-...
[+] [-] vasilipupkin|11 years ago|reply
[+] [-] wodzu|11 years ago|reply
The needs of HFT are such that technology barriers are pushed to the limits. Some of the technologies developed in the process make it back to the general public.
[+] [-] Paradigma11|11 years ago|reply
Where is the commitee that decides on the allowed actions?
[+] [-] aet|11 years ago|reply
[+] [-] cheepin|11 years ago|reply
[+] [-] innguest|11 years ago|reply
This is again how scientificism goes. We think machines improve everything. "If only we trade faster, that will help the markets!" It's not hard to see this is BS.
[+] [-] CyberDildonics|11 years ago|reply
[+] [-] tptacek|11 years ago|reply
1. _Flash Boys_ has been panned by technical experts and makes very little sense to people with a basic understanding of how the markets work. I can link to technical reviews if you'd like to see them. I'd be interested in any technical reviews of the book that defend it; I haven't seen any. As a fan of Michael Lewis: this book was surprisingly bad.
2. Front-running is an agency problem. It's what happens when your broker knows you're about to move a lot of stock and, acting on that insider knowledge, jumps in front of your trade. Market-making isn't front running. To make a colorable argument that HFTs are frunt-runners, describe a hypothetical sequence of trades in which that happens ("BOB BUY 10 @ $10, CAROL SELL 10 @ $10.01, &c"). If HN history is any guide, that exercise will quickly deflate the notion that there's front-running (or even any unfairness) happening.
3. HFTs were preceded in the markets by the human specialist system, in which people deliberately increased the spreads to scalp money from trades. HFTs fight over pennies. Human specialists rigged the stock markets so that every share included a premium denominated in dimes and quarters paid directly to the specialists.
[+] [-] dllthomas|11 years ago|reply