I got into quant finance 12 years ago with the mistaken idea that I was going to successfully use all these cool machine learning techniques (genetic programming! SVMs! neural networks!) to run great statistical arbitrage books.
Most machine learning techniques focus on problems where the signal is very strong, but the structure is very complex. For instance, take the problem of recognizing whether a picture is a picture of a bird. A human will do well on this task, which shows that there is very little intrinsic noise. However, the correlation of any given pixel with the class of the image is essentially 0. The "noise" is in discovering the unknown relationship between pixels and class, not in the actual output.
Noise dominates everything you will find in statistical arbitrage. R^2 of 1% are something to write home about. With this amount of noise, it's generally hard to do much better than a linear regression. Any model complexity has to come from integrating over latent parameters or manual feature engineering, the rest will overfit.
I think Geoffrey Hinton said that statistics and machine learning are really the same thing, but since we have two different names for it, we might as well call machine learning everything that focuses on dealing with problems with a complex structure and low noise, and statistics everything that focuses on dealing with problems with a large amount of noise. I like this distinction, and I did end up picking up a lot of statistics working in this field.
I'll regularly get emails from friends who tried some machine learning technique on some dataset and found promising results. As the article points out, these generally don't hold up. Accounting for every source of bias in a backtest is an art. The most common mistake is to assume that you can observe the relative price of two stocks at the close, and trade at that price. Many pairs trading strategies appear to work if you make this assumption (which tends to be the case if all you have are daily bars), but they really do not. Others include: assuming transaction costs will be the same on average (they won't, your strategy likely detects opportunities at time were the spread is very large and prices are bad), assuming index memberships don't change (they do and that creates selection bias), assuming you can short anything (stocks can be hard to short or have high borrowing costs), etc.
In general, statistical arbitrage isn't machine learning bound(1), and it is not a data mining endeavor. Understanding the latent market dynamics you are trying to capitalize on, finding new data feeds that provide valuable information, carefully building out a model to test your hypothesis, deriving a sound trading strategy from that model is how it works.
(1: this isn't always true. For instance, analyzing news with NLP, or using computer vision to estimate crop outputs from satellite imagery can make use of machine learning techniques to yield useful, tradeable signals. My comment mostly focuses on machine learning applied to price information. )
A few years ago I graduated with a PhD in statistics with lots of ML inspiration. Since then I have always dreamed of applying my knowledge and skill in this domain. However, despite the belief I was 'probably' in a decent position to do so, I consistently read about how impossible it was. I have a boring 'normal' persons job, but, posts like this are somewhat reassuring that I made a reasonable decision to abandon a life of fruitless datamining and overfitting.
> Most machine learning techniques focus on problems where the signal is very strong, but the structure is very complex. For instance, take the problem of recognizing whether a picture is a picture of a bird. A human will do well on this task, which shows that there is very little intrinsic noise. However, the correlation of any given pixel with the class of the image is essentially 0. The "noise" is in discovering the unknown relationship between pixels and class, not in the actual output.
Could it be that by looking only at the prices timeseries we are not looking at the actual information but only at the output of a irreversible function and that for effectively predicting the prices we need a model that captures what actually happens in the real world?
Could you elaborate on news & NLP in regards to stocks?
We tried sentiment analysis in uni a few years ago and had no good results:
The idea was essentially: news says: 'stock A is great' -> it goes up shortly thereafter
We tested our algorithms against classifying Amazon reviews & Tweets by sentiment. Those are filled with sentiment and easy to detect if it is a 5 star review or a 1 star review. The news articles we parsed all had near neutral sentiment. We ended up building a classifier that could detect the news category of an article quite easily instead.
My initial idea was sparked by the Gulf spill and the subsequent dip in BP, I wanted to detect and capitalise big events like that, but the news sources we parsed always seemed to significantly lag behind the stock movement, too.
That distinction from Hinton is quite interesting, I often work with "Machine Learning" models, but for the most part the models are regression models and/or tree classification models that are easier to understand conceptually. I have yet to actually implement any complex machine learning techniques because I fear even if they do or do not work, I won't be able to interpret them in terms of their statistics.
What's your view on cases like Renaissance Technologies? There's an interview online on James Simons (https://www.youtube.com/watch?v=QNznD9hMEh0) - and he explicitly talked about using math models to detect anomalies, e.g. trends. It's also known that they've used Hidden Markov Models, at least in the early days.
Most academic CS literature is complete BS. The vast majority of papers fit into a simple formula of "We apply method X to problem Y, and outperform other approaches using approaches similar to method X".
Meanwhile, no one uses method X or any of its cousins, because in the real world the problem is solved very differently with a combination of both principled algorithms and heuristics derived from real-world datasets.
The paper also fails to give any theoretical reason or mathematical insight as to why their version of X is better.
Thus, it doesn't actually solve a real world problem OR advance scientific understanding.
I think this is a reasonable point, but I would just add that a lot of people in CS academia are well aware of this. The problem is that we all serve multiple masters and one of the things we have to do is publish frequently. I think you'll find that many CS academics try to strike a balance between publishing for the sake of publishing and actually working towards a larger scientific goal. Personally speaking I'm definitely guilty of publishing work that looks good on my CV, but does not advance my deeper scientific agenda. That said the science is always on the front of my mind even if it only makes up 10-20% of my actual publications.
In regards to solutions it would be great if we focused less on the frequency at which we published and editors were more willing to publish work that had novel ideas even if it did not have state of the art performance (yet). Although like any job there will always be parts that are tedious, involve politics and yes parts that are even counter productive. At some point as an individual you just have to play the game while still thinking about and trying to advance the bigger picture.
This sort of comment does very little to advance the conversation. I made several attempts to write a response that talks about heuristics, algorithms, what industry does and what academia does, but in the end I found your portrayal of academia and industry to be so hyperbolic and unrealistically one sided that I felt it not even worth getting into the discussion.
Let's try to have respect for each other on both sides of the "academia/industry" fence and not resort to unrealistic black and white views of the world.
Thats complete BS. Certain journals are complete BS, and certain subfields of CS are superfluous and often times lack substance in publications. But CS literature, from top tier journals, has vast merits.
Long story short, all "research" is pretty much B.S.
I used to assume this is because people want to make money, so they keep the good analysis secret. However, after working in the industry a few years; it's mostly because they just don't know how to apply the algorithms or if it's even possible.
I think my favorite example is the seminal paper on using twitter sentiment to predict stock movement[1]. They don't use a large enough data set, and more importantly they use granger causality to identify "casualty"[2] between sentiment and stock value. They then claim they found a specific range which has a p-value indicating they are correlated... Of course you'll find a correlation when you look at two normalized signals and try to match them up.
Now, if they had not use the DJIA (Dow Jones Industrial Average) and instead used 500 individual stocks, and found the sentiment on twitter correlated with stock value(s) 90% of the time between 5 and 10 days. I'd argue they probably have something.
However, because their method is literally a BFS on only two signals in an attempt to find a correlation, they must correct for the p-value. i.e. "look and you shall find"[3]
This is just one of the hundred issues I've found, but really sheds light on how bad that industry is.
[2] Granger Causality offsets two signals in an attempt to find a correlation between them at an offset between two times. AKA find "causality" by finding correlation at an offset in time
> Long story short, all "research" is pretty much B.S.
Easy there! When I start feeling cynical about things like almost every sentiment analysis paper, I go back and read some of my favorite papers, especially ones from the 70s and 80s (eg, [1])
Another useful heuristic is to read the best paper award winners from conferences in a particular area. It's not a perfect metric, but the signal/noise ratio is better.
> Long story short, all "research" is pretty much B.S.
I'm curious about this as it relates to Piglet. Are you monetizing on the idea that most people won't realize this and will pay for Piglet with the hopes that they signals they get there are somehow positively correlated to what's happening in the real world for a particular stock/idea/company/politics when in reality you don't think it really does?
I was happy to help. Migrations are very complicated especially when in the face of massive traffic to move from one server to another and keep it live. I apologize that the dns transition wasnt seemless, but I managed a full migration in under 30 minutes. I'm glad I was able to help.
Does anyone know of any paper that describes a reproducible method of generating above normal returns in the mature western financial markets? Nope. Me neither.
[2] Jegadeesh and Titman, 1993. Returns to Buying Winners and Selling Losers: Implications for Stock Market Efficiency . https://www.jstor.org/stable/2328882
These don't use machine learning but they do answer your question.
Would be nice to see a standard academic platform for backtesting. Then, the paper could say "we submitted our implementation of this strategy to Backtest (which includes transaction costs and slippage)."
[+] [-] murbard2|8 years ago|reply
Most machine learning techniques focus on problems where the signal is very strong, but the structure is very complex. For instance, take the problem of recognizing whether a picture is a picture of a bird. A human will do well on this task, which shows that there is very little intrinsic noise. However, the correlation of any given pixel with the class of the image is essentially 0. The "noise" is in discovering the unknown relationship between pixels and class, not in the actual output.
Noise dominates everything you will find in statistical arbitrage. R^2 of 1% are something to write home about. With this amount of noise, it's generally hard to do much better than a linear regression. Any model complexity has to come from integrating over latent parameters or manual feature engineering, the rest will overfit.
I think Geoffrey Hinton said that statistics and machine learning are really the same thing, but since we have two different names for it, we might as well call machine learning everything that focuses on dealing with problems with a complex structure and low noise, and statistics everything that focuses on dealing with problems with a large amount of noise. I like this distinction, and I did end up picking up a lot of statistics working in this field.
I'll regularly get emails from friends who tried some machine learning technique on some dataset and found promising results. As the article points out, these generally don't hold up. Accounting for every source of bias in a backtest is an art. The most common mistake is to assume that you can observe the relative price of two stocks at the close, and trade at that price. Many pairs trading strategies appear to work if you make this assumption (which tends to be the case if all you have are daily bars), but they really do not. Others include: assuming transaction costs will be the same on average (they won't, your strategy likely detects opportunities at time were the spread is very large and prices are bad), assuming index memberships don't change (they do and that creates selection bias), assuming you can short anything (stocks can be hard to short or have high borrowing costs), etc.
In general, statistical arbitrage isn't machine learning bound(1), and it is not a data mining endeavor. Understanding the latent market dynamics you are trying to capitalize on, finding new data feeds that provide valuable information, carefully building out a model to test your hypothesis, deriving a sound trading strategy from that model is how it works.
(1: this isn't always true. For instance, analyzing news with NLP, or using computer vision to estimate crop outputs from satellite imagery can make use of machine learning techniques to yield useful, tradeable signals. My comment mostly focuses on machine learning applied to price information. )
[+] [-] aatchb|8 years ago|reply
[+] [-] ithkuil|8 years ago|reply
Could it be that by looking only at the prices timeseries we are not looking at the actual information but only at the output of a irreversible function and that for effectively predicting the prices we need a model that captures what actually happens in the real world?
[+] [-] somehnreader|8 years ago|reply
We tried sentiment analysis in uni a few years ago and had no good results:
The idea was essentially: news says: 'stock A is great' -> it goes up shortly thereafter
We tested our algorithms against classifying Amazon reviews & Tweets by sentiment. Those are filled with sentiment and easy to detect if it is a 5 star review or a 1 star review. The news articles we parsed all had near neutral sentiment. We ended up building a classifier that could detect the news category of an article quite easily instead.
My initial idea was sparked by the Gulf spill and the subsequent dip in BP, I wanted to detect and capitalise big events like that, but the news sources we parsed always seemed to significantly lag behind the stock movement, too.
[+] [-] hobolord|8 years ago|reply
[+] [-] bitanarch|8 years ago|reply
[+] [-] unknown|8 years ago|reply
[deleted]
[+] [-] pg_is_a_butt|8 years ago|reply
[deleted]
[+] [-] dkural|8 years ago|reply
Meanwhile, no one uses method X or any of its cousins, because in the real world the problem is solved very differently with a combination of both principled algorithms and heuristics derived from real-world datasets.
The paper also fails to give any theoretical reason or mathematical insight as to why their version of X is better.
Thus, it doesn't actually solve a real world problem OR advance scientific understanding.
[+] [-] self_assembly|8 years ago|reply
In regards to solutions it would be great if we focused less on the frequency at which we published and editors were more willing to publish work that had novel ideas even if it did not have state of the art performance (yet). Although like any job there will always be parts that are tedious, involve politics and yes parts that are even counter productive. At some point as an individual you just have to play the game while still thinking about and trying to advance the bigger picture.
[+] [-] tensor|8 years ago|reply
Let's try to have respect for each other on both sides of the "academia/industry" fence and not resort to unrealistic black and white views of the world.
[+] [-] wyas|8 years ago|reply
Thats complete BS. Certain journals are complete BS, and certain subfields of CS are superfluous and often times lack substance in publications. But CS literature, from top tier journals, has vast merits.
[+] [-] tanilama|8 years ago|reply
[+] [-] jfoutz|8 years ago|reply
This seems like something an amateur could do. Wouldn't that tear down a bunch of this junk?
[+] [-] lettergram|8 years ago|reply
I've spend an unreasonable amount of time reading research related to finance for my web app Piglet:
https://projectpiglet.com/
Long story short, all "research" is pretty much B.S.
I used to assume this is because people want to make money, so they keep the good analysis secret. However, after working in the industry a few years; it's mostly because they just don't know how to apply the algorithms or if it's even possible.
I think my favorite example is the seminal paper on using twitter sentiment to predict stock movement[1]. They don't use a large enough data set, and more importantly they use granger causality to identify "casualty"[2] between sentiment and stock value. They then claim they found a specific range which has a p-value indicating they are correlated... Of course you'll find a correlation when you look at two normalized signals and try to match them up.
Now, if they had not use the DJIA (Dow Jones Industrial Average) and instead used 500 individual stocks, and found the sentiment on twitter correlated with stock value(s) 90% of the time between 5 and 10 days. I'd argue they probably have something.
However, because their method is literally a BFS on only two signals in an attempt to find a correlation, they must correct for the p-value. i.e. "look and you shall find"[3]
This is just one of the hundred issues I've found, but really sheds light on how bad that industry is.
[1] https://scholar.google.com/scholar?hl=en&q=twitter+stock+sen...
[2] Granger Causality offsets two signals in an attempt to find a correlation between them at an offset between two times. AKA find "causality" by finding correlation at an offset in time
[3] https://stats.stackexchange.com/questions/5750/look-and-you-...
[+] [-] mayank|8 years ago|reply
Easy there! When I start feeling cynical about things like almost every sentiment analysis paper, I go back and read some of my favorite papers, especially ones from the 70s and 80s (eg, [1])
Another useful heuristic is to read the best paper award winners from conferences in a particular area. It's not a perfect metric, but the signal/noise ratio is better.
[1] the flajolet-martin algorithm: http://algo.inria.fr/flajolet/Publications/FlMa85.pdf
[+] [-] aagha|8 years ago|reply
I'm curious about this as it relates to Piglet. Are you monetizing on the idea that most people won't realize this and will pay for Piglet with the hopes that they signals they get there are somehow positively correlated to what's happening in the real world for a particular stock/idea/company/politics when in reality you don't think it really does?
[+] [-] kusmi|8 years ago|reply
[+] [-] zacharydavid|8 years ago|reply
[+] [-] nyounker|8 years ago|reply
[+] [-] arjie|8 years ago|reply
[+] [-] nyounker|8 years ago|reply
[deleted]
[+] [-] nyounker|8 years ago|reply
[deleted]
[+] [-] zacharydavid|8 years ago|reply
[+] [-] zacharydavid|8 years ago|reply
[+] [-] rubatuga|8 years ago|reply
[+] [-] chvid|8 years ago|reply
[+] [-] MichaelDickens|8 years ago|reply
[1] Fama and French, 1992. The Cross-Section of Expected Stock Returns. http://faculty.som.yale.edu/zhiwuchen/Investments/Fama-92.pd...
[2] Jegadeesh and Titman, 1993. Returns to Buying Winners and Selling Losers: Implications for Stock Market Efficiency . https://www.jstor.org/stable/2328882
These don't use machine learning but they do answer your question.
[+] [-] proofofstake|8 years ago|reply
[+] [-] dogruck|8 years ago|reply
[+] [-] sgt101|8 years ago|reply