I have spent the better part of my intellectually conscious life trying to algorithmically “beat the market.” (To be precise, I started around age seventeen; I am now twenty-five.) According to some very well-tested academic theories regarding markets, I am pursuing a fool’s-dream. I have continued to labor this long under the assumption that there is a pretty obvious selection problem when it comes to publishing findings that contradict financial orthodoxy: if you were to find a method that earned out-sized returns, I don’t believe academic prestige trumps monetary gains. It is my perception that people who end up as professors of finance are typically people who had the desire to study markets in order to profit from them, but who never found their holy grail. If they had found something spectacular, I don’t think the incentive to publish is very high. (There are exceptions, but nothing ground-breaking.)
After eight years, I have nothing concrete to show for my efforts. As a consequence of shifting needs, I have learned a lot of computer science (e.g. compiler design, algorithms, and some unnecessarily high number of languages). Obviously, this skill set is valuable, but I have no successful projects to use as credentials. Every few months, I find myself excited over the preliminary results of my increasingly sophisticated simulations, only to be disappointed a few short weeks later to find that I was simply wrong. This has happened so many times that I no longer grow excited when I see positive results — I’ve grown into a hardened, semi-depersonalized skeptic.
My latest iteration of development appears exceptionally promising, but I expect it to bear no fruits. I learn each time, and my understanding of markets (and complex systems in general) is approaching some level of refinement, but I have no way of estimating when I might cross the line into profitability; worse I may be approaching this level asymptotically, with my limitations acting as a ceiling just below my goal. I feel like a modern day Tantalus.
I recognize that “beating the market” algorithmically may be either impossible or simply out of my reach, but I soldier on because I still find it fascinating. I believe that, had I switched course years ago, writing off the project as foolish, I would have probably, or at least possibly, been wealthy by other means by now. (Every time I started pursuing a product development project, I found myself shifting back towards market soon after the initial new project euphoria had faded.) If I could offer advise to my younger self, prior to perusing this path, I’d probably say don’t make the attempt. I have neither ethical nor moral objections to profiting by speculation. I merely believe I could have acquired the satisfaction that comes from achievement a long time ago, instead of bearing the frustration that accompanies not achieving something in spite of my best efforts. Nonetheless, I will not stop trying. I’m not blind to the possibility that I am a smart fool, but I want this more than anything else. I’m not sure where I would draw the line, where I would finally say giving up is the proper thing to do. I hope I never have to make that decision. I hope success finally obliterates the need for that decision.
I’m not sure why I wrote this. To some degree, it might be a warning notice to those who are considering following this path. As I said, I find markets fascinating, but most people (my earlier self included) enter the fray believing it to be a sure and short path to riches; it’s not. There are far less risky paths to wealth, especially for entrepreneurial programmers.
Would being wealthy be more fun than having a challenge that fasincates you?
@Dale: Wealth affords you with the opportunity to pursue challenges that are otherwise out of reach. The preference would be for becoming wealthy as a byproduct of my work on this particular challange. At a certain point, if this does not become sustainable, (i.e. income producing) I believe it will represent an increasing liability — one that also inhibits what other challenges I can take on. If I find myself at that point, I suppose that is when I will stop.
Did you ever read “The Predictors”? Given the rampant fraud that’s been the basis of various markets for several consecutive years, I keep wondering how *their* algorithms have been holding up. Maybe you should read the book and send them an email with a link to this post?
@Mike: I knew of Farmer, Packard, and McGill, but I have never read the book. I just ordered it (for $4.00 via Amazon used books.) Thanks for the recommendation.
Maybe you should have a chat with the guys at http://janestreet.com/ they seem to be doing pretty well with their algos. If nothing else maybe you can get a well paid gig there doing what you obviously love.
I founded and built my own high frequency algo trading firm, BlueStone Investments, back in 2007. I can tell you that there are profits to be had. For example We beat the market by nearly 50% in 2008. Also there are many hedge funds and even algo depts at your favorite investment banks that continue to produce alpha.
Finally I’m attending the University of Chicago Booth for my MBA while running a few startups and they DEFINITELY believe in the efficient market hypothesis but one of the most prolific professors, Eugene Fama, developed the Fama 3 Factor model that continually produces alpha. They explain it away by saying these and any returns above market can be attributed to risk and risk alone. Google it and you’ll find his website with huge datasets and performance numbers from running historical data tests back to 1929. Aside from trading fees and commissions you should be able to produce positive returns simply by following this methodology.
Stay the course my friend!
@Henrik: Thanks for the advice. I’ve been wary about taking jobs at firms like that because of IP concerns. I always believe I am on the cusp of something great, and wouldn’t want to share my algorithms. (Really, how would you prove you were not stealing their IP?) At a certain point, this might be the best way of continuing.
@Sean: In my head, I was referring to Fama’s three factor model (including extensions for momentum) when I said, “There are exceptions, but nothing ground-breaking.” I do appreciate hearing stories of success though. Congratulations to you, sir.
If you’re doing what you love, you’ve already won.
I feel a similar way pretty often. I have a couple of visions of software I want to build that are far out of my technical reach. I’ve been prototyping them, on and off, for about eight years too, depending how you count.
They’ve all failed. I start with a flash of optimism, and code until I realize I was wrong. I then stop, and rethink things until I think of another guess.
This is exactly how great accomplishments happen. You may be working on an impossible problem, or you may be working on a 20 year problem. Or maybe you’re lucky and it’s a 10 year problem.
But even if your goal is impossible, it’s very likely that if you work on it for 20 years, you *will* solve a 20 year problem. It might be slightly askew to what you thought you were solving, but it will happen. It’s almost inevitable.
Regardless, I wholly support what you’re doing. Keep going. And read widely… explore… make sure you’re playing in the margins of your field, and exploring odd projects far outside the scope of your work. It’s those places that you will make that accidental discovery… that in all these years of trying to “beat the market” you made a fundamental discovery in artificial intelligence, or network theory, or whatever.
And yeah, if you’re doing what you love, you’ve already won.
The assertions of success are interesting, but how granular are they? What do the algorithms do on the days when the Fed’s Plunge Protection Team has to go into action, for example?
My experience is that doing what I love over the long term requires a supportive community. Breakthroughs happen when you have someone to turn problems upside-down for you, or force you to come and talk over the bits of your thought-process that are too painful to make yourself think about.
So a good question is: why are you doing this alone? Yes, if you go to Jane Street, you won’t have the chance to make billions of dollars all for yourself. But if you find *the* secret, you’ll still be making far more than you could otherwise spend efficiently and you’ll have people to celebrate and struggle with.
More importantly, I wonder about the inherent value of this project. Make money by yourself, for yourself and you will be intellectually stimulated and rich, but still just yourself. Why not take those same algorithmic chops and try to trace government corruption or re-analyze scientific data? You’ll have friends and colleagues supporting you, you won’t have to fear having your work stolen, and maybe you’ll make the world a better place for other people, too.
Read Taleb’s books on market uncertainty.. i found them to be quite thought provoking.
You are only 25. How you can think you have failed, when really your life has only just begun!
The markets are an interesting thing, there’s a factor of luck in it all though. Out of interest what are you trading on? You have to factor in that big financial organisations do not pay the same as average joe trying to day trade or spread bet. Therefore have a higher chance of making profit overall.
Ok, here’s another point for you to consider. I was reading again about Long-Term Capital Management. At one point, they succeeded and became massively successful. But they had to also *do* something with that money they made. It’s not like they could just stop and say they made enough. Do you have an exit strategy for yourself, should you nail this and start raking in money? Given the amount of time you’ve put in, wouldn’t you want more and more payback? This is something you should consider because all the bright people at LTCM obviously didn’t!
You haven’t been at it long enough to have failed. Come back in another two decades and we’ll have some common ground to discuss whether you have succeeded or failed.
Thank you for writing this post. I’m 24 and I have achieved nothing also. You’re actually in a better position than me because you’ve stuck at one thing where I’ve spent—I think—on average 9 months on various projects for the last 6 years.
BTW, it’s ’soldier on’, not ’shoulder on’.
Don’t try to beat the market. Be the market.
@Erik: “if you work on it for 20 years, you *will* solve a 20 year problem” and “make sure you’re playing in the margins of your field”. Agreed whole-heartedly. I enjoy social systems in general, so their is a lot of room for exploration. (For example, I really enjoy simulating political systems.) Sometimes I feel like I do things backwards, experimenting first than reading about how people far smarter than I have solved the same problem. Sometimes I’ve found that my naïveté is an asset, and my solution is more robust.
@Adam: I have the upside-down part right, but I don’t have the community of people to bounce ideas off of. I think it’s hard to build a community like that for trading in particular, but not complex social systems in general. I am probably starting a Masters/Ph.D. program in computational social sciences this fall to get access to that community.
@Dave: I have read his popular books, in addition to his more rigorous Dynamic Hedging. I like him, but his style bothers me. His ideas are neither new nor original; he hints that this is the case, but his style suggests otherwise. (If you like him, you might find it ironic that he has a twitter account where he tosses out witty one-liners.)
@Dougie: I suppose the title was too provocative. It was just a reflection of my mood at the time of writing. I run my simulations against any instrument for which I can get good data. The only real-money trades I make are not engaged in under algorithmic direction; instead, they are usually macroeconomic plays.
@Mike: LTCM is fascinating, especially when Lowenstein tells the story. I don’t know what my exit strategy would be, but I would certainly have no problem saying, “I have enough.” If you look around my blog, you’ll see I like science in general, but medical research in particular. If I had the opportunity to use earned revenue for medical ventures, I’d be ecstatic.
@Evan: On that day, we’ll grab some beer.
@Ollie: If they are projects where your failing quickly, that’s probably a much smarter move than one long project.
@Ariel: (Sigh.) Thank you. I’d be lying if I said it was merely a mistake in mental translation. I think I should start listening to Orwell and avoid using them altogether.
@a web dev: By that, are you suggesting I become a market maker (something I am interested in) or build markets (something I am very very interested in)?
I’m on the same path as you…here are some things that have helped me find success in the past:
1) Keep a long reading list of blogs, relevant news sources, etc. and keep a sharp eye out for market anomalies. You’re right that nobody would be stupid enough to publish a winning strategy. Most of the time what happens is someone discovers an anomaly and posts about it, not sure what to do with it, and its your job to figure out a decision process that turns the anomaly into cash in your pocket.
2) unless you have crazy resources or skillz, there is no way you will be able to compete on speed (Getco, etc.) or information (Goldman Sachs, etc.). So if you’re looking for a general stat arb program that scales up to $100MM or something, you might want to shoot lower. Think about what disadvantages these firms might have. For example, large institutions won’t look at strategies that don’t scale well, such as various penny stock strategies, and high frequency firms are uncomfortable with running anything that doesn’t utilize their speed advantage or requires overnight position risk.
Shoot me an email if you’d like to chat. This is a highly competitive business and we all need some encouragement sometimes.
I too tried to solve this problem for a long time and failed. I too learned a lot in the process.
One piece of advice, if you continue down this road. Abandon all attempts to solve the problem for any time frame greater than a few minutes. Too many complexities and uncertainties crop up in longer time frames. That is the great lesson of chaotic systems. Yes, they are deterministic, but your inability to model every degree of freedom makes reality rapidly diverge from any model you can possible create.
The big boys only do algorithmic trading in short time frames. They may be on to something.
There is only one way to become “rich” if that is your goal, a stupid one if I may express myself.
Put away money to a savings account every month that you don’t need immediately.
That is all, most the rich people use this as their base, some get to be stinking rich mostly because of a luck factor, nothing else.
You won’t need any sophisticated algorithms for this.
I hope for you that you had a nice time doing all this because the nice time should be your main pursuit in life because that is what it’s all about.
This is pretty simple. You don’t beat the market by playing by the rules. You beat the market by insider trading, sweetheart deals, payola to congressmen that went to the same ivy league college as you, etc.
Here’s a thread you may want to check out at Reddit, but a successful independent algo trader:
http://www.reddit.com/r/IAmA/comments/9s9d7/iama_100_automated_independent_retail_trader_i/
He’s pretty cool to talk to via PM too.
… work the system, don’t let the system work you.
What does making a profit mean to you? winning 70% of your trades or maybe 50%?
For example, in short, you will never be able to beat the Forex market. Why? Because it is comprised of people (traders) like yourself, each attempting to guess the next move. Each trader is an individual, makes his or her own decision weather to buy or sell. Unless you get all of the traders on a Skype chat and synchronize the trading decisions you are simply wasting your time.
The good news is that these traders are humans and humans are repetitive. Why is there traffic every weekday at 5pm? Because most of us get out of work at 5pm. These are simple repetitive patterns. To win the market you have to identify and then follow the patterns. There are hundreds of them, trust me. Forget about creating a alg that guesses which way the market will go and create a alg that identifies and follows patterns.
Ive been on the path that you are now on and it never ends. I was just focusing on the wrong variables and it sounds like you are too. Change your mindset, create a new plan of attack, understand the environment, stay focused and win.
P.S. All that work is going to pay off soon.
Why would you want to “beat the market” ? Does it occur to you that earning a lot of money from the deficiencies of the market is anti-social ? At the root of wealth are people, working in factories, in arts, in research, etc. Trying to suck money out of this real economy is not generating any wealth at all, i. e. you are just trying to steal the workers’ product.
But maybe you are just trying to make a point ? What point exactly ?
All the financial crisis we are in right now, in most parts of the world, is mainly due to people trying to beat the rules of the market. Greece and most of EU countries are setting socially-regressing rules just because the wealth their workers produce is being stolen by people having no shame in taking advantage of other people’s hard work.
I wish there were regulations so that nobody could take advantage of fluctuations of the market (stopping real-time readjustment of the values, efficiently punishing arbitrage exploitation, etc.).
Great story!
Reminded me of “The Dip” by Seth Godin. I don’t even know you but I think you should stick with it. At about the 10 year point you will really have come as close to mastery of this subject as anyone in the world – success may be closer than you think if you can stay determined.
It takes 10 years of deliberate practice to become master at something.
You had your 8. Don’t give up yet.
One thing also.
The playing on wall street is zero sum game, where you try to extract as much wealth to you while your gain is other peoples loss.
Creating some new product, is a win-win situation.
I think you can’t beat the market, you can win a battle but not the war. The “market” is just people, you can’t win over the big boys, they have better means and fiscal laws than you.
If you discover something new, your knowledge will be “in the other side” in little time when you make it public(you will if it is valuable), and you will “change” the market, so your rule doesn’t work anymore.
Of course yes you can win over the bast majority of the population. They don’t know what they are doing. But is that ethical or moral? Not for me.
There are two kinds of ways to make money in stock market. Creating wealth or helping other people to do it with your capital(the slow way 10-20%interest per year)or taking other people’s lunch(much faster). This is just stealing other people work and savings over their entire lives in my opinion.
As Buffet said: all the markets have a fool, if you don’t know who he is, odds are it’s you!
I had made money in the stock market with good investments (doubled it in value a few times), and could had benefited from things I knew were going to happen before they burst, like the EU crisis with derivatives, but I didn’t want to. I just want my savings not loose all their value with the inflation.
I will be rich but at the same time miserable. I’m spending my time trying to improve other people lives, and somewhat this is what makes me rich and happy.
What do you want all this money? For expensive hookers? For your ego? For traveling the world alone or semi-alone(someone that cares more about your money than you). For being the richest in the cemetery?
Just curiosity.
Have you seen the movie “Pi”? >_>
i think to play the markets you also have to be able to “play the people” its not all numbers and books, u have to understand people’s psychology too i think, because to buy or sell is a decision people make not just logically i think… so its just a question of seeing all the variable. So i would say just keep trying cause if u get it right u could be unbelievably rich!! and u will have fun doing it since u say u r fascinated by the whole thing… so jut keep at it thats the most beautiful thing you can do.
You haven’t contributed anything to society. You haven’t created anything of value. Beyond your sense of entitlement how do you figure you’ve earned a reward?
You would have been better off just trying to get a job at Reniassance Technologies (http://en.wikipedia.org/wiki/Renaissance_Technologies).
They’ve found the “Print Money” formula and have been using it for the last 20 years.
In a gold rush you can either 1) pan for gold or 2) sell a “Prospector Package” composing of all the needed tools to pan for gold or 3) be the gold rush.
The first option is like trying to win the lottery (what you are doing now), the second option is where most of the real money is, and the third option is of course creating the gold rush and the system that profits from it.
You’ve done #1. Now you should move on to #2. And maybe in another 10 years you’ll figure out how to do #3.
But whatever you do, stop this nonsense of trying to beat the system; you have no advantages over it.
Work it, or create it, but don’t get worked by it.
Why not consider going the “rule-based discretionary trader” route? Dr. Van Tharp has stated that most successful traders he has known are of this type, not purely mechanical. I would argue that your brain is still superior to any program (except at very small timeframes requiring very rapid execution). Also, entries are not that important. It is your exits and position sizing that determine profitability. I cannot emphasize this enough…
I think you might want to go through Peter Bernstein’s Against the Gods: The Remarkable Story of Risk, especially the last chapters that deal in a great detail with the rise of automatic systems and their place in the financial spectrum.
Putting things in a perspective, he argues that there is no universal algorithm that can beat the market consistently. However, a good algorithm and an experienced trader who bases his decisions on the predictions and on his own sentiment make a difference. The point is that models stay models, while the reality changes, the world evolves through uncertainty that cannot be captured, hence the need of humans in the business..
thanks for the deep post. You remembered me a book that i’m reading, it’s “the lateral thinking” by Edward De Bono. You are incarnating the idea of lateral thinking: there’s another way to get to the objective and you are trying to find it without following the typical recipes of the success (means conventional study). Those whom survive to their thoughts are the only successful people. Maybe it’s just needed to rethink the meaning of “success”