Friday, February 24, 2012

Add database functionality to Matlab with SQLite.

Matlab data structures are fine for most of research work, but running a daily trading business is a different thing. A trader or an account manager often needs to maintain a list of trades, accounts, strategies, clients etc., often dealing with relational data An SQL database is ideal for such a task, but most solutions (like MySQL) are are quite an overkill, as we don't need a concurrent client database, data trees etc.
A solution comes in form  SQLite, a serverless database engine that stores the whole database in a single file. It is widely used in anything from mobile phones to mainframes and runs on almost anything.
Quite some time ago I have written a post on using SQLite in Matlab. As I am still a happy user, time for an update and a demo.
Installation: you only need to download and unzip the files from here: http://mksqlite.berlios.de/mksqlite_eng.html
(note: you'll probably need to compile from source on a 64bit system).

Demo: I have written a simple script to demonstrate how to keep a list of portfolio positions for thee separate accounts.


%{

Copyright: Jev Kuznetsov 
License: BSD

demo of SQLite for portfolio management.

%}

% test sqlite
clear all;
clc;
mksqlite('open','test.db');

tables = mksqlite('show tables');
disp(tables);

%% create a new data table 
mksqlite('DROP TABLE tbl_portfolios');
sql = 'CREATE TABLE tbl_portfolios ( id INTEGER PRIMARY KEY AUTOINCREMENT, accountName TEXT, symbol TEXT, position INTEGER)'; 
mksqlite(sql);

%% now add some random data 
symbols = {'ABC','DEF','GHI','XYZ','AAA','BBB','CCC','DDD'};
accounts = {'acct1','acct2','acct3'}; 

%mksqlite('PRAGMA synchronous=OFF'); % speed tweak, see sqlite doc
mksqlite('BEGIN'); % bundle multiple inserts  into one transaction, speed boost!
tic
for i=1:100
    symbol = symbols{ceil(length(symbols)*rand)}; % pick a random symbol from symbols
    account = accounts{ceil(length(accounts)*rand)}; % same for account
    position = ceil(1000*rand); 
    
    fprintf('adding account: %s symbol:%s position:%i \n', symbol,account,position);
    
    % first, check if symbol is already in portfolio
    res = mksqlite(sprintf('SELECT id FROM tbl_portfolios WHERE accountName="%s" AND symbol="%s"',account,symbol));
    if isempty(res)
      fprintf('Adding symbol \n');
      mksqlite(sprintf('INSERT INTO tbl_portfolios (accountName, symbol, position) VALUES ("%s","%s",%i)',account,symbol,position));
    else
      fprintf('Updating symbol \n');
      mksqlite(sprintf('UPDATE tbl_portfolios SET position=%i WHERE id=%i',position,res.id));
    end
    
end
%mksqlite('PRAGMA synchronous=NORMAL');
mksqlite('END');

toc

%% now pull the data from database

fprintf('\nGetting data from database\n');

res = mksqlite('SELECT * FROM tbl_portfolios ORDER BY accountName ASC');
fprintf('Account\tSymbol\tposition\n-----------------------\n');
for i=1:length(res)
  fprintf('%s\t%s\t%i\n', res(i).accountName, res(i).symbol,res(i).position);
end
  
%% try some handy sql stuff
% unique account names
res= mksqlite('SELECT DISTINCT accountName FROM tbl_portfolios') 
% sum of all positions in acct1
res= mksqlite('SELECT SUM(position) as sm FROM tbl_portfolios WHERE accountName="acct1"') 

Friday, October 14, 2011

Which programming language to choose?

When I started programming as a kid somewhere in the early nineties, choosing a programming language was easy, as there were not many to choose from.
I first started in Pascal and since then have programmed in Delphi, C, C++, C#,Java , VB, PHP, Matlab, Python, SPIN and even ASM. All this without even being a professional programmer (I am a physicist). I did not learn all these languages for fun, as I have better things to do (like actual work), but I needed to as I had no ‘swiss army knife’ language for all my needs.
Ideally, I would like learn only *one*  language that is suited for all kind of stuff: number crunching, application building, web development, interfacing with APIs etc. This language would be easy to learn, the code would be compact and clear, it would run on any platform. It would enable me to work interactively, enabling the code to evolve as I write it and be at least free as in speech. And most importanty, I care much more about my own time than the cpu time of my pc, so performance is less important than productivity.
I also need a language that is capable of handling large arrays of data in a clear way. Luckily, after considering all these wishes, only two contestants remain: Matlab and Python.

Matlab is king of the hill when it comes to technical computing. If your tasks a limited to research only, it is probably the best tool there is. Good IDE, fantastic plotting function, great documentation. It is less well suited for application development or as a general purpose language. Expect to pay ~2k$ for a basic commercial license. Extra toolboxes cost more.

Python is very similar to Matlab and is free! Interactive work in Python is a bit less easy as in Matlab, but what you get is a programming language that can complete almost any task, from data mining to web development.

If I’d have to start all over again, I would choose Python as it would save me the trouble of learning another language for Gui and web development.

After flirting with Python for some time, I’ve fallen in love with it and decided to use it for most of my work from now on.

I'll be running a series of Python posts on my other blog: http://tradingwithpython.blogspot.com/

Sunday, October 9, 2011

Why should I learn a programming language?

People are bad at math. Some are better than others, but who can calculate ln(sqrt(345)*34)^3.4 in less than 1/1000th of a second? Even the simplest computer can calculate at a rate million times faster than a human, in fact this is the main reason computers have been invented. To use the full capacity of a computer the user should learn how to progam it.
People are quickly bored. Imagine you'd have to download around 1500 excel sheets of stock data and then combine them together? Unless you have an autistic disorder you would probably get bored to death. A computer would not mind doing the task, but you'd have to program it.
Another example is reaction speed.  It takes ages measured in computer time, for a human to see a price change in a stock and then just press a button.Curious about how fast you can react? Take a test here. Your reaction speed should be around 0.2s,  while even a simple microcontroller (the ones you’ll find almost in any electronic gadget)  will do the same task in 0.0000001s.  Want to compete? If you know what action should be taken in a certain sitiuation, a computer can do the job for you a couple of million times faster, but still you need to tell it what to do by means of programming.
Still, even with all that raw computer power nowadays, measured in gigahertz and terabytes, the 'heart' of computers haven't changed much since their invention: they are still not much more than 'calculators'. Yes, we've come a long way in terms of size factor, power consumption, user friendliness and price, but still, computers are incapable of creative thinking, adaptation to new environments and unpredictable situations. A cockroach is more intelligent measured by these standards than the most advanced computer. By combining the human power of creative thinking with the raw processing power of a computer a trader can become supertrader, achieving a consistent return.
Financial markets are just like nature, the smartest ones with the best skills for the current (and ever changing) environment survive. The ones that do not adapt get extinct at some point.
Using computers for trading today is a fact of life, and a trader has got no choice but to adapt. Who does not use some form of charting tool to find entry and exit points or an excel sheet to keep track of performance? Not much algorithm here, but it is already a form of computer-assisted trading. Another level is to purchase special purpose software, like spread trading tools. Often it includes some form of 'black box' logic, so you can only hope that it works and continues to work in the future. But why not move to the top of the food chain and learn how to process incredible amounts of data, design own algorithms, backtest strategies, automatically place hundreds of orders and analyze trading performance? By writing your own tools and programs, you'll be able to quickly adapt to every new market environment.
Your time invested in learning programming techniques will have a ten-fold return in time saved from doing boring daily trading tasks.

Coming next: which programming language to choose?

Sunday, September 25, 2011

Getting started with Matlab & finance

I am thinking about creating a series of screencasts showing how to use Matlab for financial research. This is the first one, covering the basics and aimed at people not yet familiar with Matlab.
The whole series would include (among others) topics like:

  • Getting data from the web (yahoo, google, CBOE etc)
  • Aligning and filtering datasets
  • Nearest-neighbor classification
  • Designing & backtesting strategies
  • Interfacing with Interactive Brokers
  • Keeping track of strategy performance

Creating quality material takes a lot of time and effort, so I don't think that I will be able to offer whole material for free.
If you are interested in the series please let me know what you think by filling in the poll on the right.

Get source files

Wednesday, August 10, 2011

XIV is stealing my money!

With current volatility exceeding 40, it seemed like a good idea to short the VIX. Two very popular ways to do this is either to short VXX or go long XIV. I choose to go long XIV a couple of days ago, after making sure that it tracks the inverse daily return of VXX.
Now a strange thing has happened that both puzzles me and pisses me off: yesterday there was a tracking error of 2% between the two and today another 1%, both in my disadvantage. It seemed like somebody 'stole' 3% of my position! At this moment I am less than happy with this XIV product.
Take a look at this chart:

Here I plotted daily returns of VXX and XIV against each other, where the outliers have been plotted in red. Please note that there are another 2 outliers of similar magnitude, occuring on  4 and 5 January 2011, but these two cancel each other out pretty nicely.
I thought that both etfs were based on the same SPVXSTR index, but while their intraday path relative to each other is very  stable (no arbitrage possibilities), the tracking offset was present throughout the whole day.
I understand that XIV could get a positive tracking error because it was banned from short selling on both days (just like VXX), but a negative tracking error is a mystery to me.

Can someone shed a light on what is going on here???

Thursday, August 4, 2011

Are we in for a rebound?

Today we've had a pretty heavy drop in the SPY, that has been preceded by a series of down days. It could be tempting to join the panicking crowd, but the crowd usually gets it wrong.
The put/call ratio and the VIX/VXV ratio tell their own story:
Both of them are pretty high, the situation that is usually accompanied by a pop in the market.
Disclosure: I'm long.

Thursday, July 28, 2011

Going Python II

The trading platform to be now has got a name: Nautilus. It is available on Google code.
I have decided to share the project in an early stage. Because of this it may currently be of interest to a limited group of people willing to write more than a couple of lines of code.
Python is very new to me. That, combined with the fact that I am more of a researcher than a programmer, can lead to a less-than-optimal code. I hope that the open nature of the project will help it to improve in quality and grow.