Taking Out the Outliers II

This is a follow-up to my previous post in which I described Rousseeuw's FastMCD algorithm for using the minimum covariance determinant approach to calculating a robust covariance matrix. I want to use the example from that post (the relationship between the returns of QIM and Quantica over the last 5 years) to explore the behavior of the FastMCD algorithm. Finally I want to suggest a way of combining the advantages of the robust and conventional approaches to covariance analysis.

Taking Out the Outliers

If you do anything that depends upon correlation, you owe it to yourself to be aware of outliers and how to deal with them. For example, a mean-variance optimized portfolio using Modern Portfolio Theory or the Black Litterman model is critically dependent upon, and very sensitive to the portfolio covariance matrix. In turn, the covariance matrix is extremely sensitive to outliers (leverage points).

This post was inspired by a posting on the Tr8dr Blog. The original source paper in which the details of the algorithm are specified is "A Fast Algorithm for the Minimum Covariance Determinant Estimator" Rousseeuw, Van Driessen, 1998. This paper is an implementation of the ideas detailed in "Least Median of Squares Regression", Rousseeuw, 1984. Both papers are easy to find.

In this post I hope to add a little extra color, detail and exploration to the original sources above.

Best Buy: A Blogging Resource

I hope you have noticed the face-lift I have given the blog. I spent many hours wrestling with html within the Google Blogger environment where stuff goes on behind the scenes, thwarting your efforts to use CSS. Fortunately there are many great meta-blogs on using Blogger which provided the solutions every problem I had.

I have 2 pcs, both running Windows XP (!). I use Chrome, myself, and I have IE8 on both machines. Beyond that I have no way to see how my efforts look on any other type of machine (e.g. iPhone, iPad, etc) or browser (Safari). I happened to be in Best Buy picking up some iTunes gift cards (10% off) and of course, couldn't resist checking out the various computers on display.

It occurred to me I could also review my blog's appearance. So I tried everything from smartphones running android and IOS5 through tablets to an iMac with a 27in display. I was pretty happy with the rendering of my static page, and I may add some more of them in the near future. I highly recommend the exercise - go to your nearest big computer store and see how your blog looks to other users!

PS: I apologize to anyone visiting the Boulder Best Buy wondering why every computer in the shop is tuned in to some odd blog they never heard of!

Strata of Monte Carlo

This will be the final Monte Carlo related post for a while. Good job, really, I can't keep coming up with titles! I am hoping this one will pull together some of the the themes nicely and show a really cool way of implementing MC integrations using a procedure called "stratification".

I wanted to show the process by animating at least one of the charts in this post, but after hours of reading about the R "animation" package and trying to figure out how to use SWFTools, I gave up! If any kind soul wants to help me figure out how to animate a series of charts produced in R, don't hold back! I will put up the R code I used to generate the charts in this post - feel free to download and experiment with it.

In my previous posts, we have seen that

• The absolute error of an integration depends upon the number of trials
• The absolute error depends upon the degree to which the integrand fills the space bounding the simulation
• We have the flexibility to change the size and shape of the space bounding the simulation if it suits us

Stratification makes use of all these ideas.

Triangulating Pi in Monte Carlo

I have made several posts on the basics of Monte Carlo integration techniques. I introduced the topic using MC integration to estimate the value of pi. Then I took a look at the way the convergence characteristics of the estimate depend upon the number of trials. Finally, I looked at how the relative size of the area being estimated affects convergence. Finally, I want to put some of these pieces together and introduce a final basic concept: the choice of the sampling distribution.