Enter TweetWheel, a visualization tool that does just this. You need to only enter your Twitter username and TweetWheel does the rest, linking contacts as well as their contacts.
Be warned, however, that it can take a while if you have a huge contact list, like Scoble for example. The end result is the picture you see here, with all contacts arranged together. You can hover over any of them to know how many contacts they have in common with you, and clicking will take you to their twitter page.