Here are some general tips regarding a portfolio site, and they are views held by people in both the gaming and animation industries, as things like this were said by everyone that ever spoke at my uni and the events Ive attended.
- Have a nice clean layout
- Stay clear of fancy Flash intros/sites and such. If people have to wait an age before they can even judge your work, they probably wont bother
- Re: videos. Basically, use quicktime to encode - you can be guaranteed that a person can view it. Ok, they might have to install it, but its not some obscure codec that might not work.
The most important thing you need to remember, is that a portfolio is there to show off your work. Make it as easy as possible for the user to get to it; one click to a gallery instead of scrolling to the bottom of a page, or it being linked away behind other pages could make the difference of your work getting seen, or getting ignored. (seriously, people in the indusrtries are that harsh at times - they have to deal with too many applications and such to waste time on a badly designed site)
Onto more focussed comments.
I think you're limiting yourself far too much with the space allowed for your content, though I guess it depends what you're showing. More muted colours will definately work better, as I think you've realised. They distract the user less, and allow them to focus on the important stuff.
Everyone has different views naturally, but I hope some of what Ive said makes sense and can help in some way. Good luck with it.