I was getting the very same anomaly, (a diagonal, white on top/grey on
bottom that flashed color when playing), and I just stumbled upon a
1. Right-click in the standard youtube video window (non-full
2. Choose 'Settings...' from the Flash Player's pop-up menu.
3. Uncheck the 'Enable Hardware Acceleration'
4. Click 'Close'.
Full-screen should now work, or at least it did for me. Some kind of
Flash incompatibility with latest Firefox build, it would seem. I am
running vista, so maybe that has something to do with it, as well.
Hope this helps.
I did as suggested above, and it started working fine.