Keep your 3D models simple. Icons are by nature small. Even if Vista and Seven support image sizes up to 256x256 pixels, they are never used at this resolution under normal circumstances. Instead, they are resampled to a reasonable size. This will hold true even in the future when resolution increases -> the relative physical size stays the same.
Pay attention to lighting. A great 3D model is only half of the equation. Play with light sources, consider using more light sources with lower intensity for smoother shadows. Try changing the gamma correction of the ray-tracing renderer.
Do you need shadows? If you don't, do not be afraid to turn them off in the renderer configuration.
Use high enough resolution for rendering. When creating Vista icons, it is wise to increase resolution in the properties panel of the rendered image to enable anti-aliasing by super-sampling.
Do the surfaces look blocky? If you can see artifacts introduced by usage of discrete triangles instead of smooth shapes in rendering, increase LOD factor in renderer configuration to get smoother shapes.
Using semitransparent objects? Be sure to enable transparency in the rendered image properties panel.