Since you time everything we should add this to the mix. While it's usually not crucial to propagate rays up to a certain point in time (but instead propagate them up to a certain point in space), since you already do that, we might as well affect the speed dependent on the refractive index of the medium (and obstacles). This would be good down the line if and when we want to include diffraction.
Want to back this issue? Post a bounty on it! We accept bounties via Bountysource.