Would it be reasonable to make 'time' an optional dependency?

As far as I can tell, it's never referenced outside of timestamp.rs, and 'chrono' has been the default Timestamp backend for a while now.

We'd need to add a check in build.rs to make sure only one time-backend feature is enabled. But that will actually make things much simpler for #3484, I believe.