I'd argue it's a physical thing. Aging is proof of time having a physical effect on objects.

It might be possible, with enough understanding, to reverse, freeze (relatively speaking), or accelerate growth/development of specific things in controlled situations. Or at least in theory. That, I think, will be about as close to time travel as one will get. I don't think it would ever be possible to reverse, freeze, or accelerate the growth/development of every single thing in the universe at once. And even then, that wouldn't be time travel so much as mass-manipulation. "Time" would still be moving forward, not backwards. You'd just be manipulating the effect it had on whatever it is you're manipulating.