We definitely know travel forward in time is possible with just raw speed. You can go for a drive, and because of the speed you travel, you move ahead in time. It may be an imperceptible fraction of a fraction of a fraction of a nanosecond, but it's still time travel. I don't see any reason why there couldn't be some method of going backward, too. We just have not discovered it yet... and we probably never will, even if a method exists.