There is only one building that will effect the movement of both infantry and cav.
So yeah I got it, risky though.