I'm on server 2 and it normally comes back in 15 minutes.

Now are you sure that the server didn't come back in 15 minutes, but as everyone is suddenly trying to access it again at the same time, that you're not able to get on during this peek time, and hence it look like it taking 2-4 times longer?