If their loyalty drops far enough there is always a chance they could leave. The things that would cause loyalty to drop are, as mentioned above, not having enough gold to pay the salary, utilizing too many levys or having your city attacked.