When gates are closed attackers have to be able to overcome wall defenses (traps, abatis, logs, archer towers, etc). If they succesfully overcome defenses they then plunder. If they are defeated they do not plunder. The walls do not drop in level ever. You just have to rebuild the defenses in the wall when they are wiped out.

When the gates are open the wall defenses and your troops fight together. You can still incur losses to your wall defenses as well as your troops.