I don’t know who came up with this idea, but they’ve just wasted about six hours of my Saturday: dynamic firewalling on a VPN network. It appears to block access to tcp ports, on the fly, if there’s nothing listening at the remote end – and then leave them blocked for an extended period of time. So, what happens when you shut down a three-node Oracle cluster for some maintenance? Some users – or other automated processes – try to connect to them while they’re down, and when they come back up again, no-one can connect at all.
This is then followed by a long, frantic attempt to prove that nothing has changed on the servers between reboots, because “this was all working fine before it was rebooted and now it doesn’t work” is rather hard to argue with.
So, thanks large-telco security people. I only had four hours’ sleep last night, and today I didn’t even get to see daylight.