I am exhausted…
Sequence of events…
The cluster lost one of the blades today for the first time ever (this blade is the 2nd master DB server). The primary master writes everything to the 2nd master, then the 2nd master replicates out to all the slaves (read only access). Something happened with mysqld before, but this time it was a hardware issue.
I screw with it remotely as best I can. Once I figure out it’s a hardware problem, I go to data center.
The blade is on, but not able to be pinged. Okay…. let’s just power cycle it.
Weird, the power button doesn’t work on the blade.
Check the full post… Sorry Shawn!