![](\"http:\/\/cdwan.org\/pics\/temp\/artisan.jpg\")
Today was characterized by a victory of luck over skill. We were supposed to relocate a 50TB Apple SAN from building A to building B. They’re about 3 blocks apart. Of course, moving 168 disks in 12 chassis, plus seven servers and assorted switches can be kinda tricky.<\/p>\n
The team moved the gear on Friday, and I arrived this morning to make sure that it stood up properly.<\/p>\n
This was made more stressful by the fact that we were doing it without a usable backup of the data on said SAN. I’m imagining the conversation: “Hey, the backup didn’t finish in time. Should we proceed with the plan that starts ‘DO NOT PROCEED UNLESS YOU HAVE A USABLE BACKUP?’ All the disk chassis were fine. All the servers were fine. All the networks and switches were fine, yet somehow the system couldn’t see its metadata. This is akin to having access to all of the letters in a book, but not having the ability to lay them out in the correct order on the pages. We stared at the system for a while, got sandwiches, stared some more, and were emotionally prepared for what I termed ‘plan Hail Mary,’ in which I was going to nuke the OS on the servers and hope that a clean install would correctly detect the perfectly usable data on the perfectly usable disks. This had a good chance of working, but it was also a one-way door if it failed. <\/p>\n Then, on a whim, I walked behind the servers and swapped two (working) cables with each other. The system sprang to life and began to serve up data again. I’m still trying to come up with an explanation for why this worked. I’ve got a decent story I can tell in hindsight, but it was really pure luck at the time.<\/p>\n So I guess I earned my nickel for the day.<\/p>\n Today was characterized by a victory of luck over skill. We were supposed to relocate a 50TB Apple SAN from building A to building B. They’re about 3 blocks apart. Of course, moving 168 disks in 12 chassis, plus seven servers and assorted switches can be kinda tricky.<\/p>\n![](\"http:\/\/cdwan.org\/pics\/temp\/natasan.jpg\")
As a reward, I got to rack and stack 11 more disk chassis for the frankenSAN system we’re building out of off warrantee and surplus gear tomorrow. I actually sort of like the physical exertion and visual gratification of rack-and-stack … though I rarely get to do it anymore.<\/p>\n","protected":false},"excerpt":{"rendered":"