What if your day-wrecking “OH $#!+” moment actually created the opportunity for some brilliant IT solutions? For instance, can you imagine turning a ransomware attack on your network into a serious case for more comprehensive employee security awareness training?
There’s little question that stuff happens in IT. Honestly, it’s surprising there aren’t more IT fire drills considering what businesses are up against: more endpoints, shadow IT, smarter hackers, low budgets, and a steady stream of really dumb help-desk requests.
Former Google employee Cozi Namer is all about the idea of “failing forward.” He’s not alone—some of the world’s smartest people are strong believers in using face-plants as growth opportunities. Behind the veil of anonymous screen names, IT pro Redditors gave some beautifully raw confessions about how and when they went wrong—and how they fixed it. Turns out, the following real “OH $#!+” moments had serious potential to turn into “heck yes” wins.
1. A typo turns into serious downtime
“I once mistyped a registry key for Exchange and knocked out about half a hospitals’ worth of users for a few hours,” admits mobomelter. “In my defense, I set the key at about 5 a.m., and I had been up since 6 a.m. the previous day.” Ouch. If mobomelter’s exhaustion was due to their workload, the inevitable awkward conversation about what went wrong could have been a good time to ask their boss to hire more staff.
This isn’t even the worst typo to happen recently: An Amazon Web Services employee’s typo in late February 2017 took down services for hours, impacting clients like Slack and Trello. Kudos to the cloud giant for their transparency and decision to turn it into a learning opportunity. An AWS blog post clarified how they modified their technical operations to make sure capacity couldn’t just disappear again in the future.
2. Years of data down the drain
One company suffered a snafu when they discovered their cloud backup had been failing for two years and all the data went down the drain.
“We are now working with a data recovery firm to recover our data to the tune of $18k,” the user IrishBro wrote. “It’s not looking good.” IrishBro escalated the problem as soon as it was discovered but failed to follow up and has subsequently “aged 5 years in 1 month” after a conversation with the COO.
Take this as a personal lesson to be a bulldog when it comes to known issues, but more importantly, consider your processes. Data loss can be a springboard to creating accountability around checking your backups often and testing restore processes.
3. The terminated employee is hacking your network
IT pros are often waging an uphill battle against culture when it comes to employee security. Case in point: Department managers can repeatedly fail to let IT know to disable accounts for employees who no longer work there. They simply forget, but it’s not a small deal when ex-employees still have access to your data. There’s over a dozen well-documented examples of seriously expensive revenge hacking incidents.
You definitely want to create an IT solution before you’ve got a ticked ex-employee in your network. Redditor htproto recommend setting a nightly query for all active users against active payroll database members. While it’s a smart idea, parkowork admits to billing departments for “extra account costs that remain active for a termed employee,” like monthly app subscriptions.
“Amazing how fast ruining a [department] budget gets people to act properly,” they wrote.
4. Someone’s personal mobile device gets phished
Phishing is scary enough, but mobile phishing risks can be downright messy. Can you imagine trying to explain to your boss that your entire company network is now compromised because someone clicked on some ad malware or a spammy text message on their personal device? These risks only multiply when you think about how many personal devices are connected to your company Wi-Fi at any given time.
“It’s difficult to tell someone who makes twice your salary that they can’t connect their phone to the Wi-Fi,” admits Yake404. Plus, will they listen?
While you hopefully won’t suffer an IT mistake resulting from mobile phishing on a personally owned employee device, this would be a smart way to start a hard conversation about employee security. To avoid this bad day, consider separating Wi-Fi for nonmanaged devices.
5. When the problem’s between the chair and keyboard
What about all the times when IT gets pulled into another user’s mistake? Former help-desk workers can commiserate with some of the silliest mistakes—like the one time StacyE’s coworker complained their print jobs were always smudged because they were yanking out active print jobs. Or the time aaron4395 opened a help-desk ticket complaining the printer didn’t give “fair warning” it was running low on toner. Or when Dan04048’s coworker was confused as to why they couldn’t send a print job to a label maker.
Mistakes happen, especially when people interact with technology. But what if the most cringe-worthy help-desk requests you get were a call for smarter IT solutions, like a business printer that does give fair warning when it’s running out of ink? It’s definitely something to think about.
IT pros aren’t perfect. Everyone makes occasional typos or ignores someone’s shadow IT because it’s just easier than awkward conflict. While embracing the idea of failing forward doesn’t make your snafus any less painful when they happen, it can turn those mistakes into lasting benefits.