Canary in a CrowdStrike

Cybersecurity giant's blunder is a warning for the whole world

“A full-blown rollout from a security vendor to every customer within minutes is very dangerous.” – Dave DeWalt, former CEO of cybersecurity company McAfee, in a recent Wall Street Journal article.

---

CrowdStrike has stumbled upon a obvious insight after its buggy software update shuttered governments, businesses, hospitals, airlines and consumers around the globe – leading to billions in damages.

The cybersecurity giant calls it “canary deployment.” This means gradually rolling out software updates in the future so that the company can check for looming catastrophes.

This way, not every Microsoft Windows user gets the dreaded “blue screen of death.” This way, only one airline shuts down instead of several of them. This way, only a few hospitals take a hit while CrowdStrike fixes its glitch.

Somehow, coming to this revelation is costing the company billions in market value, restitution and damage to its otherwise stellar reputation.

Not using canary deployment is like staging an outdoor rally for a presidential candidate – a risky proposition in itself – and not checking rooftops for snipers.

Thousands are paying the price. CrowdStrike should not have risked leaving all of its customers CrowdStuck.

Whose bird-brained idea was this? (Illustration: AI generated.)

The hottest software stock of 2024?

CrowdStrike stock has plunged nearly 30% since its disaster unfolded last week. Thousands of Americans who never heard of CrowdStrike are aware of it now after being stranded at airports, locked out of their computer systems, and stymied in business.

CrowdStrike said 8.5 million computers, mostly those running Microsoft Windows, took the hit. CEO George Kurtz, has been summoned to testify before Congress to explain. Global financial losses from the debacle have been estimated as high as $15 billion, with much of it uninsured.

Before all this happened, CrowdStrike was a Wall Street darling. For example, Downtown Josh Brown called it the “hottest software stock of 2024” on July 8, and gave Kurtz an absolutely cloying interview.

Brown is a money manager and frequent bloviator on CNBC. He’s smart, quick-witted and has a tight grip on the data, but like a lot of talking heads on CNBC, he talks his book. Following the crash, he wrote a piece on why he’s not selling his Crowdstrike shares.

For now, though, CrowdStrike has put the “down” in Downtown Josh Brown.

The lesson here is that you can lose a lot of money getting your stock tips on platforms where optimism bias abounds and nobody adequately explores the risk factors.

What could possibly go wrong? Everything, everywhere, all at once.

This will keep happening

This has happened before and it will happen again because global corporations are increasingly centralizing everything on computer systems.

A recent column from MarketWatch recalls a 2021 glitch from Amazon’s cloud services that crippled websites around the world:

“As tech companies get bigger, they create more ‘single points of failure’ that can lead to cascading catastrophes across our connected world.”

A case of the butterflies

As big tech consolidates, it creates fewer options when global resilience requires more. The latest revolution in artificial intelligence will only accelerate this trend.

We live in a complex web of cause and effect and the digital world puts us all in great peril.

If you’d like to learn more about how the tiniest of actions can lead to enormous outcomes, I recommend Fluke: Chance, Chaos, and Why Everything We Do Matters by political scientist Brian Klaas.

Klaas recently wrote a Substack post observing that the CrowdStrike debacle is a warning of things to come.

He writes:

“There is often a trade-off between maximum optimization and resilience.

“Our endless worship at the Altar of Endless Optimization, in which we aim to squeeze every last drop of inefficiency out of our social systems, is making us more prone to disaster.

“Can we really trust our species to flawlessly govern unimaginably complex systems – systems we don’t always fully understand – that can be brought down by a single screw-up?”

Like the proverbial wings of butterflies spawning hurricanes, a software developer farts and the whole world gets gassed.

We’re going to need a lot more canaries.

Don’t miss these Blunders

• Drunks Wanted This Week in Blunders – Jan. 19-25

• Blowing A Fartcoin Are meme coins heralding the next market crash?

• Billionaires Are Destroying Everything, And They Know It The World Economic Forum's 2025 Global Risks Report is a roadmap for the devastation to come

• Chicken Soup For The Soulless This Week In Blunders – Jan. 12-18

• Can Musk Single-Handedly Capture A Regulator? A relatively petty case against the multibillionaire may signal who is really running the country

• You Can Plunder, Too Private Equity firms want average Americans to join in the looting

• Florida Man Goes To The Movies This Week In Blunders – Jan. 5-11

• Bigots Fly Free United Airlines may be under fire for its diversity efforts, but it just settled a nasty racial discrimination case

• Lock ‘Em Up Welcome to the Business Blunders Hall of Shame commemorating top business leaders convicted of felonies

• Tip Your Banker? This Week in Blunders – Dec. 29 - Jan. 4

• Business Blunders Hall Of Shame From magnate to inmate: These are the nation's most notable white-collar convicts

• Unshackling Ponzi Perps It’s a bad idea to let convicted swindlers go free. Biden just gave clemency to a slew of them

• Married To Madoff Victims of a fraud that regulators missed decades ago are finally getting most of their money back

• MAGA Mad About Musk This Week in Blunders – Dec. 22-28

• Biggest Business Blunders of 2024 It was an ignominious year for economic forecasters, CrowdStrike, Boeing, FTX, Mattel, Truth Social, Red Lobster, Fisker, Spirit Airlines, TicketMaster and the dumbest bank embezzler in history

• Go To Zelle This Week in Blunders – Dec. 15-21

• Basketball Jones Former Harlem Globetrotter and NC State point guard gets seven years for a Covid-19 loan scam

• A CEO’s Road To Hell It began with forbidden love and ended in felony charges for Comtech’s Ken Peterman

• Sorry About That Opiod Thing This Week In Blunders – Dec. 8-14

• Spoiled, Rotten Day Trader Ian G. Bell touted a wealthy background to allegedly dupe investors in his money-losing investment scheme

• Shot In The Back Did maddening back pain drive Luigi Mangione to murder UnitedHealthcare CEO Brian Thompson?

• The Killionaire Here's why the man who gunned down UnitedHealthcare's CEO became an instant folk hero

• Enron Was A Parody Of Itself Here's where CEO Ken Lay ate before he died

• Big Blunders Come In Small Packages Macys blames an employee for hiding 'small package delivery' expenses. Anyone remember the boxes at Miniscribe?

• Billion-Dollar Reprobate Bill Hwang lost billions investing 'according to the Will of God.' Now he's going to prison for cheating Wall Street banks

• Hitting The 'Undo' Button At FTX Gary Wang wrote the code that enabled an $8 billion fraud. Prosecutors and a federal judge sang his praises

• Ponzi Once, Ponzi Twice A New Jersey man got probation for running a $300 million scam. Then he launched a $658 million fraud with ads on Fox News

• Hello? What About The $36 Trillion Tab? America's balance sheet is about to get worse and Republicans don't care

• A Mouse That Roared A small-time CEO launched a proxy war with $17.32 in his company's brokerage account. Now he's going to prison.

• Bozo Bezos The Washington Post's billionaire owner makes a clownish excuse for killing a presidential endorsement

• CEOs Gone Wild Abercrombie & Fitch had to know its CEO was a freak

• X Marks the Spot An Alabama man hacks the Securities and Exchange Commission with an iPhone, then returns it for a refund

• Danger, Will Robinson Destiny Robotics promised AI-powered humanoids that could form meaningful relationships, but it delivered a wig stand

• All the Kosher Meals You Can Eat Lufthansa gets sacked with a record fine for banning Jews from a flight

• Magic Mushroom ‘Pump and Dump’ Here's what can happen when 'shroom heads take control of a penny stock

• InHumana Bad reviews matter, and this health care insurer is paying a hefty price for ignoring them

• And the Winner is … Walmart The nation's largest grocer must be loving government efforts to break up the $25 billion Kroger-Albertsons deal

• The Incredible Shrinking Billionaire The chips are down for EchoStar's poker-playing Charlie Ergen

• Plastic Balls ExxonMobil put polymers in our testicles

• Bankster Bonus A top Old National Bank executive bags $2.6 Million after his arrest for allegedly molesting a child

• Watery Grave CEOs usually don't implode as literally as OceanGate's Stockton Rush

• Drivin’ that Train It's the end of the line for the embattled Norfolk Southern CEO

• Stinky Stonk Anatomy of a nightmare investment that's poised to get worse

• The Music Man Michael Smith allegedly played his AI-generated music billions of times to bots, ripping off streaming services and artists

• One Sorry Ferrari Online gaming CEO hustles car broker and scams investors with an IPO that never happened

• Adam Family Brothers allegedly raised $60 million with a phony crypto bot and spent investors' money on new homes and luxury goodsPiggy Banker A Bank CEO robs his own bank while crypto scammers rob him

• Clowns In Court An FTX felon wants to take back his guilty plea after prosecutors charged his girlfriend with campaign finance violations

• Piggy Banker A Bank CEO robs his own bank while crypto scammers rob him

• The Wonders of Blunders Why I follow business failures, and why you should, too.

• Holy Payday Batman A CEO bags $250 million driving his hospital chain into bankruptcy

• Design Your Life An alleged Ponzi schemer bagged $1 million a day with happy talk

• Nike Air Athletic-apparel giant's top executives preach climate change as they rev up their corporate jets

• If you can't woo 'em, sue 'em Elon Musk ran off advertisers; now he's suing them for leaving

• Drunken Chicken Tyson Foods' suspended chief financial officer could make a comeback

• In Real Life A "serial founder" finds a serial sucker in SoftBank

• Smoke on the Water Luxury Miami condo association sues to evict chain-smoking fraudster

• Beyond Stupid TV stock tips are for chumps

• Canary in a CrowdStrike Cybersecurity giant's blunder is a warning for the whole world

• Boeing’s Lost Spirit Outsourcing proves bad for business

• Lizards of Ozy The chameleons at failed Ozy Media are turning orange

• A Harvard Business Blunder Graduates duped in a ridiculous Ponzi scheme

• Con Air Fly the felonious skies with Boeing

• Smell the Glove Milwaukee Tool reeks of forced Chinese labor, lawsuit alleges

• Fisker the Frisker An electric car entrepreneur offers a valuable business lesson: If at first you don't succeed, file, file again

• Musk Mouth It takes a genius a long time to address an obvious blunder

• Sorry Seems To Be the Hardest Word A simple apology proved difficult for crypto tycoon Do Kwon after the $45 billion collapse of his Terraform Labs

• Epoch Crimes The finance chief of a far-right publication allegedly went far wrong

• Ticketbastard Live Nation deserves every ounce of contempt it has provoked

• How to Boil a Lobster Endless shrimp didn't drive this soggy seafood chain into bankruptcy

• Take Out Your Trash The Federal Deposit Insurance Corp. insures bank deposits, bullies, racists and sexual predators

• Sam in the Slam FTX founder Sam Bankman-Fried invests in beans and rice

• Billionaire Prison Time Here's how a hedge fund honcho managed incarceration

• ‘Bitcoin Jesus’ Saves Crypto promoter charged with dodging $48 million in taxes

• The Power of Red Failed Republic First Bancorp learns that 'fans' are fickle

• Our Deadhead Fed Head Jerome Powell has flubbed inflation

• Google This Billionaire A Silicon Valley legend makes his dumbest trade ever

• Don’t Listen to Sam Boeing engineer warns of 'catastrophic accidents and passenger fatalities'

• Calling Spirit Airlines on the Ouija Board Will customers miss this carrier if it dies?

• Musk’s F-Ing Message The genius of our age is scaring Tesla's customers

• A CEO Who Made His Bed Mattress Firm's latest buyer may be short on memory foam

• They Bought Crypto From a Klepto FTX founder Sam Bankman-Fried's trail of tears spills into court

• What’s in Your Wallet? Abusing company credit cards is a common employee grift

• My Latest Blunder Lessons from an epic failure at The Messenger