5 Lessons Learned as Incident Commander of the Biggest Security Incident of My Career

Cybersec Café #69 - 05/27/25

May 27, 2025

I’ve been working in Detection and Incident Response for over half a decade now.

In that time, I’ve seen my fair share of security incidents (like anyone in this field), but it’s not every day that you find yourself leading the response to an incident with real, high-stakes consequences.

For confidentiality reasons, I won’t go into the specifics of what happened. The technical details aren’t the point here.

What I will share are the key lessons I learned from acting as Incident Commander during an investigation, response, and remediation that stretched just about a month.

My hope is that these takeaways will help you make smarter decisions now, and prepare you to lead with confidence if you ever find yourself in the same chair I did.

Let’s get into it.

Interdepartmental Relationships are Important

In security, you’ll never have all of the context needed to fully understand the business impact of an incident, or how to best respond, resolve, and recover to minimize disruption.

That’s why strong relationships across departments matter. You’ll often need to lean on colleagues in engineering, IT, legal, or product to move quickly, safely, and effectively during an incident.

But the time to build those connections isn’t during a crisis, it’s long before.

Start early. Reach out to people in other teams. Look for opportunities to collaborate on open tickets or jump into projects where your disciplines might intersect.

The primary goal here is to build rapport. And even small contributions can go a long way in building trust. This could be:

Submitting a pull request yourself rather than just assigning a ticket.
Sharing thoughtful input during project discussions.
Providing security reviews.
Proactively flagging and helping to address security concerns before they escalate.

Building rapport, chemistry, and trust with your team is important for if/when things go wrong.

You want to have faith that the people around you can do their job, and more importantly, you want them to have faith you can do yours.

Trust Your SMEs

Security Incidents aren’t the time for nitpicking or proving how much you know.

If you’ve done the work of building interdepartmental relationships, you should know who your go-to Subject Matter Experts (SMEs) are. When things break, bring in the people you trust and trust those people to do their job.

It's an unrealistic expectation that as a security engineer, you should understand how everything works in the environment: infrastructure, architecture, app behavior, microservices, business logic - it’s too much for any one person.

But that’s what SMEs are for.

Your job as Incident Commander isn’t to solve every technical problem yourself - it’s to orchestrate the right people toward the right outcome. That means putting your ego aside and giving space for the people who do know the answers to speak and act quickly.

Security incidents are a test of leadership, not just knowledge. The fastest path to resolution will generally be surrounding yourself with capable people and enabling them to achieve the outcome.

And when you’ve already started building those interdepartmental relationships, you’ll know exactly who you want in the room when things go sideways.

- Today’s Sponsor -

Selecty is a database-agnostic, sidecar query assistant - built to integrate seamlessly into your workflow without dulling your edge. Generate smart, contextual queries, optimize them to your use case, break them down into plain English, and debug faster than ever - all in one sleek interface. Check it out!

Learn More

Stay High Level

As a security professional, your instinct will be to dive into logs and hunt for the root cause. It’s a natural and useful reflex, but when you’re the Incident Commander, your role changes.

Your responsibility is to stay above the weeds and maintain a holistic view of the incident.

There are going to be multiple work streams running in parallel, and it’ll be easy to be pulled into one high-priority thread and lose sight of the bigger picture. What your team really needs is someone orchestrating the response - prioritizing, delegating, and keeping the machine moving.

On smaller teams, you’ll probably still need to get your hands dirty, which is totally fine. But just make sure your deep dives don’t come at the expense of broader progress.

You’ll need to make quick decisions and delegate with conviction. Listen to your team, weigh their input, and pull the trigger once you’ve got enough information.

When possible, you’ll also want to bring in a scribe, or an Incident Deputy. They’ll act as your “right-hand man,” tracking actions, taking notes, documenting key decisions, and keeping the timeline intact. Their job is to support you throughout the incident and free you up to move across workstreams, support as a technical resource when needed, and guide the overall strategy without getting bogged down by documentation.

You’ve got to approach it with the mindset: “This is my incident.” Own it. Lead it. Don’t let go of the reins.

You Need Scalable Processes

Over the course of my career, I’ve come to realize that Incident Management, specifically for a Security security use-case, is an extremely neglected area of cybersecurity.

Sure, there are incident management tools out there, but most are designed with Infrastructure or DevOps in mind and don’t really provide what you need for a complex security incident.

That’s why so many security teams end up building homegrown solutions to handle documentation, timelines, task tracking, artifact management, and regular check-ins.

And while these duct-taped setups may work fine for smaller or “standard” incidents, the cracks start to show when things scale. Here are a few common tools and where they fall short:

Jira is great for task tracking, but there’s no cohesive way to get a detailed view of the work streams, artifacts, and timeline in one place.
Google Docs is great for real-time collaboration, but the longer the incident runs, the harder it gets to navigate. Important details get buried fast.
Google Sheets are great for breaking down tasks into smaller efforts, but they hit limitations quickly when you’re juggling multiple work streams or cross-functional input.
Slack/Teams/Discord are essential for real-time async communication, but good luck finding that one key update buried in a noisy thread when you can’t remember any exact terminology to search (if you know, you know).

And don’t forget those daily check-ins. Depending on the severity of the incident, you may have one, two, or even three per day. As Incident Commander, you’re responsible not just for scheduling and facilitating those calls, but also for capturing meeting notes and linking out to relevant artifacts. That’s a heavy lift without the right process in place.

I covered some of this in my How to Create Incident Response Documentation article, and I think it’s important to stress-test your process through Tabletop Exercises (TTXs). It’s the only way to know if your documentation and workflows hold up under pressure.

While I was extremely thankful to have that framework during this incident - I realized I pushed the limits for how much a Google Doc could really scale. This incident was easily 10x larger than anything I’ve dealt with before, and while the process worked, it showed its limits.

The good news? I’m working on a solution to fix that.

If you’re tired of relying on scattered tools and fragile workflows, keep an eye out - I’ll be releasing something to help soon, so make sure you’re subscribed so you don’t miss it.

Post-Mortem is Just as Important as the Response

Once you’ve taken an incident from detection to investigation to remediation, your job isn’t over. It’s time to conduct a Post-Mortem (or Retro, After-Action Report - whatever name you prefer).

This is where you reflect, evaluate, and identify opportunities to improve.

As Incident Commander, it’s your responsibility to facilitate the conversation with the goal of understanding how the team performed, identifying what went well, and most importantly, where things broke down.

Since you had the most holistic view of the incident, some prep work is essential. Come prepared with the right questions. But, when the meeting starts - listen. The people in the trenches responding to the incident experience the pain points, the delays, and the successes. Let them drive the discussion.

If you skip these meetings or treat them as box-checking exercises, your incidents will continue to pile up. Patterns will repeat. Risks will compound. And the team will fatigue.

That’s a recipe for burnout.

Your top priority should be Root Cause Analysis (RCA). If you don’t truly understand what caused the incident, how do you expect to prevent it from happening again? If the RCA wasn’t completed during the incident already, it must be the first priority after recovering.

Once the root cause is nailed down, shift into building your Remediation Roadmap. Make every improvement item actionable and assign a clear owner. The tasks need to be prioritized, especially in the case people are assigned more than one. Don’t let the follow-ups vanish into the ether, either. Document them, assign them, and keep them visible.

I like to close out every incident incident with a final message in the Slack channel composed of a few items:

A thank you to everyone who helped respond.
Public recognition of key wins or contributions.
A summary of action items and their assigned owners.

The Post-Mortem isn’t just about documentation - it reinforces interdepartmental relationships and a culture of accountability, collaboration, and continuous improvement.

💬 If you’ve been an Incident Commander and have some knowledge to share, drop it below for the Cybersec Café community!

The Cybersec Café

Discussion about this post