Incident Response Talent

by Walker

For those who are getting into the computer security profession or looking to change focus, one job worth considering is in Incident Response (IR).

IR is a rewarding career where you get to help people by being a detective and problem solver, going on the defense and offense against malicious actors, and constantly learning new technology and attack methods.

I have been in the incident response field for over ten years and currently manage a team that works with clients.  As a manager, one of the challenges I face is finding new talent; I encourage everyone to get into the field.

For people starting out, I look for recent college graduates, SOC analysts, newly minted certificate holders from places like SANS and Security+, or relevant experience.  For more senior talent, technical and soft skill experience is weighted much more than college degrees and certifications.  Individuals do not need a background in incident response but should have some of the criteria listed below.

At its core, incident response requires a mix of strong communication skills both verbal and written, project management, a healthy dose of curiosity and problem solving, a drive to keep learning, and technical skills.  Many of these skills can be learned on the job with experience.  I was terrible at incident calls starting out, fumbling for the correct questions, feeling insecure in my decisions, intimidated with the audience.  With practice, these soft skills got better where they are now second nature, allowing me to focus on more technical problems.

Technical skills are listed last because there is no one skill needed for an IR team.  A strong IR team will be staffed to address major technology stacks in a corporate or client environment.  My team has Windows and Linux experts, experts in AWS and Azure, Windows forensics experts, experts in malware analysis.  It is good to have a wide breadth of knowledge, but realistically no one person can be an expert in all technologies.  I suggest to younger staff that they explore many topics to find ones they are passionate about.

Communication skills are very important.  Victims of cyber incidents are often in a heightened state of anxiety.  Attacks are stressful, especially if they involve a potentially business-ending event.  A calm and steady IR lead may help instill confidence that the situation is under control.  An IR team often communicates directly with leadership and heads of companies.  In the same call, you could be talking to the tech lead, head of legal, and the CEO at the same time.  Knowing how to customize your narrative for each of these individuals is important in explaining the entire situation.  The CEO will need different information than the tech lead, though it all stems from the same incident.

Project management is a major aspect of incident response.  An incident involves many moving parts, log and artifact collection, business impact analysis, communications, legal analysis, etc.  IR often schedules meetings with stakeholders, assigns and follows up on action items, and conducts or leads technical analysis.  The IR team lead must keep track of all these parts moving, documenting all steps and decisions taken, often with multiple incidents occurring at the same time.  An IR team lead must keep all these threads managed, otherwise the incident could quickly get out of control.

Written- and detail-orientated skills are essential.  Every scrap of evidence should be written down.  You may come across an important IP address in thousands of lines of logs that will be quickly forgotten.  A post-incident report describes how the incident happened, what was done to remediate the issue, what was done to bring the business back online, and lessons learned.  A fact-based and accurate report is essential to help make sound business decisions that will hopefully prevent the next incident and leave the business more secure from a technical and legal posture.

Curiosity and problem solving skills are a must for IR.  Depending on the incident, you may spend countless hours pouring through log files, and correlating IP addresses, accounts, and network pipes across systems for signs of lateral movement.  You may have to review decompiled malware to find Indicators of Compromise (IOC) that might indicate source and function.  Insider threat response may have you scour Windows file systems for evidence of fraud and criminal activity.  In your downtime, you may develop new tools, scripts, and processes to make these activities more efficient, or run threat hunting programs.

You have to have the patience to review this seemingly endless supply of data, to have the drive or voice in the back of your head pushing you to find the needle in the haystack.  Most lines of inquiry are dead ends, but occasionally you find the nugget of evidence that brings the whole incident into focus.  That is an unbelievably fantastic feeling! My favorite incident is one that I have not seen before, that challenges my technical and problem solving skills.

An IR team must never stop learning.  The security and technical landscape is always evolving, with threat actors constantly changing tactics and finding new ways to compromise people and systems.  IR teams constantly practice response activities, including reviewing and updating runbooks; conducting tabletop exercises; and teaching each other new topics, methods, and technology.

Finally, I wanted to address an issue that has plagued our industry: burnout.  Incident response is a 24/7 job.  There are often times of immense stress, unbelievably short deadlines, and multiple incidents to juggle at one time.  A well staffed and managed team spreads work so that no one person is responsible for being on call 24-hours-a-day.  Burnout can be avoided if management provides the support framework that allows individuals to feel safe, thrive, feel appreciated, and maintain a healthy work/life balance.

When interviewing for an IR position, ask about the program maturity, staffing levels, responsibility matrix, internal communication pathways, continuing education opportunities, and how often people have to work on nights and weekends.  This should hopefully give you the full picture before you walk into the next position.

You may find me on Mastodon at infosec.exchange/@walker where I talk about security, sports, and other random topics.