How to Prepare for and React to a Security Breach

Jay Botelho, Senior Director of Products at Savvius

Enterprise security teams have historically spent most of their time, human resources and money on defenses like firewalls and Intrusion Detection Systems (IDS) to protect and monitor the security of their networks. However, a quick look at the news will tell you that these barriers are far from foolproof. With breaches becoming more common (and costly), enterprise teams are turning to tools that help them respond quickly to security incidents as soon as the attack has been discovered. Network forensics looks at information such as log data, network flow and packet data to answer the question ‘How did the attackers get in?’. It’s similar to what you would expect a detective to do at a crime scene – look for clues to recreate the crime. The goal of network forensics is to identify the source of the breach faster in order to minimize the resulting damage, and to analyze them so that future attacks can be prevented.

Consider this example about port scans:

Port scans are attempts to detect and penetrate open server ports from a remote location. Every enterprise is subject to attacks like these on a daily basis. In most cases, the security appliances shrug off unwanted scans.

But in this instance a specialized scan hidden amongst the others detects a known vulnerability in a web server. The hacker then uses a known exploit to infiltrate the server and identify information like encrypted password files to retrieve and crack. Then they exfiltrate the data back to their attack server. The enterprises’ IDS detects the exfiltration and signals an alert.

The alarm has been sounded, and the security team knows about the attack, right? Perhaps not. IDS devices typically produce many alerts per day – sometimes hundreds, if not properly configured. It is common to receive over 500 alerts every day marked as “severe/critical,” yet a general lack of resources means that often they are only able to investigate and resolve 1 percent of those alerts. Most IT departments simply cannot respond to the deluge of alerts and false positives, which can allow real attacks to slip through unnoticed.

So what’s the solution? Examining network data such as network flow, TCP or IP events can help trained investigators eliminate false positives quickly. That leaves them with a reasonable number of potentially legitimate alerts to investigate. An effective network forensics tool will only capture network data associated with alerts, so investigators can easily focus on the data that matters.

Unfortunately, not all organisations are adequately equipped to investigate breaches. Access logs will indicate access attempts, but do nothing to highlight exploited vulnerabilities or malware-based attacks. System logs and network security logs (from a firewall, IDS, etc.) usually will not generate an urgent alert unless a login is preceded by several failed attempts, which clever attackers can easily avoid. The most useful information in network forensics is the original packet data.

In the above example, you may have noticed that the IDS only triggered once the stolen data was exfiltrated. The issue is that most tools today start capturing packet data only when the event has been triggered, which is too late to see which web server was attacked, which exploit was used and which port scan detected the vulnerability. Effective network forensics requires buffered data that can allow security investigators to examine the network activity immediately prior to and following the alert in question.

This brings us back to why network forensics is so important. Without the original packets to help piece together the cause of an alert, it takes significantly longer to find real breaches, meaning more stolen data and ultimately a greater cost to the company. It takes an average of 200 days to identify a breach and an additional 70 days to contain it. The average cost of a breach is almost £2.8 million.

The unfortunate truth is that no organisation is safe from attack. With that in mind, here are some critical steps every organisation should take to prepare for, and react to a security breach:

• Preparedness

Employees are sometimes the weakest link in security. It is important that you conduct regular training with employees on basic security best practices such as using strong passwords, how to identify phishing emails, and not plugging unknown devices into work machines.

• Identification

Automate the process of data collection so that it is easier to investigate and identify security events.

• Containment

Once a breach has been confirmed, determine exactly how far the problem has spread within the company’s network and minimize further damage by disconnecting affected systems and devices.

• Eradication

Resolve the root cause of the vulnerability and remove all traces of malicious code. Ensure that the flaw is completely resolved by running penetration tests and looking at server logs again to define whether other servers and devices might also be susceptible.

• Recovery

Restore all data and software from clean backup files. Monitor systems for any sign of weakness or recurrence.

• Lessons learned and remediation

Conduct a thorough post-mortem to analyze the incident and how it was handled. Identify prevention and response processes that can be improved.

About Jay Botelho:

Jay Botelho is the Senior Director of Products at Savvius, Inc., a leader in actionable network visibility for network performance management and security investigations. Jay holds an MSEE, and is an industry veteran with more than 25 years of experience in product management, product marketing, program management and complex analysis. From the first mobile computers developed by GRiD Systems to modern day network infrastructure systems, Jay has been instrumental in setting corporate direction and specifying requirements for industry-leading hardware and software products. He is based at Savvius’ headquarters in Walnut Creek, California.

Jay Botelho, Senior Director of Products at Savvius

Enterprise security teams have historically spent most of their time, human resources and money on defenses like firewalls and Intrusion Detection Systems (IDS) to protect and monitor the security of their networks. However, a quick look at the news will tell you that these barriers are far from foolproof. With breaches becoming more common (and costly), enterprise teams are turning to tools that help them respond quickly to security incidents as soon as the attack has been discovered. Network forensics looks at information such as log data, network flow and packet data to answer the question ‘How did the attackers get in?’. It’s similar to what you would expect a detective to do at a crime scene – look for clues to recreate the crime. The goal of network forensics is to identify the source of the breach faster in order to minimize the resulting damage, and to analyze them so that future attacks can be prevented.

Consider this example about port scans:

Port scans are attempts to detect and penetrate open server ports from a remote location. Every enterprise is subject to attacks like these on a daily basis. In most cases, the security appliances shrug off unwanted scans.

But in this instance a specialized scan hidden amongst the others detects a known vulnerability in a web server. The hacker then uses a known exploit to infiltrate the server and identify information like encrypted password files to retrieve and crack. Then they exfiltrate the data back to their attack server. The enterprises’ IDS detects the exfiltration and signals an alert.

The alarm has been sounded, and the security team knows about the attack, right? Perhaps not. IDS devices typically produce many alerts per day – sometimes hundreds, if not properly configured. It is common to receive over 500 alerts every day marked as “severe/critical,” yet a general lack of resources means that often they are only able to investigate and resolve 1 percent of those alerts. Most IT departments simply cannot respond to the deluge of alerts and false positives, which can allow real attacks to slip through unnoticed.

So what’s the solution? Examining network data such as network flow, TCP or IP events can help trained investigators eliminate false positives quickly. That leaves them with a reasonable number of potentially legitimate alerts to investigate. An effective network forensics tool will only capture network data associated with alerts, so investigators can easily focus on the data that matters.

Unfortunately, not all organisations are adequately equipped to investigate breaches. Access logs will indicate access attempts, but do nothing to highlight exploited vulnerabilities or malware-based attacks. System logs and network security logs (from a firewall, IDS, etc.) usually will not generate an urgent alert unless a login is preceded by several failed attempts, which clever attackers can easily avoid. The most useful information in network forensics is the original packet data.

In the above example, you may have noticed that the IDS only triggered once the stolen data was exfiltrated. The issue is that most tools today start capturing packet data only when the event has been triggered, which is too late to see which web server was attacked, which exploit was used and which port scan detected the vulnerability. Effective network forensics requires buffered data that can allow security investigators to examine the network activity immediately prior to and following the alert in question.

This brings us back to why network forensics is so important. Without the original packets to help piece together the cause of an alert, it takes significantly longer to find real breaches, meaning more stolen data and ultimately a greater cost to the company. It takes an average of 200 days to identify a breach and an additional 70 days to contain it. The average cost of a breach is almost £2.8 million.

The unfortunate truth is that no organisation is safe from attack. With that in mind, here are some critical steps every organisation should take to prepare for, and react to a security breach:

• Preparedness

Employees are sometimes the weakest link in security. It is important that you conduct regular training with employees on basic security best practices such as using strong passwords, how to identify phishing emails, and not plugging unknown devices into work machines.

• Identification

Automate the process of data collection so that it is easier to investigate and identify security events.

• Containment

Once a breach has been confirmed, determine exactly how far the problem has spread within the company’s network and minimize further damage by disconnecting affected systems and devices.

• Eradication

Resolve the root cause of the vulnerability and remove all traces of malicious code. Ensure that the flaw is completely resolved by running penetration tests and looking at server logs again to define whether other servers and devices might also be susceptible.

• Recovery

Restore all data and software from clean backup files. Monitor systems for any sign of weakness or recurrence.

• Lessons learned and remediation

Conduct a thorough post-mortem to analyze the incident and how it was handled. Identify prevention and response processes that can be improved.

About Jay Botelho:

Jay Botelho is the Senior Director of Products at Savvius, Inc., a leader in actionable network visibility for network performance management and security investigations. Jay holds an MSEE, and is an industry veteran with more than 25 years of experience in product management, product marketing, program management and complex analysis. From the first mobile computers developed by GRiD Systems to modern day network infrastructure systems, Jay has been instrumental in setting corporate direction and specifying requirements for industry-leading hardware and software products. He is based at Savvius’ headquarters in Walnut Creek, California.