Network Forensics in a 10G World

Network Forensics
in a 10G World
WHITE PAPER
With highly utilized networks, capturing network traffic with individual SPAN ports and taps typically results in spotty
overall visibility of your network. In today’s 10 Gigabit (10G) world, you need a purpose-built network forensic
solution in place capturing ALL network data, 24x7, to ensure a stable and safe network. Network forensics isn’t
just about uncovering and analyzing security breaches; it can and should be used every day to examine far more
common issues on your network, like spikes in utilization, drops in VoIP call quality, and increased latency, whether
network or application.
WildPackets, Inc.
1340 Treat Blvd, Suite 500
Walnut Creek, CA 94597
925.937.3200
www.wildpackets.com
Network Forensics in a 10G World
Introduction...............................................................................................3
Understanding the Unique Challenges of Highly-Utilized
10G Networks...........................................................................................3
Establishing Guidelines for Ongoing Network Data Collection.................5
Capture.............................................................................................5
Storage..............................................................................................6
Analysis.............................................................................................6
Additional Challenges for Real-Time Protocols on
Highly-Utilized 10G Network Segments...................................................8
Conflicting Demands of Traditional (TCP/IP) Data Analysis and VoIP
Analysis.............................................................................................8
Capturing and Analyzing Mixed Network Data..................................9
WildPackets Network Forensic Solutions.................................................9
Learning More................................................................................. 11
About WildPackets, Inc................................................................... 11
Conclusion..............................................................................................12
www.wildpackets.com
WHITE PAPER
2
Network Forensics in a 10G World
Introduction
Network forensics is the capture, recording, storage, and analysis of network events. Typically, network forensic tools
employ simple and complex filters to mine stored data to reveal anomalies (what caused them and what the results
were on network performance).
The common perception is that network forensics is solely used to discover the source of security attacks. With the
increasing deployments of 10G – and soon 40 Gigabit (40G) – gear, network forensics can and should be used every
day to examine far more common issues on your network, like spikes in utilization, drops in VoIP call quality, and
increased latency, whether network or application.
Think of network forensics as the ‘network time machine’ that helps you with everything from identifying the source
of data leaks to pinpointing the source of intermittent performance issues. For example, an incident occurred on your
network and you need to find where it happened and why it happened. Situations like this make it important to have
a network forensic solution in place, because your monitoring data alone will not provide sufficient detail to tell you
where the problem occurred. Yes, simple monitoring tools might alert you to a problem, but without a recording of the
actual network data you may never know what caused your network to fail. With network forensics, you can capture
and handle problems that occurred hours or days ago. At 10G speeds, this isn’t easy to accomplish, but with the right
solutions you’ll make quick work of it.
Understanding the Unique Challenges of Highly-Utilized
10G Networks
Your entire network infrastructure isn’t impacted equally when you start seeing 10G speeds. Device monitoring
is basically the same at 10G as it was at 1G. In flow record monitoring, you’ll see incremental increases as more
applications such as video and VoIP start to take advantage of the increased bandwidth. With deep packet inspection,
however, you’ll see a big impact almost immediately.
“Packet monitoring really is the most definitive, most complete source of performance data you can get for managing
networks and for troubleshooting in particular.”
— Jim Frey, Managing Research Director, Enterprise Management Associates, Inc.
Traditionally, you had a lot of flexibility with network analysis. Traffic would stream across your network until a problem
arose, at which time, you would connect your network analyzer, start a trace, and if the issue was an ongoing
problem, reproduce and analyze it.
www.wildpackets.com
WHITE PAPER
3
Network Forensics in a 10G World
Figure 1: Typical Network Analysis Workflow
Let it Roll!
Alerts/Alarms
NO
User
Complaints
Problem?
YES
Connect the
Analyzer
Start a Trace
Reproduce if
Necessary
For traditional network analysis, you could use almost any network interface card (NIC) and almost any computer
both to capture and analyze your traffic in real-time. Little or no special hardware was needed, and there was little to
no impact on existing network traffic while you ran your capture and conducted your analysis. For 10G, none of this
is true anymore. Network analysis at 10G requires you to be proactive. Before a problem occurs, you need to have
identified key analysis points and have deployed a 24x7 monitoring solution.
Figure 2: 10G Network Analysis Workflow
Identify Key
Analysis Pts
Deploy 24x7
Monitoring
Alerts/Alarms
NO
Problem?
YES
Rewind Data
www.wildpackets.com
Analyze
Tune if
Necessary
WHITE PAPER
4
Network Forensics in a 10G World
On highly utilized 10G links there’s too much traffic going by for you to analyze on the fly. With 10G, traditional
network analysis no longer works for a variety of reasons:
•Traditional NICs, such as 10/100 or 10/1000, are not up to the task. These cards were designed to route
and move packets, not capture those packets reliably.
•Processing power is a limiting factor. You’re not going to be able to connect your laptop with network
analysis software to a 10G link and expect it to process the number of packets that will flow through there in a
given second or millisecond.
•Storage capacity is a limiting factor. Traditional network analysis was heavily dependent on the file
system, requiring a lot of overhead to store each individual file. Each packet capture was typically stored
as an individual file, straining the storage capacity of the appliances. Network forensics attempts to take full
advantage of disk space, dedicating most storage to the capture and storage of packet data.
•I/O bus and disk write speeds are a limiting factor.
•Large volumes of data are generated quickly, leaving you searching for a needle in a haystack. Analysis
requires that you drill into the data. With network forensics, expert filters help you isolate problems in advance,
reducing the amount of data you need to sift through.
•Line rate doesn’t mean lossless packet capture at 10G.
Establishing Guidelines for Ongoing Network Data Collection
Capture
To increase the odds that your investigation will be successful, record every piece of network traffic – all emails, all
database queries, anything that is traversing on your network – to a single repository that can be examined after
the fact. You can do this with a purpose-built network forensic solution, such as the WildPackets TimeLine network
recorder, which is capable of capturing all of your 10G network data, 24x7, at line rates up to 11.2Gbps.
Another way to simplify the collection of 10G network data for detailed analysis is using an aggregation device
instead of connecting an appliance directly to the 10G network.
Figure 3: 10G Network Data Capture
10Gb/s
10G Network
VSS V24
1Gb/s
1Gb/s
www.wildpackets.com
WHITE PAPER
5
Network Forensics in a 10G World
An aggregation device, such as the V24 tap from VSS Monitoring, taps into the 10G network itself with a 10G card
and splits the 10G traffic into various streams. It can redirect the 10G traffic in its entirety to a TimeLine Network
Recorder or split the traffic into individual 1G streams and send those to devices that might not be designed to handle
10Gbps. Note that this could complicate your analysis as you’re splitting streams based on a given criteria, for
example by subnet or by protocol, to different appliances.
Storage
Being able to capture all of your 10G network data isn’t sufficient; you also need to store the data so that it’s available
when you begin your investigation. Let’s say we’re monitoring a fully utilized 1G network or 10G network that’s utilized
at 1Gbps. To simplify our calculation, let’s assume there’s no storage overhead.
1Gbps x 1 bits / 8 bytes x 60 s/min x 60 min/hr x 24 hr/day = 11 TB/day
If you have a 32TB appliance, you’re able to go back in time 2.9 days, which means if a problem occurs over the
weekend, you’ll probably be able to find out what happened and recreate it on Monday. You’ll have every packet that
was transmitted through the link that you’re monitoring, so data reconstruction at any level is possible. Once you get
to 3 days, you’ll begin losing the earliest packet data you captured.
Now let’s look at a 10x increase in traffic, 10Gbps on a fully utilized 10G network.
10 Gbps x 1 bits / 8 bytes x 60 s/min x 60 min/hr x 24 hr/day = 110 TB/day
If you still only have a 32TB appliance, now you’re only able to go back in time 7.0 hours. If a problem occurs
overnight, you might be able to find out what happened, but if the problem occurred over the weekend, you won’t
have the packet data you need for your investigation. This might be another reason you’d want to include an
aggregation tap in your network infrastructure. The tap would allow you to send packet data to multiple appliances for
retrieval at a later date. Or, this may be the time to consider connecting your network recorder to a SAN for additional
data storage.
Analysis
Knowing how you expect your network to be performing is all the more critical when trying to analyze highly utilized
10G segments. In advance of an investigation, you’ll want to know what’s in your network. If you’re already embroiled
in a complex network analysis firefight it’s too late to realize that your ability to assess “normal” conditions on the
network may be lacking.
To get a sense of “normal” conditions before trouble arises, you should perform and archive baseline measurements
across specific network traffic like HTTP and key business applications over typical cycles – like an hour, a day,
and a week, for the network as a whole. Other metrics to consider include understanding packet size distribution as
well as protocol and node usage over time, uncovering cycles in these metrics, which provide a “fingerprint” of your
utilization. That way you will always have a clear view of the network for comparison when trouble arises. Only after
convincing yourself that the basic data is in place and being collected and analyzed should you embark on detailed
analysis and drill-down of packet-level data.
www.wildpackets.com
WHITE PAPER
6
Network Forensics in a 10G World
When faced with an issue on your network, such as a spike in utilization or increased latency, whether network or
application, you’ll want to first analyze the essentials. The temptation is to try to capture and analyze everything,
especially when the source of the problem is not immediately known. You do, however, know certain things about
your network, which allows you to be selective in the analysis options you choose. Often a variety of conditions
can be immediately ruled out, and using these clues to limit the collection and analysis to only what is necessary
dramatically improves network analysis performance. For example, if you’re looking at a 10G network link, you’re
probably not capturing wireless traffic; turn off the wireless analysis. If your network forensics solution has VoIP or
Video analysis, you may be able to turn that off as well. Turning off analyses that aren’t relevant to your investigation
refines your search, making it more specific, and increases the processing power and throughput of the appliance
you’re using. Think about what you need so that you can maximize the disk space for the data you want.
Even after analysis has been streamlined to only essential areas of the network, data capture for network analysis
on 10G networks generates a great deal of data quickly, and managing the data becomes a significant challenge.
Effective analysis requires you know your limits, not just the available space for storage, but the processing limits of
your appliance as well as how many users can access the appliance concurrently and perform analysis.
Regardless of the system used, the data is typically stored for subsequent retrieval and post-capture analysis. The
two most common formats are standard packet files and databases. In either case, two metrics to manage closely
are file size and frequency of disk writes. Though intuition may lead you to think that the larger the file size the better,
this is often not the case as very large files require very large memory footprints to open. If the files are too large they
will be unworkable on the computer being used for analysis. Smaller files, however, typically lead to more frequent
disk writes, and this can rob the system of precious resources for performing the actual packet capture. Optimum
performance is achieved with a balance of these two demands, and this is different depending on the hardware
resources available. One rule of thumb to keep in mind is that if files are being created every 30 seconds or less, it’s
going to increase strain on achieving the maximum packet capture rate significantly. Starting with reasonable sized
buffers – 256MB buffer for packet capture – and files – 128MB – makes all the difference. After a few captures you’ll
quickly determine if either of these parameters can be better optimized for your system. Also, try to use the lowest
number of simultaneous captures as possible. In several systems, you’re allowed to create as many captures as you
want, but you need to remember that for each capture you open more memory is reserved for buffering and less is
available for data processing.
Something else to consider, is whether you’ll be performing real-time analysis or post-incident or forensics analysis.
Real-time analysis is tricky at 10G, but you can still use real-time data to pinpoint developing problems, highlighting a
period of time you want to look at further, drilling down later using forensic analysis.
Finally if you’re just doing network performance analysis or network performance tuning, you may not need the
packet payloads and can slice the payload from your data, significantly increasing the other data that you can store. If
you’re using your network recorder for network security in parallel with your IDS/IPS and they miss an attack, without
the packet payload you’ll be unable to replay exactly what happened.
www.wildpackets.com
WHITE PAPER
7
Network Forensics in a 10G World
Additional Challenges for Real-Time Protocols on
Highly-Utilized 10G Network Segments
Network analysis on highly utilized 10G network segments is already a complicated business, add time-sensitive
protocols like Voice over IP (VoIP) into the mix, with its associated complexities like Quality of Service (QoS)
configuration and MPLS segmentation, and you could have a real mess on your hands.
Conflicting Demands of Traditional (TCP/IP) Data Analysis and VoIP Analysis
VoIP and other real-time, interactive IP-based (RTIPC) communications require special handling, especially on a 10G
network, where the conflicting demands of RTIPC and traditional data are magnified. Traditional network applications
are very tolerant of jitter, latency, and even some degree of packet loss, potentially suffering only a gradual
degradation in network speed. VoIP and video, on the other hand, are very sensitive to these parameters. Levels of
jitter, latency, and packet loss that would be easily tolerated on a data network can be devastating on a converged
VoIP network. RTIPC traffic needs to be prioritized over other traffic to ensure successful delivery. Additionally,
problems with RTIPC traffic require at least some level of real-time analysis.
Prior to deploying VoIP or video, you must understand your network’s ability to accommodate the time-sensitive
traffic. What’s the current latency, jitter, and packet loss? Do you have QoS capabilities? What’s your network’s
current bandwidth utilization – is there any room for VoIP?
Let’s take a quick look at latency, jitter, and packet loss and their effect on VoIP traffic. Some latency is always going
to exist in your network. Simply put, latency is the time it takes data to get from point A to point B and includes queue
latency, decision latency, network propagation latency, encoding/decoding, compression/decompression, and jitter
buffer latency. The ITU recommends a maximum one-way delay of 150ms for VoIP. For some network segments,
especially WAN circuits, elevated latency may be a way of life; it can typically stay below the recommended threshold
for VoIP, but contributes considerably to the overall latency for a given call.
Jitter is closely related to latency, it’s a variation in the latency. For example, a G.711 packet is sent every 20ms.
As the packets traverse the network the spacing between them can drift, so that when they reach the destination
the delivery of packets is not at a regular 20ms interval, but may vary. This deviation of the packet spacing from the
norm is jitter. In general, you should strive for zero jitter. Jitter values up to about 100ms may be managed by the
jitter buffer in your VoIP handset, but keep in mind the recommended maximum one-way delay of 150ms – this is the
maximum budget for all forms of delay, including that introduced by a jitter buffer. Packets with jitter that exceeds the
jitter buffer are simply dropped, creating gaps in the conversation.
The third factor, packet loss, is most commonly caused by packets being dropped due to physical layer corruption,
congestion without adequate QoS provisions, or jitter buffer discards due to excessive latency. Packet loss often
occurs in “bursts.” To maintain adequate quality, the number of consecutive received packets needs to be less than
the required minimum number of packets. Gmin for VoIP = 16 and for video services approximately 64 to 128. The
more “bursty” the packet loss is, the worse the quality of the call.
Finally, while your 10G network link may be able to support a large number of concurrent calls, the addition of one
www.wildpackets.com
WHITE PAPER
8
Network Forensics in a 10G World
more call can cause poor quality for all calls. Unlike data, where only certain users may experience problems as
utilization spikes, with VoIP you can have 2 or 200 simultaneous calls without any quality problems, but when the
201st call is added, because all have priority, it tips your VoIP system over the edge. After deploying VoIP and video,
you must maintain a constant vigil to watch for these imminent troubles.
Capturing and Analyzing Mixed Network Data
Most network engineers are concerned about the amount of traffic on their networks: utilization (percentage of
bandwidth) and throughput (bits or bytes per second). With VoIP and video, you also need to be concerned about
individual utilization components.
•How much bandwidth and throughput can be attributed to each application or process? This identifies which
application traffic may need to be tuned or controlled.
•How well or poorly will the baseline (trended) behavior of each application interact with VoIP? Also consider
the reverse case, how will VoIP impact your existing applications?
With VoIP and video, the quality of your network traffic is potentially more important than its quantity. Many traffic
streams are “bursty” in nature. Prolonged rises in utilization may decrease the number of calls that can occur
simultaneously. Sharp spikes may also cause very noticeable quality issues with ongoing calls. Remember that with
VoIP performance degradation is abrupt. Be sure that your baseline monitoring considers not only averages and longterm trends, but also short-term peaks and dips that characterize your traffic flow.
WildPackets Network Forensic Solutions
Network forensics, or your ‘network time machine,’ helps you pinpoint the source of intermittent performance issues
and conduct investigations to identify the source of data leaks, HR violations, or security breaches. Get ready now –
before a specific event actually happens – so digital evidence is collected and ready to help you find that needle in
the haystack.
With WildPackets Network Forensic solutions, data is always available for reconstruction and easy analysis of
intermittent issues, cyber attacks, and network security or data breaches. All pertinent network traffic is collected in
a single location, rather than scattered across the network. Data is captured in a common data format and does not
need to be transferred or translated in any way for analysis.
Using our network forensic data mining tools, network engineers have the data they need to identify and fix problems
users are complaining about that only occur intermittently, and security teams can reconstruct the sequence of events
that occur at the time of a network breach or cyber attack and get the complete picture.
While other network forensics products force you to capture with one product, then transfer gigabytes or terabytes of
data to another tool for analysis, WildPackets Network Forensic solutions enable you to analyze data at the point of
capture, and eliminate the need for large data transfers that consume time and bandwidth. By utilizing Intelligent Data
Transport™, WildPackets Network Forensic solutions minimize traffic loads on the network and let you find the data
you’re looking for, quickly and easily.
www.wildpackets.com
WHITE PAPER
9
Network Forensics in a 10G World
24x7 Enterprise-Wide Network Monitoring and Troubleshooting
Network Operations Center
Expert Analysis, 10/100, Gig, 10G, WLAN, Voice
Server
Farm
10/100, Gig, 10G, WLAN Expert Analysis
VoIP Console
Server
WAN
AP
Controller
AP
Corporate Wireless Users
Laptop, Tablet, VoFi
Access Point (AP)
AP
Server
Remote Office
Corporate Campus
Figure 4: WildPackets Network Forensic Solutions
24x7 access to ALL network data and network forensics mining tools lets you:
•Ensure network and security data are captured 24x7 and not sacrificed when SPAN ports are needed for other
applications
•Reduce Mean-Time-To-Resolution (MTTR) by eliminating the time consuming step of having to reproduce
problems before they can be analyzed and responding to issues in real-time, often solving issues before
mission critical applications are impacted
• Understand service-level compliance within your organization
•Comply with government regulations and Human Resources policies by auditing and tracking all network
activity
If you’re not already reaching for network forensics to address a pesky intermittent network issue, benchmark
application performance for SLAs, or investigate a data breach, you should be. WildPackets Network Forensic
solutions offer the following capabilities:
• Comprehensive data collection: Hours or even days of network traffic – anything that crosses the network,
whether email, IM, VoIP, FTP, HTML, or some other application or protocol – collected by a single system and
stored in a common, searchable format. Terabytes of data available through a single interface.
•Flexible data collection: Collect all data on a network segment for future inspection or focus on a specific
user or server.
www.wildpackets.com
WHITE PAPER 10
Network Forensics in a 10G World
•High-level analysis: Eliminate the need for brute-force analysis across disparate data sources with access to
WildPackets’ award-winning Expert Analysis, graphical reports, and application performance scoring.
With WildPackets Network Forensic solutions in place, you can conduct various types of forensic investigations:
•Network performance benchmarking for detailed reporting on network performance, bottlenecks, activates, etc.
•Network troubleshooting for handling any type of network problem, especially those that happen intermittently.
•Transactional analysis for providing the “ultimate audit trail” for any transactions where server logs and other
server-based evidence doesn’t provide a thorough picture of a transaction. Remember, packets don’t lie!
•Security attack analysis for enabling security officers and IT staff to characterize and mitigate an attack that
slipped past network defense such as a zero day attack.
Learning More
•“Network Forensics 101: Finding the Needle in the Haystack” Think network forensics is just for security? This
white paper defines network forensics, dispels some common misperceptions, and describes what you could
and should be using it for.
•“Network Forensics: How to Optimize Your Digital Investigation” Time is of the essence when your
organization’s network security comes under attack or you’re investigating a business-critical issue such as
a data breach or an industry regulation violation. This white paper shows how a network forensic solution
with four basic elements – data capture, data discovery, data analysis, and data recording – reduces your
mean time to resolution. Also covered are the three phases of digital investigation: separating network data,
performing packet drill down, and enumerating the data.
•“Network Forensics in a 10G World” With highly utilized networks, capturing network traffic with individual
SPAN ports and taps typically results in spotty overall visibility of your network. In today’s 10 Gigabit (10G)
world, you need a purpose-built network forensic solution in place capturing ALL network data, 24x7, to
ensure a stable and safe network. This white paper identifies the unique challenges of highly-utilized 10G
networks, establishes guidelines for ongoing network data collection, and addresses the conflicting demands
of traditional (TCP/IP) data analysis and VoIP analysis.
All of these white papers and more can be found at www.wildpackets.com under the Resources section.
About WildPackets, Inc.
WildPackets develops hardware and software solutions that drive network performance, enabling organizations of all
sizes to analyze, troubleshoot, optimize, and secure their wired and wireless networks. WildPackets products are sold
in over 60 countries and deployed in all industrial sectors. Customers include Boeing, Chrysler, Motorola, Nationwide,
and over 80 percent of the Fortune 1000. WildPackets is a Cisco Technical Development Partner (CTDP).
To learn more about WildPackets solutions, please visit www.wildpackets.com, or contact WildPackets Sales:
sales@wildpackets.com or (925) 937-3200.
www.wildpackets.com
WHITE PAPER 11
Network Forensics in a 10G World
Conclusion
As networks get faster and more complex, SPAN ports and taps become more unreliable and oftentimes fail to
provide the data needed for network analysis when you need it most. Also, with 10G and now even 40G networks
in place, it can even be too much to ask for a single network management appliance to handle these extremely high
data rates while still providing the necessary detailed analysis. It is essential in today’s high-speed networks to have
a complete network analysis solution in place, one which employs both a network traffic capture system, as well as
network analysis appliances, to help you quickly identify and solve problems at the network level, as well as achieve
compliance at the business level.
Finding an issue on your 10G network is like trying to find a needle in a stack of needles – many issues can look the
same at first glance. Instead of simply relying on traditional network analysis, think about network forensics. Network
forensics can be a powerful tool, eliminating the time it takes you to recreate the problems that occur on your network.
If you are dealing with a 10G network that is highly utilized, a network forensics solution is no longer nice to have; it’s
a must have as SPAN ports and taps are inefficient when it comes to meeting your security, compliance, and network
analysis needs. In today’s 10G world, you need to have a system in place that can capture ALL network data, 24×7,
to ensure a stable and safe network.
www.wildpackets.com
WHITE PAPER 12