Analyzing Network Captures
3.10.2017
Terms of use
All the content and resources have been provided in the hope that it will be useful. Author do not take responsibility for any misapplication of it. The document is distributed in the hope that will be useful, but WITHOUT ANY WARRANTY.
Introduction
The document describes the methods how to perform analysis of the network capture (pcap) with the goal to detect anomalies, perform incident investigation or try to understand the network flow.
Wireshark UI
Wireshark UI is not itself well suited to perform more complex analytics tasks and it is becoming slow when working with larger pcap. Example of slow and not optimal taks is simple ordering/sorting of the packets in the larger pcap. Therefore the wireshark UI should be used more as packet viewer and not analytic tool. However some useful and specialized tools exists which are integrated in wireshark (e.g. SIP/VoIP tools, call flows and other).
tshark + Elasticsearch + Kibana dashboards
Kibana can be well suited for creating various dashboards for analyzing of the network capture. The pcap can be first imported into elasticsearch by using tshark. The description of tshark export into elasticsearch json is described here https://www.h21lab.com/tools/tshark-elasticsearch.
The process can be quickly summarized in the following steps:
1. Install Elasticsearch + Kibana
For example for Ubuntu follow this tutorial
2. Convert pcap into json
Example for all fields for Elasticsearch 5.X and older
tshark -T ek -r trace.pcap > trace.json
For Elasticsearch 6.X and newer
tshark -T ek --no-duplicate-keys -r trace.pcap > trace.json
3. (OPTIONAL) Delete current data in Elasticsearch
curl -XDELETE 'http://localhost:9200/packets-*'
4. Import data into Elasticsearch
cat trace.json | curl -XPUT http://localhost:9200/_bulk --data-binary @-
5. Access Kibana using web browser
Check if the data has been loaded by accessing Kibana UI
http://<elk_host>:5601
6. Create custom dashboard using Kibana Visualize and Dashboard functionality
In Kibana dashboards the different perspectives can be modelled.
The basic examples of the visualisation for common IP traffic from desktops could be following:
Number of packets grouped by src IP
Number of packets grouped by dst IP
Number of packets grouped by dst port
Number of packets grouped by dest URL
However also additional visualisations could be created including also sub-buckets and various types of charts using Kibana Visualization.
Python scripts for pcap anomaly detection
The following python scripts can be used to try to detect anomalies in pcap files by using machine learning.
https://www.h21lab.com/tools/anomaly-detection
The scripts are using various approaches. The input used is always json generated by tshark from input pcap. For more details follow the information on the web page and github repository.