3.10.2017

Terms of use

All the content and resources have been provided in the hope that it will be useful. Author do not take responsibility for any misapplication of it. The document is distributed in the hope that will be useful, but WITHOUT ANY WARRANTY.

Introduction

The document describes the methods how to perform analysis of the network capture (pcap) with the goal to detect anomalies, perform incident investigation or try to understand the network flow.

Wireshark UI

Wireshark UI is not itself well suited to perform more complex analytics tasks and it is becoming slow when working with larger pcap. Example of slow and not optimal taks is simple ordering/sorting of the packets in the larger pcap. Therefore the wireshark UI should be used more as packet viewer and not analytic tool. However some useful and specialized tools exists which are integrated in wireshark (e.g. SIP/VoIP tools, call flows and other).

tshark + Elasticsearch + Kibana dashboards

Kibana can be well suited for creating various dashboards for analyzing of the network capture. The pcap can be first imported into elasticsearch by using tshark. The description of tshark export into elasticsearch json is described here https://www.h21lab.com/tools/tshark-elasticsearch.

The process can be quickly summarized in the following steps:

1. Install Elasticsearch + Kibana

For example for Ubuntu follow this tutorial

2. Convert pcap into json

Example for all fields for Elasticsearch 5.X and older

tshark -T ek -r trace.pcap > trace.json

For Elasticsearch 6.X and newer

tshark -T ek --no-duplicate-keys -r trace.pcap > trace.json

3. (OPTIONAL) Delete current data in Elasticsearch

curl -XDELETE 'http://localhost:9200/packets-*'

4. Import data into Elasticsearch

cat trace.json | curl -XPUT http://localhost:9200/_bulk --data-binary @-

5. Access Kibana using web browser

Check if the data has been loaded by accessing Kibana UI

http://<elk_host>:5601

6. Create custom dashboard using Kibana Visualize and Dashboard functionality

In Kibana dashboards the different perspectives can be modelled.

The basic examples of the visualisation for common IP traffic from desktops could be following:

• Number of packets grouped by src IP

• Number of packets grouped by dst IP

• Number of packets grouped by dst port

• Number of packets grouped by dest URL

However also additional visualisations could be created including also sub-buckets and various types of charts using Kibana Visualization.

Python scripts for pcap anomaly detection

The following python scripts can be used to try to detect anomalies in pcap files by using machine learning.

https://www.h21lab.com/tools/anomaly-detection

The scripts are using various approaches. The input used is always json generated by tshark from input pcap. For more details follow the information on the web page and github repository.