tshark + Elasticsearch in Docker
8.10.2019
The below bash commands provides lightweight example how to use tshark + Elasticsearch to perform simple pcap analytic using Docker images.
- Docker Elasticsearch + Kibana images are initialized
- After the tshark is used to import the pcap into Elasticsearch
# ===========================================================================
# Download and run elasticsearch in Docker
# https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html
# ===========================================================================
$ docker pull docker.elastic.co/elasticsearch/elasticsearch:7.4.0
7.4.0: Pulling from elasticsearch/elasticsearch
b38629870fdb: Pull complete
4a2fc9d810b8: Pull complete
e2926999e93a: Pull complete
1bfda44c7b09: Pull complete
901547b54de2: Pull complete
7432709cdcf3: Pull complete
5b3b6dd860e9: Pull complete
Digest: sha256:ccacb1463adc6daee970ed45e34cc46c14ba22116b64d5d4fac58044dfd61e8c
Status: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.4.0
$ docker image ls
REPOSITORY TAG IMAGE ID CREATED SIZE
docker.elastic.co/elasticsearch/elasticsearch 7.4.0 dd156dd42341 11 days ago 859MB
...
$ docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.4.0
# ==========================================================
# Download and run Kibana in Docker
# https://www.elastic.co/guide/en/kibana/current/docker.html
# ==========================================================
$ docker pull docker.elastic.co/kibana/kibana:7.4.0
7.4.0: Pulling from kibana/kibana
b38629870fdb: Already exists
5aa3b7be7b30: Pull complete
fd526df94d20: Pull complete
aa8dd3e3c5d9: Pull complete
ddc3f7f261d5: Pull complete
eb3fcf01993a: Pull complete
cb75917081c1: Pull complete
90c451fb58e9: Pull complete
d67b5b54b5d6: Pull complete
ccce49090460: Pull complete
Digest: sha256:a12bd702c9844b102f6bb81336a9c6a2c5b0286eebb4586e49cf53155ac92bcf
Status: Downloaded newer image for docker.elastic.co/kibana/kibana:7.4.0
$ docker container ls
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
84014d8108f9 docker.elastic.co/elasticsearch/elasticsearch:7.4.0 "/usr/local/bin/dock…" 6 seconds ago Up 5 seconds 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp serene_feistel
# link here points to elasticsearch container ID from the previous command
$ docker run --link 84014d8108f9:elasticsearch -p 5601:5601 -d docker.elastic.co/kibana/kibana:7.4.0
dd80900bad294cc2cd90db72bd1334c233dadd1a5c730da219f97353ef057023
# open browser to access Kibana UI
http://localhost:5601
# =========================================
# Decode pcap and import into Elasticsearch
# =========================================
# OPTIONAL: install more recent wireshark in Ubuntu
$ sudo add-apt-repository ppa:wireshark-dev/stable-staging
$ sudo apt-get update
$ sudo apt-get dist-upgrade
# import pcap
tshark -T ek -x -r ./http.cap > ./http.json
curl -H "Content-Type: application/x-ndjson" -XPOST http://127.0.0.1:9200/_bulk --data-binary @http.json
# OPTIONAL: delete index after pcap analysis
curl -XDELETE http://127.0.0.1:9200/packets-*
# Now the Elasticsearch should contain the data and it should be possible to access it over Kibana
# ================================================
# OPTIONAL: generate and add Elasticsearch mapping
# ================================================
# Note: that the current tshark output is not compatible with latest Elasticsearch and the import produce errors.
# So manual modification would be required of the mapping/template file.
# generate mapping
tshark -G elastic-mapping --elastic-mapping-filter ip,udp,dns > elastic.mapping
# load mapping
$ curl -H "Content-Type: application/json" -XPUT 'http://localhost:9200/packets-XXXX-XX-XX/_mapping' -d @elastic.mapping
# check mapping
curl -XGET 'http://localhost:9200/packets-XXXX-XX-XX/_mapping'
# or load template
curl -H "Content-Type: application/json" -XPUT 'http://localhost:9200/_template/packets' -d @elastic.mapping