tshark + Elasticsearch in Docker
8.10.2019
The below bash commands provides lightweight example how to use tshark + Elasticsearch to perform simple pcap analytic using Docker images.
Docker Elasticsearch + Kibana images are initialized
After the tshark is used to import the pcap into Elasticsearch
# ===========================================================================# Download and run elasticsearch in Docker# https://www.elastic.co/guide/en/elasticsearch/reference/current/docker.html# ===========================================================================
$ docker pull docker.elastic.co/elasticsearch/elasticsearch:7.4.07.4.0: Pulling from elasticsearch/elasticsearchb38629870fdb: Pull complete 4a2fc9d810b8: Pull complete e2926999e93a: Pull complete 1bfda44c7b09: Pull complete 901547b54de2: Pull complete 7432709cdcf3: Pull complete 5b3b6dd860e9: Pull complete Digest: sha256:ccacb1463adc6daee970ed45e34cc46c14ba22116b64d5d4fac58044dfd61e8cStatus: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.4.0
$ docker image lsREPOSITORY TAG IMAGE ID CREATED SIZEdocker.elastic.co/elasticsearch/elasticsearch 7.4.0 dd156dd42341 11 days ago 859MB...
$ docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.4.0
# ==========================================================# Download and run Kibana in Docker# https://www.elastic.co/guide/en/kibana/current/docker.html# ==========================================================
$ docker pull docker.elastic.co/kibana/kibana:7.4.07.4.0: Pulling from kibana/kibanab38629870fdb: Already exists 5aa3b7be7b30: Pull complete fd526df94d20: Pull complete aa8dd3e3c5d9: Pull complete ddc3f7f261d5: Pull complete eb3fcf01993a: Pull complete cb75917081c1: Pull complete 90c451fb58e9: Pull complete d67b5b54b5d6: Pull complete ccce49090460: Pull complete Digest: sha256:a12bd702c9844b102f6bb81336a9c6a2c5b0286eebb4586e49cf53155ac92bcfStatus: Downloaded newer image for docker.elastic.co/kibana/kibana:7.4.0
$ docker container lsCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES84014d8108f9 docker.elastic.co/elasticsearch/elasticsearch:7.4.0 "/usr/local/bin/dock…" 6 seconds ago Up 5 seconds 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp serene_feistel
# link here points to elasticsearch container ID from the previous command$ docker run --link 84014d8108f9:elasticsearch -p 5601:5601 -d docker.elastic.co/kibana/kibana:7.4.0dd80900bad294cc2cd90db72bd1334c233dadd1a5c730da219f97353ef057023
# open browser to access Kibana UIhttp://localhost:5601
# =========================================# Decode pcap and import into Elasticsearch# =========================================
# OPTIONAL: install more recent wireshark in Ubuntu$ sudo add-apt-repository ppa:wireshark-dev/stable-staging$ sudo apt-get update$ sudo apt-get dist-upgrade
# import pcaptshark -T ek -x -r ./http.cap > ./http.jsoncurl -H "Content-Type: application/x-ndjson" -XPOST http://127.0.0.1:9200/_bulk --data-binary @http.json
# OPTIONAL: delete index after pcap analysiscurl -XDELETE http://127.0.0.1:9200/packets-*
# Now the Elasticsearch should contain the data and it should be possible to access it over Kibana
# ================================================# OPTIONAL: generate and add Elasticsearch mapping# ================================================
# Note: that the current tshark output is not compatible with latest Elasticsearch and the import produce errors.# So manual modification would be required of the mapping/template file.
# generate mappingtshark -G elastic-mapping --elastic-mapping-filter ip,udp,dns > elastic.mapping
# load mapping$ curl -H "Content-Type: application/json" -XPUT 'http://localhost:9200/packets-XXXX-XX-XX/_mapping' -d @elastic.mapping# check mappingcurl -XGET 'http://localhost:9200/packets-XXXX-XX-XX/_mapping'
# or load templatecurl -H "Content-Type: application/json" -XPUT 'http://localhost:9200/_template/packets' -d @elastic.mapping
$ docker pull docker.elastic.co/elasticsearch/elasticsearch:7.4.07.4.0: Pulling from elasticsearch/elasticsearchb38629870fdb: Pull complete 4a2fc9d810b8: Pull complete e2926999e93a: Pull complete 1bfda44c7b09: Pull complete 901547b54de2: Pull complete 7432709cdcf3: Pull complete 5b3b6dd860e9: Pull complete Digest: sha256:ccacb1463adc6daee970ed45e34cc46c14ba22116b64d5d4fac58044dfd61e8cStatus: Downloaded newer image for docker.elastic.co/elasticsearch/elasticsearch:7.4.0
$ docker image lsREPOSITORY TAG IMAGE ID CREATED SIZEdocker.elastic.co/elasticsearch/elasticsearch 7.4.0 dd156dd42341 11 days ago 859MB...
$ docker run -p 9200:9200 -p 9300:9300 -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.4.0
# ==========================================================# Download and run Kibana in Docker# https://www.elastic.co/guide/en/kibana/current/docker.html# ==========================================================
$ docker pull docker.elastic.co/kibana/kibana:7.4.07.4.0: Pulling from kibana/kibanab38629870fdb: Already exists 5aa3b7be7b30: Pull complete fd526df94d20: Pull complete aa8dd3e3c5d9: Pull complete ddc3f7f261d5: Pull complete eb3fcf01993a: Pull complete cb75917081c1: Pull complete 90c451fb58e9: Pull complete d67b5b54b5d6: Pull complete ccce49090460: Pull complete Digest: sha256:a12bd702c9844b102f6bb81336a9c6a2c5b0286eebb4586e49cf53155ac92bcfStatus: Downloaded newer image for docker.elastic.co/kibana/kibana:7.4.0
$ docker container lsCONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES84014d8108f9 docker.elastic.co/elasticsearch/elasticsearch:7.4.0 "/usr/local/bin/dock…" 6 seconds ago Up 5 seconds 0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp serene_feistel
# link here points to elasticsearch container ID from the previous command$ docker run --link 84014d8108f9:elasticsearch -p 5601:5601 -d docker.elastic.co/kibana/kibana:7.4.0dd80900bad294cc2cd90db72bd1334c233dadd1a5c730da219f97353ef057023
# open browser to access Kibana UIhttp://localhost:5601
# =========================================# Decode pcap and import into Elasticsearch# =========================================
# OPTIONAL: install more recent wireshark in Ubuntu$ sudo add-apt-repository ppa:wireshark-dev/stable-staging$ sudo apt-get update$ sudo apt-get dist-upgrade
# import pcaptshark -T ek -x -r ./http.cap > ./http.jsoncurl -H "Content-Type: application/x-ndjson" -XPOST http://127.0.0.1:9200/_bulk --data-binary @http.json
# OPTIONAL: delete index after pcap analysiscurl -XDELETE http://127.0.0.1:9200/packets-*
# Now the Elasticsearch should contain the data and it should be possible to access it over Kibana
# ================================================# OPTIONAL: generate and add Elasticsearch mapping# ================================================
# Note: that the current tshark output is not compatible with latest Elasticsearch and the import produce errors.# So manual modification would be required of the mapping/template file.
# generate mappingtshark -G elastic-mapping --elastic-mapping-filter ip,udp,dns > elastic.mapping
# load mapping$ curl -H "Content-Type: application/json" -XPUT 'http://localhost:9200/packets-XXXX-XX-XX/_mapping' -d @elastic.mapping# check mappingcurl -XGET 'http://localhost:9200/packets-XXXX-XX-XX/_mapping'
# or load templatecurl -H "Content-Type: application/json" -XPUT 'http://localhost:9200/_template/packets' -d @elastic.mapping