Logstash Docker

Overview
Step by Step Guide
Further reading

Page last updated:

Overview

With Logstash you can create pipelines to transform and forward your logs to a log storage e.g. elasticsearch.

Step by Step Guide

This section describes how to setup your Logstash-instance in Cloud Foundry using the official Docker image from Elastic.

Prerequisites

Logstash is the last component to set up in the Elastic Stack. Therefore you need to have an Elasticsearch-Service as well as a running Kibana instance. It’s also required to set up a proper index lifecycle management. - Elasticsearch - Kibana Docker - Index Lifecycle Management

Create the centralized Logstash pipeline in Elasticsearch

To be able to use the upstream Logstash Docker images, we will configure our Logstash pipeline via Elasticsearch. The easiest way of creating the pipeline in Elasticsearch is via Kibana as outlined in https://www.elastic.co/guide/en/logstash/current/logstash-centralized-pipeline-management.html.

If you already have an existing pipeline, you can insert it into Elasticsearch (Kibana->Management->Logstash->Pipelines). If you have been using separate Grok pattern files, those must be included in your pipelines filter section instead of separate files. Also note that the http input port is 5044, as the upstream Docker image exposes the default Beats port.

If this is your fist pipeline, you may find the example pipeline below helpful, to get you started with parsing logs from apps running in Cloud Foundry. Go To Kibana->Management->Logstash->Pipelines and create a new pipeline with a distinguishable id and a pipeline according to this example:

input {
  http {
    port => "5044"
    user => "arbitrary-username"
    password => "arbitrary-password"
  }
}

filter{
  mutate {
    remove_field => ["headers"]
  }
  grok {
    match => { "message" => "%{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:syslog5424_ts}|-) +(?:%{DATA:syslog5424_host}|-) +(?:%{NOTSPACE:syslog5424_app}|-) +(?:%{NOTSPACE:syslog5424_proc}|-) +(?:%{WORD:syslog5424_msgid}|-) +(?:%{SYSLOG5424SD:syslog5424_sd}|-|)%{SPACE}%{GREEDYDATA:message}" }
    tag_on_failure => []
    overwrite => ["message"]
    break_on_match => false
  }
  grok {
    match => { "syslog5424_proc" => "^\[(%{WORD:cf_component})?/.+\]$" }
    tag_on_failure => []
    break_on_match => false
  }
  if [cf_component] == "RTR" {
      grok {
        match => { "message" => "%{HOSTNAME:host} - \[%{TIMESTAMP_ISO8601}\] \"%{WORD:http_method} %{URIPATHPARAM:uri_param}  HTTP/%{NUMBER:http_version}\" %{POSINT:http_response} %{NONNEGINT:http_request_size} %{NONNEGINT:http_response_size} \"%{DATA:referrer}\" \"%{DATA:user_agent}\" \"%{IPORHOST:ip_source}:%{POSINT:ip_source_port}\" \"%{IPORHOST:ip_destination}:%{POSINT:ip_destination_port}\" x_forwarded_for\:\"%{DATA:x_forwarded_for}\" x_forwarded_proto:\"%{DATA:x_forwarded_proto}\" vcap_request_id:\"%{UUID:vcap_request_id}\" response_time:%{NUMBER:response_time} gorouter_time:%{NUMBER:gorouter_time} app_id:\"%{UUID:app_id}\" app_index:\"%{NONNEGINT:app_index}\" x_b3_traceid:\"%{DATA:x_b3_traceid}\" x_b3_spanid:\"%{DATA:x_b3_spanid}\" x_b3_parentspanid:\"%{DATA:x_b3_parentspanid}\" b3:\"%{DATA:b3}\"" }
        tag_on_failure => ["PARSE-FAILED"]
        break_on_match => false
      }
      mutate {
        remove_field => ["message"]
      }
  }
  if [cf_component] == "APP" {
    mutate { add_tag => ["CF-APP"] }
    if [message] =~ /^{.*}/ {
          json {
            source => "message"
            add_tag => [ "json", "parsed"]
          }
          mutate {
            remove_field => ["message"]
          }
        }
  } else {
    mutate { add_tag => ["CF-PAAS"] }
  }
  grok {
    match => { "syslog5424_proc" => "^\[APP/PROC/WEB/(%{INT:cf_app_instance_index})?\]$" }
    tag_on_failure => []
    break_on_match => false
  }
  grok {
    match => { "syslog5424_host" => "^%{DATA:cf_org}\.%{DATA:cf_space}\.%{DATA:cf_app}?$" }
    tag_on_failure => []
    break_on_match => false
  }
  mutate { add_field => { "[@metadata][target_index]" => "index-%{+YYYY.MM}" } }
}

output {
  elasticsearch {
    hosts => ["https://<host>.elasticsearch.lyra-836.appcloud.swisscom.com"]
    user => "logstash-system-<id>"
    password => "<logstash-system-password>"
    index => "%{[@metadata][target_index]}"
    manage_template => true
    template_name => "logstash-template"
  }
}

For the other options, the default values are usually fine.

The above pipeline will parse logs from Cloud Foundry and add common tags like cf-org, cf-space and cf-app. Furthermore, it tags platform logs (like those from the Gorouter) as CF-PAAS and app logs as CF-APP for future filtering. If the grok-filter detects that the stripped application log message is formatted as JSON (line 36), it will treat it as such and let Elasticsearch index the fields accordingly. If you’re a Java developer, the Logstash Logback Encoder project might be of special interest to you: https://github.com/logstash/logstash-logback-encoder Make sure you use a strong password for your user in the http-section, as this endpoint will be reachable from the Internet.

Alternatively the configuration pipeline can also be created using the Elasticsearch REST API: https://www.elastic.co/guide/en/kibana/master/logstash-configuration-management-api-create.html

Push Logstash with Docker

Create the following manifest.yml in an empty folder, to configure your Logstash to use the configuration pipeline, which is stored in your Elasticsearch cluster.

---
applications:
  - name: mylogstash
    memory: 2G
    instances: 1
    routes:
      - route: mylogstash.scapp.io
    services:
      - my-elasticsearch
    docker:
      image: docker.elastic.co/logstash/logstash:7.5.2
    command: |
      export XPACK_MANAGEMENT_ELASTICSEARCH_HOSTS=$(echo $VCAP_SERVICES | grep -Po '"host":\s"\K(.*?)(?=")') &&
      export XPACK_MANAGEMENT_ELASTICSEARCH_USERNAME=$(echo $VCAP_SERVICES | grep -Po '"full_access_username":\s"\K(.*?)(?=")') &&
      export XPACK_MANAGEMENT_ELASTICSEARCH_PASSWORD=$(echo $VCAP_SERVICES | grep -Po '"full_access_password":\s"\K(.*?)(?=")') &&
      export XPACK_MONITORING_ELASTICSEARCH_HOSTS=$(echo $VCAP_SERVICES | grep -Po '"host":\s"\K(.*?)(?=")') &&
      export XPACK_MONITORING_ELASTICSEARCH_USERNAME=$(echo $VCAP_SERVICES | grep -Po '"full_access_username":\s"\K(.*?)(?=")') &&
      export XPACK_MONITORING_ELASTICSEARCH_PASSWORD=$(echo $VCAP_SERVICES | grep -Po '"full_access_password":\s"\K(.*?)(?=")') &&
      /usr/local/bin/docker-entrypoint
    env:
      XPACK_MANAGEMENT_ENABLED: true
      XPACK_MANAGEMENT_PIPELINE_ID: '["logstash_pipeline"]'
      XPACK_MONITORING_ENABLED: true
      PATH_CONFIG: ""

Use XPACK_MANAGEMENT_ELASTICSEARCH_URL instead of XPACK_MANAGEMENT_ELASTICSEARCH_HOSTS and XPACK_MONITORING_ELASTICSEARCH_URL instead of XPACK_MONITORING_ELASTICSEARCH_HOSTS for Logstash 6.x.

Now you may push your Logstash as usual using:

cf push

With cf logs mylogstash you can monitor the startup of your Logstash-instance. The following lines indicate that the pipeline has been configured successfully:

   2019-05-13T08:47:54.60+0000 [APP/PROC/WEB/0] OUT [2019-05-13T08:47:54,599][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["https://ac9537fc444c489bb63ac44064c54519.el
asticsearch.lyra-836.appcloud.swisscom.com"]}
   2019-05-13T08:47:54.78+0000 [APP/PROC/WEB/0] OUT [2019-05-13T08:47:54,784][INFO ][logstash.pipeline        ] Starting pipeline {:pipeline_id=>"logstash_pipeline_1", "pipeline.workers"=>8, "pipeline.batch.size"=>125, "pipeline.batch.delay
"=>5, "pipeline.max_inflight"=>1000, :thread=>"#<Thread:0x3e822881 run>"}
   2019-05-13T08:47:54.84+0000 [APP/PROC/WEB/0] OUT [2019-05-13T08:47:54,842][INFO ][logstash.pipeline        ] Pipeline started {"pipeline.id"=>"logstash_pipeline_1"}
   2019-05-13T08:47:54.91+0000 [APP/PROC/WEB/0] OUT [2019-05-13T08:47:54,916][INFO ][logstash.agent           ] Pipelines running {:count=>2, :pipelines=>[".monitoring-logstash", "logstash_pipeline_1"]}

Alternative: Create your own Docker image

If you wish to use configuration files instead, you can fork the official Docker image and ADD your configuration files in your own Dockerfile. You will have to host your personal Docker image in a Docker Repository like Dockerhub or Artifactory.

Create a User Provided Service to forward the Logs from your App

cf cups my-logstash-drain -l https://arbitrary-username:arbitrary-password@mylogstash.scapp.io

Bind Your App

Bind the app which should forward logs to Logstash via logstash drain. Restarting your application is not necessary.

cf bind-service my-app my-logstash-drain

You should start seeing logs in your Elasticsearch index as soon as your app receives traffic. With this step you are now done with this basic Elastic Stack guide. If you need further information about the Elastic Stack, please refer to the official documentation.