Elasticsearch Buildpacks

Page last updated:

Deprecated

This page is deprecated. Please refer to Elasticsearch.

Overview

The Elasticsearch cluster consists of 3 nodes in 3 different availability zones. By default, each index in Elasticsearch has 5 primary shards and 1 replica allocated to it. If you don’t want to use the default settings, make sure to configure the number of shards & replicas accordingly in the index templates.

Quick Tutorial

1. Create the service
2. Configure and push Logstash buildpack
3. Configure and push Kibana buildpack
4. Create a user provided service
5. Bind your app to the user provided service
6. See and search through your logs

Step by Step Guide

This section describes how to store your application logs in Elasticsearch using Logstash & Kibana with the least effort.

Push a Logstash App

To push a Logstash app in the same org / space, create the following files in an otherwise empty folder:

.
├── Logstash
└── manifest.yml

We recommend enabling authentication for your Logstash. Provide a username and password in your Logstash file as in the following example. Authentication will be disabled if no username is provided.

Logstash:

logstash-credentials:
  username: myuser
  password: mypassword

manifest.yml

applications:
- name: my-logstash
  memory: 2G
services:
  - my-elasticsearch

To push the app run the following command:

cf push -b https://github.com/Swisscom/logstash-buildpack.git

Push a Kibana App

To push a Kibana app in the same org / space, create the following files in an otherwise empty folder:

.
├── Kibana
└── manifest.yml

You need to add X-Pack as a plugin to the Kibana file in order for it to communicate with the Elasticsearch service:

Kibana:

plugins:
- x-pack

manifest.yml:

applications:
- name: my-kibana
  memory: 3G
  disk_quota: 2G
services:
  - my-elasticsearch

To push the app run the following command:

cf push -b https://github.com/Swisscom/kibana-buildpack.git

Note: Currently, X-Pack needs a lot of resources to be installed as it perfoms some optimisation steps. You can scale down your Kibana memory using:

cf scale my-kibana -m 2G

Verify Apps

To verify that both apps are bound to the newly created Elasticsearch service run cf services. The output should look similar to this:

name               service         plan      bound apps               last operation
my-elasticsearch   elasticsearch   xxsmall   my-kibana, my-logstash   create succeeded

Create a User Provided Service

cf cups my-logstash-drain -l https://USERNAME:PASSWORD@URL-OF-LOGSTASH-INSTANCE USERNAME & PASSWORD: the username and the password you configured in the Logstash file. URL-OF-LOGSTASH-INSTANCE: This is the URL of the pushed logstash app in step 2: cf apps

Bind Your App

Bind your app of which you want to have the logs in Elasticsearch to the user provided service created in the previous step. Then restage the app.

cf bs my-app my-logstash-drain
cf restage my-app

Customization

Kibana and Logstash buildpacks are designed to be customizable, so that they fit your needs. Please have a look at the appropriate buildpack documentation on Github:

• Kibana buildpack
• Logstash buildpack