Configuring the OpenTelemetry Collector

Page last updated:

You can deploy an OpenTelemetry Collector for Cloud Foundry to egress traces, metrics, and logs.

When the OpenTelemetry (OTel) Collector is deployed, the Loggregator Forwarder Agent forwards platform and application metrics to the OTel Collector, which then processes traces, metrics, and logs using one or more pipelines that you configure.

Collector configuration structure

The OTel Collector has a standard YAML-based file format for configuration.

The structure of Cloud Foundry configuration file consists of three classes of pipeline components that access telemetry data:

• Exporters
• Receivers
• Processors

After each pipeline component is configured, add it in the service.pipelines section of the Collector configuration file.

Exporter configuration

For example, to configure the OpenTelemetry Collector to egress metrics using OTLP over gRPC, you can provide the following minimal configuration. This will use the defaults for gRPC and TLS configuration.

otlp:
  endpoint: 203.0.113.10:4317

As the OTel Collector is deployed on all VMs in Cloud Foundry, you must ensure that all VMs in your deployment are allowed to connect to the OTLP server endpoint.

Included exporters

mTLS

To configure the exporter to use an mTLS connection to the remote OTLP endpoint, you can specify additional TLS configuration.

otlp:
  endpoint: 203.0.113.10:4317
  tls:
    cert_pem: |
      PEM_ENCODED_CERTIFICATE
    key_pem: |
      PEM_ENCODED_PRIVATE_KEY
    ca_pem: |
      PEM_ENCODED_CERTIFICATE

Multiple exporters

The OTel Collector supports defining multiple exporters, each with its own configuration. For example, you can define two OTLP gRPC exporters that egress metrics to different destinations.

otlp:
  endpoint: 203.0.113.10:4317
otlp/another:
  endpoint: 203.0.113.11:4317

Receiver configuration

Currently, users should not list receivers and include them in the pipeline. By default, receivers are replaced with OTLP receiver that forwarder agent sends to automatically.

gRPC compression

You can configure the type of compression used for OTLP gRPC exporters or receivers. The OTLP gRPC exporter uses gzip compression by default. To reduce CPU usage of the OTel Collector (increasing the size of the payload) you can optionally configure the OTLP gRPC exporter to use snappy compression. Validate that the server you are sending metrics to support snappy compression before making this change.

otlp:
  endpoint: 203.0.113.10:4317
  compression: snappy

Processor configuration

Processors are used at various stages of a pipeline to pre-processes data before it is exported. By default, no processors are enabled. The order of the processors in a pipeline determines the order of the processing operations that the Collector applies to the signal.

Refer to the individual processor documentation for more information.

Currently, the Cloud Foundry distribution of the OpenTelemetry Collector supports the following processors out of the box: batch, memorylimiter. You can configure processors using the processors section of the Collector configuration file.

processors:   
  batch:  

Included processors

Authentication

The Cloud Foundry distribution of the OpenTelemetry Collector does not currently include any authenticator extensions. Please let us know in the Cloud Foundry Slack #logging-and-metrics channel if there are authenticator extensions you would find useful.

Reviewing the available components

The Cloud Foundry distribution of the OpenTelemetry Collector ships with a number of components. To see the components that are available for use, run the following command on a VM that has otel-collector deployed.

/var/vcap/packages/otel-collector/otel-collector components

Service section

The service section is used to configure what components are enabled in the Collector based on the configuration found in the receivers, processors, exporters, and extensions sections. If a component is configured, but not defined referenced the service section, then it’s not enabled.

The service section consists of two subsections: * Extensions * Pipelines

Extensions

You can enable desired extensions using extensions subsections:

service:
  extensions: [pprof, health_check, zpages]

Pipelines

A pipeline is a set of components used to process a signal. Before including a receiver, processor, or exporter in a pipeline, make sure to define its configuration in the appropriate section.

You can use the same receiver, processor, or exporter in more than one pipeline. When a processor is referenced in multiple pipelines, each pipeline gets a separate instance of the processor.

The following is an example of pipeline configuration. Note that the order of processors dictates the order in which data is processed:

service:
  extensions: [pprof]
  pipelines:  
    traces:
    receivers: [otlp/test]
    processors: [batch]
    exporters: [otlp]
    metrics:       
    receivers: [otlp/test]
    processors: [batch]
    exporters: [otlp]

Example config

Below is an example config structure: “` receivers: otlp/test:
protocols: grpc:

processors:
batch:

exporters: otlp: endpoint: otelcol:4317

extensions: pprof:

service: extensions: [pprof] pipelines:
traces: receivers: [otlp/test] processors: [batch] exporters: [otlp] metrics:
receivers: [otlp/test] processors: [batch] exporters: [otlp] ”`

Deploying the OpenTelemetry Collector

CF Deployment provides experimental operations files that you can use to deploy the OTel Collector:

• on Linux
• on Windows

For example, to deploy a fresh CF Deployment with OTel Collector enabled, run:

bosh deploy cf-deployment.yml \
  --ops-file=operations/experimental/add-otel-collector.yml \
  --var-file=otel_collector_metric_exporters=YOUR_METRIC_EXPORTER_CONFIG_FILE_PATH.yml \
  --var=system_domain=YOUR_SYSTEM_DOMAIN