Distributed tracing

Page last updated:

Cloud Foundry provides support for distributed tracing, enabling transactions to be monitored and analyzed across multiple Cloud Foundry components. For each transaction initiated in the GoRouter, a new trace context is generated and then propagated to all participating systems.

Each participating system adds a unique identifier to logs, allowing you to trace logs and debug errors specific to that transaction. Distributed tracing is particularly beneficial when used with log aggregation tools like Splunk.

Unique trace identification

For every request in Cloud Foundry, GoRouter generates a unique X-Vcap-Request-Id header. X-Vcap-Request-Id is a v4 UUID according to RFC 4122.

If Zipkin is enabled, see Enabling Zipkin tracing in Cloud Foundry.

GoRouter generates B3 headers for each request unless they were provided by the client:

• X-B3-Traceid - 128-bit in length and indicates the overall ID of the trace. Every span in a trace shares this ID. The X-B3-TraceId header is encoded as 32 lower-hex characters.
• X-B3-Spanid - 64-bit in length and indicates the position of the current operation in the trace tree. The X-B3-SpanId header is encoded as 16 lower-hex characters.
• B3 - a combination of X-B3-Traceid and X-B3-Spanid headers in the format: {TraceId}-{SpanId}

If W3C is enabled through the router.tracing.enable_w3c deployment manifest property, GoRouter generates W3C headers for each request unless they were provided by the client. W3C headers are generated according to the W3C headers format:

• traceparent - consists of 4 fields in the format: {version-format}-{trace-id}-{parent-id}-{trace-flags}
• tracestate - is generated from router.tracing.w3c_tenant_id deployment property and parent-id used in the transparent header: {w3c_tenant_id}={parent-id}

If B3 headers are provided and they are in a proper v4 UUID format, GoRouter uses them for X-Vcap-Request-Id and W3C headers. If they are not provided, GoRouter generates v4 UUID and use it for B3 headers, X-Vcap-Request-Id and W3C headers.

Using trace identification to analyze a specific Cloud Foundry request

In the following example, a request is made to Cloud Foundry to scale an application. This request is handled by several Cloud Foundry components:

1. The request is initially received by GoRouter. GoRouter generates unique trace headers for this request. It then forwards this request with the trace headers to the Cloud Controller.
2. Cloud Controller updates the LRP (Long Running Process) for the application, reflecting the new number of instances. It subsequently sends the LRP request with the appended trace headers to BBS.
3. BBS generates a new request with the trace headers to Auctioneer to schedule a new application instance.
4. Auctioneer selects a Diego Cell that has enough resources to run the application instance. It then sends a request with the trace headers to Rep on that Diego Cell, instructing it to run this new application instance.
5. Finally, Rep generates a request to Garden. This request with the same trace information, instructs Garden to create a container where application processes run.

Each component in this transaction:

• Retains the original request’s trace-id and forwards it in all downstream requests.
• Generates a unique span-id for each request, which remains exclusive to that particular component.
• Both the trace-id and span-id are recorded in logs related to that request handling.

This transaction is characterized by a single trace-id that identifies it across all hops, while each individual hop is marked by its own unique span-id.

The CF_TRACE environment variable is used to get the generated B3 trace headers.

CF_TRACE=true cf scale myapp -i 2

The output contains a sequence of requests and responses from Cloud Foundry. Initially, the CLI retrieves the application GUID, followed by an API request to scale the application. The response to this request includes B3 headers, which are generated by the GoRouter.

Scaling app myapp in org myorg / space myspace as admin...

REQUEST:
POST /v3/apps/8e0f1501-ba1f-4efe-af90-b61f2e5a1384/processes/web/actions/scale
...
{
  "instances": 2
}


RESPONSE:
HTTP/1.1 202 Accepted
...
X-B3-Spanid: 67bde5d7dcd81f84
X-B3-Traceid: 0c2cc9583f004a4167bde5d7dcd81f84
...

You can tell the GoRouter to use user-provided B3 headers. For example:

cf curl -H "X-B3-Traceid: 8c436ec437e545227fe76a3eba99ca21" -H "X-B3-Spanid: 7fe76a3eba99ca21" -d '{"instances" : 3}' /v3/apps/8e0f1501-ba1f-4efe-af90-b61f2e5a1384/processes/web/actions/scale

You must set both X-B3-Traceid and X-B3-Spanid.

The X-B3-Traceid can be used to filter logs in aggregation tools to see how this request was handled by the GoRouter, Cloud Controller, BBS, Rep, Auctioneer and how a container for a new application instance was created in Garden.