Distributed Tracing with Go

This document provides an introduction to distributed tracing in Go applications. Distributed tracing helps you understand the flow of requests through your microservices architecture, identify performance bottlenecks, and debug issues more effectively. We'll cover the fundamentals of tracing, introduce OpenTelemetry, and demonstrate how to integrate tracing with popular tracing backends like Jaeger and Zipkin.

What is Distributed Tracing?

In a microservices architecture, a single user request often spans multiple services. Traditional logging can be insufficient to track the complete path of a request and pinpoint the root cause of performance issues or errors. Distributed tracing solves this problem by providing a holistic view of request execution across services.

Key Concepts:

Trace: Represents the end-to-end journey of a single request as it flows through the system.
Span: A named, timed operation representing a segment of work within a trace. Spans have start and end timestamps, and associated metadata (attributes, logs, events).
Context Propagation: Automatically passing tracing information (trace ID, span ID) between services to correlate spans and reconstruct the trace.
Trace ID: A unique identifier for the entire trace.
Span ID: A unique identifier for a specific span within a trace.
Baggage: Mechanism to carry arbitrary key-value pairs through the trace. Useful for passing business context or feature flags along with the request.

Why Use Distributed Tracing?

Performance Monitoring: Identify slow or failing services along a request path.
Root Cause Analysis: Quickly pinpoint the source of errors, even across multiple services.
Dependency Visualization: Understand the dependencies between your services.
Improved Observability: Gain deeper insights into the overall health and performance of your system.
Latency Analysis: Identify high-latency operations and potential bottlenecks.

OpenTelemetry

OpenTelemetry is an open-source observability framework that provides a standard way to generate, collect, and export telemetry data (traces, metrics, and logs). It provides APIs and SDKs for various languages, including Go. It avoids vendor lock-in.

Benefits of OpenTelemetry:

Standardization: Provides a unified API for tracing, metrics, and logs.
Vendor Neutrality: Supports multiple tracing backends (Jaeger, Zipkin, Honeycomb, etc.).
Extensibility: Allows you to customize and extend the framework to meet your specific needs.

Integrating OpenTelemetry with Go

Install OpenTelemetry SDK and Exporter:

go get go.opentelemetry.io/otel
go get go.opentelemetry.io/otel/sdk
go get go.opentelemetry.io/otel/trace
go get go.opentelemetry.io/otel/exporters/jaeger  # Example: for Jaeger
go get go.opentelemetry.io/otel/exporters/zipkin  # Example: for Zipkin

go get go.opentelemetry.io/otel/propagation
go get go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp # for HTTP instrumentation

Initialize the TracerProvider:

This sets up OpenTelemetry and configures it to export traces to a tracing backend (e.g., Jaeger or Zipkin):

package main

import (
    "context"
    "log"
    "net/http"
    "os"
    "time"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/jaeger" // Or use zipkin
    "go.opentelemetry.io/otel/propagation"
    "go.opentelemetry.io/otel/sdk/resource"
    sdktrace "go.opentelemetry.io/otel/sdk/trace"
    semconv "go.opentelemetry.io/otel/semconv/v1.21.0"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp" // Add otelhttp import
)

// tracerProvider returns an OpenTelemetry TracerProvider configured to use
// the Jaeger exporter.
func tracerProvider(url string) (*sdktrace.TracerProvider, error) {
    // Create the Jaeger exporter
    exp, err := jaeger.New(jaeger.WithCollectorEndpoint(jaeger.WithEndpoint(url)))
    if err != nil {
        return nil, err
    }
    tp := sdktrace.NewTracerProvider(
        sdktrace.WithBatcher(exp),
        sdktrace.WithResource(resource.NewWithAttributes(
            semconv.SchemaURL,
            semconv.ServiceName("my-go-service"), // Replace with service name
            semconv.ServiceVersion("v1.0.0"),   // Replace with service version
        )),
    )
    return tp, nil
}


func main() {
    // Jaeger endpoint (replace with your Jaeger endpoint)
    jaegerEndpoint := os.Getenv("JAEGER_ENDPOINT")
if jaegerEndpoint == "" {
	jaegerEndpoint = "http://localhost:14268/api/traces" // Default Jaeger endpoint
	log.Println("JAEGER_ENDPOINT not set. Using default:", jaegerEndpoint)

}

    tp, err := tracerProvider(jaegerEndpoint)
    if err != nil {
        log.Fatal(err)
    }

    // Register our TracerProvider as the global so any created Tracer
    // will use it.
    otel.SetTracerProvider(tp)
otel.SetTextMapPropagator(propagation.NewCompositeTextMapPropagator(propagation.TraceContext{}, propagation.Baggage{}))
defer func() {
	if err := tp.Shutdown(context.Background()); err != nil {
		log.Printf("Error shutting down tracer provider: %v", err)
	}
}()


    // Example HTTP handler
    http.HandleFunc("/hello", func(w http.ResponseWriter, r *http.Request) {
    	ctx := r.Context()
    	span := trace.SpanFromContext(ctx) // Extract the span from the context
    	span.AddEvent("handling the hello request") // Add information to the span
            w.Write([]byte("Hello, world!"))
    })

// Instrument the HTTP server using otelhttp
handler := otelhttp.NewHandler(http.DefaultServeMux, "hello-server")
http.Handle("/", handler) // Wrap the handler with otelhttp

    log.Println("Server listening on port 8080")
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Replace "my-go-service" and "v1.0.0" with your actual service name and version.
You'll need to replace the Jaeger endpoint (jaegerEndpoint) with the correct address of your Jaeger instance. Use environment variables!
The defer tp.Shutdown() ensures proper cleanup of the tracer provider.

Create Spans:

Use the otel.Tracer to create spans within your application's code. Spans represent specific units of work.

import (
    "context"
    "fmt"

    "go.opentelemetry.io/otel"
)

var tracer = otel.Tracer("my-go-service")  // Use a consistent tracer name

func doSomeWork(ctx context.Context) {
    ctx, span := tracer.Start(ctx, "doSomeWork")
    defer span.End()

    // ... your code here ...

    span.SetAttributes(attribute.String("key", "value"))  // Add attributes to the span
    span.AddEvent("doing some fancy stuff") // add events to the span

    fmt.Println("Doing some work...")
    time.Sleep(100 * time.Millisecond)
}

The tracer.Start() function returns a new context containing the span. It's crucial to use the returned context for any downstream function calls that should be part of this span.
defer span.End() will automatically record the span's duration when the function returns.
You can add attributes and events to the span to provide more context about the operation.

Propagate Context:

When making calls to other services, you need to propagate the tracing context (trace ID and span ID) so that the spans in different services can be correlated. OpenTelemetry provides context propagation mechanisms for this.

Example using HTTP:
```
import (
    "context"
    "net/http"

    "go.opentelemetry.io/otel"
)

func callAnotherService(ctx context.Context, url string) (*http.Response, error) {
    req, err := http.NewRequestWithContext(ctx, "GET", url, nil)
    if err != nil {
        return nil, err
    }

    otel.GetTextMapPropagator().Inject(ctx, propagation.HeaderCarrier(req.Header))


    client := &http.Client{} // Consider using a reusable HTTP client
    resp, err := client.Do(req)
    return resp, err
}
```
- otel.GetTextMapPropagator().Inject() injects the tracing context into the HTTP headers of the outgoing request. This allows the receiving service to extract the context and continue the trace.
- Make sure the consuming service extracts the context using otel.GetTextMapPropagator().Extract(ctx, propagation.HeaderCarrier(r.Header))

Instrumenting HTTP Handlers

For incoming HTTP requests, you can utilize otelhttp.

import (
    "net/http"

    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)


func main() {

  //... (Tracer initialization)

  handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    // Your handler logic here
    w.Write([]byte("Hello, Traced World!"))
  })

  wrappedHandler := otelhttp.NewHandler(handler, "my-http-handler")
  http.Handle("/hello", wrappedHandler)

  http.ListenAndServe(":8080", nil) // or http.ListenAndServeTLS for HTTPS
}

Tracing Backends (Jaeger and Zipkin)

Jaeger and Zipkin are popular open-source tracing backends that provide a UI for visualizing and analyzing traces.

Jaeger: A distributed tracing system inspired by Google's Dapper. It's particularly well-suited for complex microservices architectures.
Zipkin: Another widely used distributed tracing system, originally developed by Twitter.

Running Jaeger with Docker:

docker run -d -p 16686:16686 -p 14268:14268 jaegertracing/all-in-one:latest

Running Zipkin with Docker:

docker run -d -p 9411:9411 openzipkin/zipkin

After running either of these, you can point your OpenTelemetry exporter to the appropriate endpoint and view the traces in the respective UI.

Recommendations

Use a consistent tracer name: Use the same tracer name throughout your application to make it easier to filter and analyze traces. Often, this is based on your service name.
Add meaningful attributes: Add attributes to spans to provide context about the operation. Useful attributes might include request parameters, user IDs, and error codes.
Use baggage for cross-cutting concerns: Use baggage carefully, as excessive baggage can impact performance.
Instrument your code strategically: Focus on instrumenting critical paths and potential bottlenecks.
Test your tracing configuration: Verify that traces are being generated and exported correctly.
Sampling: Implement sampling strategies to reduce the volume of trace data, especially in high-traffic environments. OpenTelemetry provides options for configuring sampling.

Conclusion

Distributed tracing is an essential tool for building and operating microservices architectures. OpenTelemetry provides a standardized and vendor-neutral way to integrate tracing into your Go applications. By using OpenTelemetry with a tracing backend like Jaeger or Zipkin, you can gain valuable insights into the performance and behavior of your system. Remember to propagate contexts correctly, add meaningful attributes to spans, and thoroughly test your tracing configuration.

What is Distributed Tracing?​

Why Use Distributed Tracing?​

OpenTelemetry​

Integrating OpenTelemetry with Go​

Tracing Backends (Jaeger and Zipkin)​

Recommendations​

Conclusion​