Intermediateobservabilitytelemetryopen-telemetryopentelemetrytracingmetricslogsgocloud-nativecnCF

OpenTelemetry Collector: Vendor-agnostic telemetry data pipeline

Deploy a vendor-agnostic telemetry pipeline that receives, processes, and exports traces, metrics, and logs from multiple sources to diverse backends without vendor lock-in.

Step 1
What is OpenTelemetry Collector?
The OpenTelemetry Collector is a vendor-agnostic implementation that provides a comprehensive solution for receiving, processing, and exporting telemetry data. It eliminates the need to run multiple agents/collectors by supporting open-source telemetry data formats (Jaeger, Prometheus, Zipkin, etc.) and exporting to multiple backends simultaneously.

Key objectives and capabilities:
- Usable: Reasonable default configuration, supports popular protocols, runs out of the box
- Performant: Highly stable under varying loads and configurations
- Observable: An exemplar of an observable service with built-in monitoring
- Extensible: Customizable without touching core code via the Collector Builder
- Unified: Single codebase supporting traces, metrics, and logs
The Collector can be deployed in two modes:
- Agent mode (daemon): Runs on each host/receiver, scrapes localhost targets via Prometheus
- Collector/Connector mode: Aggregates data from agents, enriches with tags, routes to backends
With 7,000+ GitHub stars from 100+ contributors, the OpenTelemetry Collector is a CNCF graduated project maintained by industry leaders including Honeycomb, Snowflake, Splunk, Dynatrace, DataDog, Elastic, and Microsoft.
```
OpenTelemetry Collector Architecture:

Sources                    Processors                Backends
 ──────                    ────────                ────────

OTLP (gRPC/HTTP)    →     Batch → Compression     →     Jaeger
Prometheus scrape    →     Sampling → Filtering    →     Prometheus
Jaeger Thrift        →     Memory Limiter          →     Splunk
Zipkin               →     Trace ID → Tagging      →     Datadog
Carbon               →     Transform attributes    →     New Relic
Fluentd              →     Filter by service name  →     Honeycomb
Skywalking           →     Add host metadata       →     Sentry
AWS X-Ray            →     Group by resource       →     Elastic APM

Receivers → Processors → Connectors → Exporters → Extension
   ↓
Telemetry Sources (Traces, Metrics, Logs)
```
Step 2
Technology stack and components
The OpenTelemetry Collector is built on a modern, high-performance architecture designed for distributed systems observability.

Primary Technology Stack:
- Go (Golang): Core implementation language, leveraging concurrency and performance
- gRPC/Protobuf: High-performance inter-service communication protocol
- OpenTelemetry Protocol (OTLP): Standard protocol for telemetry data transfer, currently v1.10.0
- YAML/JSON: Configuration formats
Core Components:

Receivers (ingestion endpoints):
- otlpreceiver: Native OpenTelemetry Protocol (gRPC + HTTP)
- prometheusreceiver: Prometheus scraping and federation
- jaegerreceiver: Jaeger Thrift and gRPC protocols
- zipkinreceiver: Zipkin API and Thrift
- Additional: Fluentd, Skywalking, AWS X-Ray, Carbon, SignalFx
Processors (data transformation):
- batchprocessor: Buffer and batch data for efficiency
- memorylimiterprocessor: Protect against overload, adaptive slowdown
- attributesprocessor: Add, remove, modify trace/metric/log attributes
- probabilisticsampler: Sample traces by probability
- tailsamplingprocessor: Post-decision sampling based on trace characteristics
Exporters (data destinations):
- otlpexporter: Send via OTLP gRPC
- otlphttpexporter: Send via OTLP HTTP
- debugexporter: Print to stdout for debugging
- Many more: Jaeger, Prometheus, Splunk, Datadog, Honeycomb, New Relic, etc.
Extensions (enhanced functionality):
- zpagesextension: TensorFlow Z-Pages for monitoring internal state
- healthcheckextension: Prometheus-compatible health check endpoint
- pprofextension: Go profiling endpoints
- oauth2extension: OAuth 2.0 client credentials
```
Collector Pipeline Architecture:

┌───────────────────────────────────────────┐
│              EXTENSIONS                   │
│  (Auth, Healthcheck, Pprof, CORS)        │
└───────────────────────────────────────────┘
                  ↓
        ┌─────────┴──────────┐
        ↓                   ↓
┌──────────────┐      ┌─────────────┐
│  RECEIVERS   │      │  CONNECTORS │
│  - OTLP      │      │  - Routing  │
│  - Jaeger    │      │  - M2L      │
│  -Prometheus │      │  - S2L      │
└──────┬───────┘      └─────────────┘
       ↓
┌──────────────┐        ┌─────────────┐
│  PROCESSORS  │        │  EXPORTERS  │
│  - Batch     ├────────┼  - OTLP     │
│  - Sampling  │        │  - Splunk   │
│  - Attrs     │        │  - Datadog  │
│  - MemLimit  │        │  - Jaeger   │
└──────────────┘        └─────────────┘

Receivers → Processors → Connectors → Exporters
```
Step 3
Prerequisites
Before deploying OpenTelemetry Collector, ensure you have the following:

For binary installation:
- Linux (amd64, arm64, ppc64le, s390x), macOS (arm64), or Windows (amd64)
- 50 MB+ disk space for binaries
- Network connectivity to receivers and exporters
For Docker deployments:
- Docker Engine 20.10+
- Docker Compose v2+ (optional, for multi-container setups)
- Available ports for receivers (OTLP gRPC: 4317, OTLP HTTP: 4318, Jaeger: 4317/14268, Zipkin: 2000)
For Kubernetes deployments:
- Kubernetes cluster 1.24+
- Helm 3.9+ installed locally
- kubectl configured with cluster access
- Ingress controller if using HTTP receivers
For custom builds with Collector Builder:
- Go 1.25.x
- Make and GCC
- Approximately 2GB RAM during build
Network requirements:
- Outbound connectivity to telemetry backends
- Inbound ports for receivers as described above
```
# Verify Docker installation
$ docker --version
$ docker compose version

# Verify Kubernetes access (if applicable)
$ kubectl version --client
$ helm version

# Check system resources
$ free -h
$ df -h
$ uname -m

# Verify network connectivity to backends
$ curl -I https://your-prometheus-server:9090
$ nc -zv your-collector-endpoint 4317
```

Step 4

Quick start with Docker

The fastest way to get started with OpenTelemetry Collector is with Docker. The official images are available on Docker Hub and ghcr.io.

Image variants:

otel/opentelemetry-collector:latest - Core distribution
otel/opentelemetry-collector-contrib:latest - Contrib distribution (200+ components)
Versioned tags: otel/opentelemetry-collector-contrib:0.115.0

Docker Compose is ideal for development and testing, while Docker Swarm or Kubernetes is recommended for production.

# 1. Quick start with contrib (most components)
docker run -d \
  --name otel-collector \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 13133:13133 \
  -p 8888:8888 \
  otel/opentelemetry-collector-contrib:0.115.0

# 2. With config file
mkdir -p otelconfig
cat > otelconfig/config.yaml << 'EOF'
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
exporters:
  debug:
    verbosity: detailed
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: []
      exporters: [debug]
    metrics:
      receivers: [otlp]
      processors: []
      exporters: [debug]
    logs:
      receivers: [otlp]
      processors: []
      exporters: [debug]
EOF

docker run -d \
  --name otel-collector \
  -p 4317:4317 \
  -p 4318:4318 \
  -p 13133:13133 \
  -v $(pwd)/otelconfig/config.yaml:/etc/otelcol-contrib/config.yaml:ro \
  otel/opentelemetry-collector-contrib:0.115.0

# 3. Docker Compose (recommended for development)
cat > docker-compose.yml << 'EOF'
version: '3'
services:
  otelcol:
    image: otel/opentelemetry-collector-contrib:0.115.0
    restart: unless-stopped
    command: ["--config=/etc/otelcol-contrib/config.yaml"]
    ports:
      - 4317:4317   # OTLP gRPC
      - 4318:4318   # OTLP HTTP
      - 8889:8889   # Jaeger compact
      - 14268:14268 # Jaeger Thrift HTTP
      - 2000:2000   # Zipkin
      - 13133:13133 # Health check
      - 8888:8888   # ZPages
    volumes:
      - ./config.yaml:/etc/otelcol-contrib/config.yaml:ro
    networks:
      - observability
networks:
  observability:
    driver: bridge
EOF

docker compose up -d

# 4. Verify
# View logs
docker logs -f otelcol

# Health check
curl http://localhost:13133/health

# Access ZPages (debugging)
open http://localhost:8888

# Send test data using opentelemetry-dotnet or any SDK, or simply:
# The collector accepts OTLP on 4317 (gRPC) and 4318 (HTTP)

# 5. Stop and remove
docker compose down

Step 5

Configuration fundamentals

OpenTelemetry Collector uses declarative YAML or JSON configuration. The configuration defines receivers, processors, exporters, and service topology.

Core configuration sections:

receivers: How telemetry enters the collector
processors: Transformations applied to telemetry
exporters: Destinations for telemetry data
extensions: Global capabilities (auth, health check)
service: Pipeline definitions and component wiring

Each pipeline (traces/metrics/logs) has its own receivers, processors, and exporters.

# Basic configuration with multiple receivers and exporters
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  prometheus:
    config:
      scrape_configs:
        - job_name: 'otel-collector'
          scrape_interval: 10s
          static_configs:
            - targets: ['localhost:8888']

processors:
  batch:
    timeout: 10s
    send_batch_size: 1024
  memory_limiter:
    check_interval: 1s
    limit_mib: 1536
  attributes:
    actions:
      - key: deployment.environment
        value: production
        action: upsert

exporters:
  debug:
    verbosity: detailed
  otlphttp:
    endpoint: https://your-backend.example.com:4318
    headers:
      Authorization: "Bearer ${OTEL_BACKEND_TOKEN}"

service:
  extensions: [zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, attributes, batch]
      exporters: [debug, otlphttp]
    metrics:
      receivers: [otlp, prometheus]
      processors: [batch]
      exporters: [otlphttp]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [debug, otlphttp]

extensions:
  zpages:
    endpoint: 0.0.0.0:55679
  health_check:
    endpoint: 0.0.0.0:13133

⚠ Heads up: Never hardcode sensitive information. Use environment variable substitution with ${VARIABLE_NAME} syntax.

Step 6

Building custom Collector distributions

The OpenTelemetry Collector Builder creates lightweight, purpose-built Collector binaries containing only your selected components. This reduces binary size, improves security, and provides better control.

Benefits:

Smaller binary size vs 200MB+ contrib binary
Reduced attack surface
Easier maintenance
Better control over component versions

Build process:

Install the builder tool
Define your components in a YAML manifest
Run the builder to generate your custom binary
(Optional) Set up CI/CD for automated builds

# 1. Install Collector Builder
CDPATH="" cd $(mktemp -d)
VERSION="v0.115.0"

# Linux/macOS
curl -fsSL "https://github.com/open-telemetry/opentelemetry-collector-contrib/releases/download/$VERSION/opentelemetry-collector-builder_\$(go env GOOS)-\$(go env GOARCH)-amd64.tar.gz" | tar xz
export PATH=\$PATH:\$(pwd)

# 2. Create builder config (builder-config.yaml)
cat > builder-config.yaml << 'EOF'
distributions:
  - name: my-otelcol
    version: 1.0.0
    output_path: bin/
    go:
      version: 1.25.0
    
    receivers:
      - github.com/open-telemetry/opentelemetry-collector-contrib/receiver/otlpreceiver
      - github.com/open-telemetry/opentelemetry-collector-contrib/receiver/jaegerreceiver
      - github.com/open-telemetry/opentelemetry-collector-contrib/receiver/prometheusreceiver
      - github.com/open-telemetry/opentelemetry-collector-contrib/receiver/zipkinreceiver
    
    processors:
      - github.com/open-telemetry/opentelemetry-collector-contrib/processor/batchprocessor
      - github.com/open-telemetry/opentelemetry-collector-contrib/processor/memorylimiterprocessor
      - github.com/open-telemetry/opentelemetry-collector-contrib/processor/attributesprocessor
      - github.com/open-telemetry/opentelemetry-collector-contrib/processor/tailsamplingprocessor
    
    exporters:
      - github.com/open-telemetry/opentelemetry-collector-contrib/exporter/otlphttpexporter
      - github.com/open-telemetry/opentelemetry-collector-contrib/exporter/splunkhecexporter
      - github.com/open-telemetry/opentelemetry-collector-contrib/exporter/datadogexporter
    
    extensions:
      - github.com/open-telemetry/opentelemetry-collector-contrib/extension/zpages
      - github.com/open-telemetry/opentelemetry-collector-contrib/extension/health_check
EOF

# 3. Build the custom binary
otelcol-builder --config builder-config.yaml

# 4. Check the generated binary
ls -lh bin/my-otelcol
./bin/my-otelcol --version

# 5. Run with your configuration
./bin/my-otelcol --config otel-config.yaml

Step 7

Kubernetes deployment

For production Kubernetes deployments, the OpenTelemetry Operator provides comprehensive management of Collector instances via Custom Resource Definitions (CRDs).

Operator features:

Custom CRDs (OpenTelemetryCollector, OpenTelemetryInstrumentation)
Automatic config generation
Rolling updates and rollbacks
Service mesh integration
Multi-namespace and cluster-scoped support

# 1. Install OpenTelemetry Operator via Helm
helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm repo update

helm install opentelemetry-operator \
  open-telemetry/opentelemetry-operator \
  --namespace otel-operator-system \
  --create-namespace

# 2. Create Collector manifest
cat > otel-collector.yaml << 'EOF'
apiVersion: opentelemetry.io/v1beta1
kind: OpenTelemetryCollector
metadata:
  name: otel-collector
  namespace: observability
spec:
  mode: deployment
  replicas: 3
  resources:
    requests:
      cpu: 250m
      memory: 512Mi
    limits:
      cpu: 1000m
      memory: 1Gi
  config:
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
    processors:
      batch:
        timeout: 10s
        send_batch_size: 1024
    exporters:
      debug: {}
      otlphttp:
        endpoint: https://backend.example.com:4318
        headers:
          Authorization: "Bearer \${OTEL_BACKEND_TOKEN}"
    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [debug, otlphttp]
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [debug, otlphttp]
        logs:
          receivers: [otlp]
          processors: [batch]
          exporters: [debug, otlphttp]
EOF

# 3. Apply the collector deployment
kubectl apply -f otel-collector.yaml

# 4. Verify deployment
kubectl get pods -n observability -l app.kubernetes.io/name=otel-collector
kubectl get svc -n observability

# 5. View logs
kubectl logs -n observability -l app.kubernetes.io/name=otel-collector -f

# 6. Access ZPages (debugging)
kubectl port-forward -n observability svc/otel-collector-zpages 55679:8888

# 7. Update collector (rolling update)
kubectl apply -f otel-collector-updated.yaml

# 8. Rollback if needed
kubectl rollout undo deployment/otel-collector -n observability

Step 8

Integration examples

OpenTelemetry Collector integrates with popular observability platforms. Common patterns include:

Multi-backend routing: Send telemetry to multiple backends simultaneously Service mesh: Integration with Istio, Linkerd, and others Cloud providers: AWS, GCP, Azure native integrations Security: OAuth 2.0, mTLS for authentication and encryption

# Multi-backend routing: Jaeger, Splunk, and Datadog
receivers:
  otlp:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317

processors:
  batch:
    timeout: 10s
    send_batch_size: 1024

exporters:
  jaeger:
    endpoint: jaeger:14250
    tls:
      insecure: true
  
  splunk_hec:
    endpoint: https://splunk.example.com:8088
    headers:
      Authorization: "Splunk \${SPLUNK_HEC_TOKEN}"
    sending_queue:
      enabled: true
      numconsumers: 4
      queue_size: 1000
    retry_on_failure:
      enabled: true
      initial_interval: 5s
      max_interval: 30s
  
  datadog:
    api:
      key: \${DD_API_KEY}
    api_url: https://api.datadoghq.com/
  
  prometheus:
    endpoint: 0.0.0.0:8889
    namespace: otel

service:
  extensions: [zpages, health_check]
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [jaeger, splunk_hec, datadog]
    metrics:
      receivers: [otlp]
      processors: [batch]
      exporters: [prometheus, datadog]
    logs:
      receivers: [otlp]
      processors: [batch]
      exporters: [splunk_hec, datadog]

extensions:
  zpages:
    endpoint: 0.0.0.0:55679
  health_check:
    endpoint: 0.0.0.0:13133

⚠ Heads up: Always use environment variables for API keys and tokens. Committing credentials is a security risk.

Step 9
Resources and community
Official Documentation: https://opentelemetry.io/docs/collector/

Getting Started: https://opentelemetry.io/docs/collector/getting-started/

GitHub Repository: https://github.com/open-telemetry/opentelemetry-collector (Apache 2.0)

Contributing Guide: https://github.com/open-telemetry/opentelemetry-collector/blob/main/CONTRIBUTING.md

Component Stability: https://github.com/open-telemetry/opentelemetry-collector/blob/main/docs/component-stability.md

Community:
- Slack: #otel-collector on CNCF Slack (cloud-native.slack.com)
- GitHub Discussions: https://github.com/open-telemetry/opentelemetry-collector/discussions
- Weekly SIG meetings: Schedule in community repo
- Issues: https://github.com/open-telemetry/opentelemetry-collector/issues
Related Projects:
- Collector Contrib: https://github.com/open-telemetry/opentelemetry-collector-contrib (200+ community components)
- Collector Builder: https://github.com/open-telemetry/opentelemetry-collector/tree/main/cmd/builder
- OTel Operator: https://github.com/open-telemetry/opentelemetry-operator (Kubernetes)
- OpenTelemetry Demo: https://github.com/open-telemetry/opentelemetry-demo
Training:
- CNCF OpenTelemetry projects: https://www.cncf.io/projects/
- OTel University: Educational content
License: Apache 2.0
```
Key Resources:
- Docs: opentelemetry.io/docs/collector/
- GitHub: github.com/open-telemetry/opentelemetry-collector
- Contrib: github.com/open-telemetry/opentelemetry-collector-contrib
- Slack: cloud-native.slack.com/channels/otel-collector

Key Commands:
# Start Collector
otelcol-contrib --config config.yaml

# Validate config
otelcol-contrib --config config.yaml --dry-run

# Build custom binary
otelcol-builder --config builder-config.yaml

# Helm install operator
helm install otel-operator open-telemetry/opentelemetry-operator
```

What is OpenTelemetry Collector?

Technology stack and components

Prerequisites

Quick start with Docker

Configuration fundamentals

Building custom Collector distributions

Kubernetes deployment

Integration examples

Resources and community

Feature requests

Discussion