OpenTelemetry & APM

Saviour's Sentinel agent includes a built-in OTLP receiver that accepts telemetry from any OpenTelemetry SDK. When enabled, Sentinel listens on every node for OTLP/HTTP traffic and forwards it directly to the Saviour backend — your traces, service maps, and APM metrics appear automatically in the Saviour UI.

You do not need a separate OTel Collector deployment. Sentinel is the collector.

---

How it works

┌─────────────────────────────────────────────┐
│  Kubernetes Node                            │
│                                             │
│  ┌────────────────┐   OTLP/HTTP (4318)      │
│  │  Your App Pod  │ ────────────────────►   │
│  └────────────────┘                    │    │
│                                        ▼    │
│  ┌──────────────────────────────────────┐   │
│  │  Sentinel (DaemonSet)               │   │
│  │  otlpReceiver on 0.0.0.0:4318       │   │
│  └──────────────────┬───────────────────┘   │
└─────────────────────┼───────────────────────┘
                      │  HTTPS → api.saviourops.com
                      ▼
              ┌────────────────┐
              │ Saviour Backend │
              └───────┬────────┘
                      │
                      ▼
            APM · Traces · Service Map

Since Sentinel is a DaemonSet, one Sentinel pod runs on every schedulable node. Your application pod uses its node's IP (status.hostIP) as the OTLP endpoint. No service discovery or sidecar is needed.

---

Step 1 — Enable the OTLP receiver

By default the receiver is off. Enable it and widen the listen address so pods on the same node can reach it:

helm upgrade saviour oci://ghcr.io/saviourops-labs/charts/saviour \
  --set sentinel.otlpReceiver.enabled=true \
  --set sentinel.otlpReceiver.listenAddress="0.0.0.0:4318" \
  --reuse-values

Why 0.0.0.0?

The default 127.0.0.1:4318 only accepts connections from the node's own localhost. Changing to 0.0.0.0:4318 allows pods on the same node (using status.hostIP) to reach Sentinel. Combined with NetworkPolicy, this remains private to your cluster.

Verify the receiver is listening on every node:

# Pick any Sentinel pod
SENTINEL_POD=$(kubectl get pods -n saviour -l app.kubernetes.io/name=sentinel -o name | head -1)
kubectl logs -n saviour $SENTINEL_POD | grep -i "otlp receiver"
# Expected: otlp receiver started on 0.0.0.0:4318

---

Step 2 — Inject the endpoint into your pods

Because Sentinel runs as a DaemonSet, each pod must send traces to its own node's Sentinel pod. Use the Kubernetes downward API to get the node IP:

# Add to your Pod spec (or Deployment template)
env:
  - name: NODE_IP
    valueFrom:
      fieldRef:
        fieldPath: status.hostIP
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: "http://$(NODE_IP):4318"
  - name: OTEL_SERVICE_NAME
    value: "your-service-name"         # change this
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: "deployment.environment=production"

The OTEL_EXPORTER_OTLP_ENDPOINT environment variable is read automatically by all official OpenTelemetry SDKs — you don't need to set the endpoint in code.

---

Step 3 — Instrument your application

Pick your language below. All examples use the standard OpenTelemetry SDK with automatic instrumentation where available.

Node.js

Install packages:

npm install @opentelemetry/sdk-node \
  @opentelemetry/auto-instrumentations-node \
  @opentelemetry/exporter-trace-otlp-http

Create tracing.js (load before your app):

// tracing.js
const { NodeSDK } = require('@opentelemetry/sdk-node');
const { getNodeAutoInstrumentations } = require('@opentelemetry/auto-instrumentations-node');
const { OTLPTraceExporter } = require('@opentelemetry/exporter-trace-otlp-http');

const sdk = new NodeSDK({
  traceExporter: new OTLPTraceExporter(),   // reads OTEL_EXPORTER_OTLP_ENDPOINT from env
  instrumentations: [getNodeAutoInstrumentations()],
});

sdk.start();
process.on('SIGTERM', () => sdk.shutdown());

Start your app with the tracer loaded:

node --require ./tracing.js app.js

Dockerfile example:

FROM node:20-alpine
WORKDIR /app
COPY package*.json ./
RUN npm ci --production
COPY . .
CMD ["node", "--require", "./tracing.js", "app.js"]

Python

Install packages:

pip install opentelemetry-distro opentelemetry-exporter-otlp-proto-http
opentelemetry-bootstrap --action=install

Option A — Zero-code auto-instrumentation (recommended):

# Run your app through the opentelemetry-instrument wrapper
opentelemetry-instrument python app.py

The wrapper auto-instruments Flask, Django, FastAPI, SQLAlchemy, requests, and more. It reads OTEL_EXPORTER_OTLP_ENDPOINT from the environment automatically.

Option B — Manual SDK setup:

# tracing.py
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.otlp.proto.http.trace_exporter import OTLPSpanExporter

provider = TracerProvider()
provider.add_span_processor(BatchSpanProcessor(OTLPSpanExporter()))
trace.set_tracer_provider(provider)

# In your app (import tracing before any instrumented code)
import tracing
from opentelemetry import trace

tracer = trace.get_tracer(__name__)

with tracer.start_as_current_span("my-operation") as span:
    span.set_attribute("user.id", user_id)
    result = do_work()

Go

Install packages:

go get go.opentelemetry.io/otel \
       go.opentelemetry.io/otel/sdk/trace \
       go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp

Setup in main.go:

package main

import (
    "context"
    "log"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
    "go.opentelemetry.io/otel/sdk/trace"
)

func initTracer(ctx context.Context) func(context.Context) error {
    exporter, err := otlptracehttp.New(ctx) // reads OTEL_EXPORTER_OTLP_ENDPOINT
    if err != nil {
        log.Fatalf("failed to create OTLP exporter: %v", err)
    }

    tp := trace.NewTracerProvider(
        trace.WithBatcher(exporter),
        trace.WithSampler(trace.AlwaysSample()),
    )
    otel.SetTracerProvider(tp)
    return tp.Shutdown
}

func main() {
    ctx := context.Background()
    shutdown := initTracer(ctx)
    defer shutdown(ctx)

    tracer := otel.Tracer("my-service")
    ctx, span := tracer.Start(ctx, "main-operation")
    defer span.End()

    // your application code...
}

HTTP handler auto-instrumentation (net/http):

import "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"

http.Handle("/api/orders", otelhttp.NewHandler(orderHandler, "orders"))

Java

Option A — Java agent (zero-code, recommended):

Download the OpenTelemetry Java agent JAR:

curl -Lo opentelemetry-javaagent.jar \
  https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/latest/download/opentelemetry-javaagent.jar

Run with the agent:

java -javaagent:opentelemetry-javaagent.jar \
     -Dotel.service.name=my-service \
     -jar app.jar

The agent auto-instruments Spring Boot, Micronaut, Quarkus, JDBC, HTTP clients, and 100+ frameworks. The OTEL_EXPORTER_OTLP_ENDPOINT environment variable is read automatically.

Option B — Maven/Gradle SDK:

<!-- pom.xml -->
<dependency>
  <groupId>io.opentelemetry</groupId>
  <artifactId>opentelemetry-api</artifactId>
  <version>1.38.0</version>
</dependency>
<dependency>
  <groupId>io.opentelemetry</groupId>
  <artifactId>opentelemetry-sdk</artifactId>
  <version>1.38.0</version>
</dependency>
<dependency>
  <groupId>io.opentelemetry</groupId>
  <artifactId>opentelemetry-exporter-otlp</artifactId>
  <version>1.38.0</version>
</dependency>

// TracingConfig.java
OtlpHttpSpanExporter exporter = OtlpHttpSpanExporter.builder().build(); // reads env

SdkTracerProvider provider = SdkTracerProvider.builder()
    .addSpanProcessor(BatchSpanProcessor.builder(exporter).build())
    .build();

OpenTelemetrySdk openTelemetry = OpenTelemetrySdk.builder()
    .setTracerProvider(provider)
    .buildAndRegisterGlobal();

---

Step 4 — Verify traces appear in Saviour

1. Deploy your instrumented app
2. Send a few requests to it
3. Open app.saviourops.com
4. Go to APM → your service should appear within 30 seconds
5. Click the service to see traces, latency percentiles, and error rates

If traces don't appear, check Sentinel logs for ingest errors:

kubectl logs -n saviour -l app.kubernetes.io/name=sentinel \
  --all-containers | grep -i "otlp\|trace\|error" | tail -20

---

Zero-code: OpenTelemetry Operator

If you prefer not to modify your application code at all, use the OpenTelemetry Operator to inject auto-instrumentation:

Install the operator:

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts
helm install otel-operator open-telemetry/opentelemetry-operator \
  --namespace opentelemetry-operator-system \
  --create-namespace

Create an Instrumentation resource pointing at Sentinel:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: saviour-instrumentation
  namespace: default
spec:
  exporter:
    endpoint: http://$(status.hostIP):4318   # resolved per-pod at runtime

  propagators:
    - tracecontext
    - baggage

  sampler:
    type: parentbased_traceidratio
    argument: "1"         # 100% — lower this in high-traffic environments

  nodejs:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-nodejs:latest
  python:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-python:latest
  java:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-java:latest
  go:
    image: ghcr.io/open-telemetry/opentelemetry-operator/autoinstrumentation-go:latest

Annotate your pod/deployment to enable injection:

metadata:
  annotations:
    instrumentation.opentelemetry.io/inject-nodejs: "default/saviour-instrumentation"
    # or inject-python, inject-java, inject-go

The operator injects the appropriate SDK and sets OTEL_EXPORTER_OTLP_ENDPOINT automatically — no code changes required.

---

Configuration reference

These Helm values control the OTLP receiver:

Value	Default	Description
sentinel.otlpReceiver.enabled	false	Enable the built-in OTLP/HTTP receiver
sentinel.otlpReceiver.listenAddress	127.0.0.1:4318	Address to listen on. Use 0.0.0.0:4318 for pod-to-pod traffic
sentinel.otlpTracing.enabled	false	Forward Sentinel's own internal traces to an external OTLP endpoint
sentinel.otlpTracing.endpoint	""	Upstream OTLP endpoint for Sentinel's own traces
sentinel.otlpTracing.samplingRate	1.0	Sampling rate for Sentinel's own traces (0.0–1.0)

Lock down ingress with NetworkPolicy — enable the NetworkPolicy and add an ingress rule for port 4318:

helm upgrade saviour oci://ghcr.io/saviourops-labs/charts/saviour \
  --set sentinel.networkPolicy.enabled=true \
  --set sentinel.otlpReceiver.enabled=true \
  --set sentinel.otlpReceiver.listenAddress="0.0.0.0:4318" \
  --reuse-values

The built-in sentinel-networkpolicy.yaml already opens port 4318 for intra-cluster ingress when networkPolicy.enabled=true.

---

Supported OTLP signals

Signal	Supported	Protocol	Port
Traces	✅	OTLP/HTTP (protobuf)	4318
Traces	✅	OTLP/HTTP (JSON)	4318
Metrics	🔜 Coming soon	—	—
Logs	🔜 Coming soon	—	—

gRPC (port 4317)

The receiver currently supports OTLP/HTTP only (port 4318). OTLP/gRPC (port 4317) is on the roadmap. To use gRPC SDKs today, configure them to use HTTP transport:

• Node.js: use @opentelemetry/exporter-trace-otlp-http (not the grpc package)
• Java: set -Dotel.exporter.otlp.protocol=http/protobuf
• Go: use otlptracehttp (not otlptracegrpc)
• Python: OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf

---

Troubleshooting

Traces not appearing in the UI

Symptom	Check	Fix
No service in APM tab	Receiver not enabled	sentinel.otlpReceiver.enabled=true, listenAddress=0.0.0.0:4318
Connection refused from pod	Wrong endpoint	Pod env must use status.hostIP, not localhost or a fixed IP
Traces sent but not in UI	Sentinel logs for forward errors	Check kubectl logs -n saviour -l app.kubernetes.io/name=sentinel | grep -i otlp
High drop rate	Receiver overloaded	Reduce sampling rate or increase maxConcurrent
No service.name in UI	Missing resource attribute	Set OTEL_SERVICE_NAME env var in your pod

Connection test from a pod

Run a quick connectivity test from your application pod:

kubectl exec -it <your-app-pod> -- \
  curl -v http://$NODE_IP:4318/v1/traces \
  -H "Content-Type: application/json" \
  -d '{"resourceSpans":[]}'
# Expected: HTTP/1.1 200 OK

If this returns connection refused, Sentinel's receiver is not running or the listen address is still 127.0.0.1.

---

What you see in Saviour

Once traces are flowing:

• APM → Services — latency (p50/p95/p99), request rate, error rate per service
• APM → Traces — individual trace waterfall with spans, duration, status codes
• Map — service dependency graph built from trace relationships
• Incidents — latency/error anomalies automatically raised as incidents

No manual dashboard configuration is required — Saviour builds all views from trace data automatically.