README

Complete observability solution for Laravel applications with OpenTelemetry tracing, structured logging, and direct Loki integration.

✨ Features

🔍 Automatic Tracing - Root span per request with W3C TraceContext & B3 propagation
📝 Structured Logging - Consistent JSON schema with automatic trace_id/span_id injection
🚀 Direct Loki Push - Send logs directly to Loki without Promtail
🔧 Auto-Instrumentation - HTTP client, database queries, queue jobs
🔒 Sensitive Data Masking - Automatic masking of passwords, tokens, secrets
💪 Fail-Open Design - Won't crash your app if observability fails
⚡ Octane/Swoole Safe - Request-scoped context, no global state leaks
🎯 Fluent API - Developer-friendly API for audit trails

📋 Requirements

PHP 8.2+
Laravel 10.x, 11.x, or 12.x

📦 Installation

composer require hsr/observability-laravel

Publish configuration:

php artisan vendor:publish --tag=observability-config

🚀 Quick Start

1. Environment Configuration

# Service identification
OTEL_SERVICE_NAME=my-service
APP_ENV=production

# OTLP endpoint (Tempo, Jaeger, or OTel Collector)
OTEL_EXPORTER_OTLP_ENDPOINT=http://tempo:4318

# Optional: Direct Loki push
LOKI_ENDPOINT=http://loki:3100/loki/api/v1/push

# Feature flags
OBSERVABILITY_TRACING_ENABLED=true
OBSERVABILITY_LOGGING_ENABLED=true

2. Configure Logging Channel

Add to config/logging.php:

<?php

return [
    'default' => env('LOG_CHANNEL', 'observability'),
    
    'channels' => [
        // Option 1: Using the custom channel factory (RECOMMENDED for file/stdout)
        'observability' => [
            'driver' => 'custom',
            'via' => \HsrObservability\Logging\ObservabilityLogChannelFactory::class,
            'stream' => 'php://stdout', // or storage_path('logs/observability.log')
            'level' => env('LOG_LEVEL', 'debug'),
        ],
        
        // Option 2: Direct push to Loki (NO PROMTAIL NEEDED!)
        'loki' => [
            'driver' => 'custom',
            'via' => \HsrObservability\Logging\LokiLogChannelFactory::class,
            'endpoint' => env('LOKI_ENDPOINT', 'http://loki:3100/loki/api/v1/push'),
            'labels' => [
                'app' => env('APP_NAME', 'laravel'),
                'env' => env('APP_ENV', 'production'),
            ],
            'batch_size' => 1,  // 1 = immediate send
            'timeout' => 5,
            'level' => env('LOG_LEVEL', 'debug'),
        ],
        
        // Option 3: Using monolog driver with tap
        'observability-tap' => [
            'driver' => 'monolog',
            'handler' => \Monolog\Handler\StreamHandler::class,
            'with' => [
                'stream' => 'php://stdout',
            ],
            'tap' => [\HsrObservability\Logging\ObservabilityLogTap::class],
        ],
        
        // Stack for multiple outputs (e.g., stdout + Loki)
        'stack' => [
            'driver' => 'stack',
            'channels' => ['observability', 'loki'],
            'ignore_exceptions' => false,
        ],
        
        // ... other channels
    ],
];

Important: After adding the package, run:

composer dump-autoload
php artisan config:clear

3. Verify Installation

php artisan observability:test

Usage

Basic Logging (Auto-Enriched)

use Illuminate\Support\Facades\Log;

// All logs automatically get trace_id, span_id, service info, etc.
Log::info('User logged in');
Log::error('Payment failed', ['order_id' => 'ORD-123']);

Using Fluent Context API

use HsrObservability\Facades\Observability;

class UserController extends Controller
{
    public function store(Request $request)
    {
        $user = User::create($request->validated());
        
        // Set observability context for audit trail
        Observability::ctx()
            ->module('USER')
            ->action('CREATE')
            ->category('DATA_CHANGE')
            ->risk('MEDIUM')
            ->change(
                fieldsChanged: ['name', 'email'],
                old: [],
                new: ['name' => $user->name, 'email' => $user->email]
            )
            ->logInfo('User created successfully');
        
        return response()->json($user, 201);
    }
    
    public function update(Request $request, User $user)
    {
        $oldData = $user->only(['name', 'email']);
        $user->update($request->validated());
        
        Observability::forUpdate('USER', 
            fieldsChanged: array_keys($request->validated()),
            old: $oldData,
            new: $user->only(['name', 'email'])
        )->logInfo('User updated');
        
        return response()->json($user);
    }
    
    public function destroy(User $user)
    {
        $userData = $user->toArray();
        $user->delete();
        
        Observability::forDelete('USER', $userData)
            ->risk('HIGH')
            ->logWarning('User deleted');
        
        return response()->noContent();
    }
}

Convenience Methods

// Create operation
Observability::forCreate('ORDER', ['total' => 1000, 'items' => 5])
    ->logInfo('Order placed');

// Update operation
Observability::forUpdate('ORDER', ['status'], 
    old: ['status' => 'pending'], 
    new: ['status' => 'paid']
)->logInfo('Order status updated');

// View operation (low risk, minimal context)
Observability::forView('ORDER')
    ->logDebug('Order viewed');

// Auth operations
Observability::forAuth('LOGIN')
    ->logInfo('User authenticated');

Observability::forAuth('LOGOUT')
    ->logInfo('User logged out');

Error Handling

try {
    // risky operation
} catch (\Exception $e) {
    Observability::ctx()
        ->module('PAYMENT')
        ->action('CHARGE')
        ->category('TRANSACTION')
        ->errorFromException($e)
        ->risk('HIGH')
        ->logError('Payment failed');
    
    throw $e;
}

Custom Context

Observability::ctx()
    ->module('REPORT')
    ->action('GENERATE')
    ->extra('report_type', 'monthly')
    ->extra('date_range', ['2024-01-01', '2024-01-31'])
    ->extras([
        'format' => 'pdf',
        'recipients' => 5,
    ])
    ->logInfo('Report generated');

Log Output Schema

All logs follow this exact JSON schema:

{
  "timestamp": 1761277589,
  "environment": "production",
  "trace_id": "5b8892f1-1e6d-4494-b725-1e7042355808",
  "span_id": "e3c5a6c1c9d04777",
  "service": {
    "name": "CORE"
  },
  "user": {
    "id": "USER-ID",
    "role_name": "Administrator",
    "name": "Aldi"
  },
  "context": {
    "module": "USER",
    "action": "CREATE",
    "category": "DATA_CHANGE",
    "http_status_code": 200,
    "latency_ms": 123,
    "error_code": null,
    "error_message": null,
    "risk_level": "MEDIUM",
    "level": "INFO"
  },
  "request": {
    "method": "POST",
    "path": "/api/v1/users",
    "query": {
      "include": "roles"
    }
  },
  "response": {
    "body": null
  },
  "change": {
    "fields_changed": ["name", "email"],
    "old_value": {
      "name": null,
      "email": null
    },
    "new_value": {
      "name": "Example Name",
      "email": "example@hsr.com"
    }
  },
  "client": {
    "ip_address": "120.120.10.1",
    "browser_name": "Chrome",
    "browser_version": "126.0.6478.61"
  },
  "location": {
    "city": "Bandung",
    "region": "Jawa Barat",
    "country": "ID",
    "timezone": "Asia/Jakarta"
  }
}

Configuration

Full configuration options in config/observability.php:

return [
    // Service identification
    'service' => [
        'name' => env('OTEL_SERVICE_NAME', 'laravel-app'),
        'version' => env('APP_VERSION', '1.0.0'),
        'environment' => env('APP_ENV', 'production'),
    ],

    // Tracing configuration
    'tracing' => [
        'enabled' => env('OBSERVABILITY_TRACING_ENABLED', true),
        'exporter' => [
            'endpoint' => env('OTEL_EXPORTER_OTLP_ENDPOINT', 'http://otel-collector:4318'),
        ],
        'sampling' => [
            'ratio' => env('OTEL_TRACES_SAMPLER_ARG', 1.0),
        ],
    ],

    // Logging configuration
    'logging' => [
        'enabled' => env('OBSERVABILITY_LOGGING_ENABLED', true),
        'json_schema_enabled' => env('OBSERVABILITY_JSON_SCHEMA', true),
        'include_request_query' => true,
        'include_request_body' => false,  // Enable with caution
        'include_response_body' => false, // Enable with caution
        'max_body_size_bytes' => 2048,
    ],

    // Sensitive data masking
    'masking' => [
        'enabled' => true,
        'replacement' => '[REDACTED]',
        'fields' => [
            'password', 'token', 'api_key', 'secret',
            'authorization', 'cookie', 'otp', 'pin', 'cvv',
        ],
    ],

    // Routes to ignore
    'ignored_routes' => [
        'healthz', 'health', 'metrics',
    ],
];

Promtail/Loki Configuration

Label Guidelines (Anti-High-Cardinality)

Safe labels (low cardinality):

service → service.name
env → environment
level → context.level
category → context.category

DO NOT use as labels (high cardinality):

trace_id → query as JSON field
span_id → query as JSON field
user_id → query as JSON field
request_path → query as JSON field

Example Promtail Config

scrape_configs:
  - job_name: laravel
    static_configs:
      - targets: [localhost]
        labels:
          job: laravel
          __path__: /var/log/laravel/*.log
    pipeline_stages:
      - json:
          expressions:
            level: context.level
            service: service.name
            env: environment
            category: context.category
      - labels:
          level:
          service:
          env:
          category:

Loki Queries

# Find logs by trace_id
{service="my-service"} | json | trace_id="5b8892f1-1e6d-4494-b725-1e7042355808"

# Find errors for a user
{service="my-service", level="ERROR"} | json | user_id="USER-123"

# Find data changes
{service="my-service", category="DATA_CHANGE"} | json | context_module="USER"

OpenTelemetry Collector Setup

docker-compose.yml

services:
  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
    command: ["--config=/etc/otel-collector-config.yaml"]
    volumes:
      - ./otel-collector-config.yaml:/etc/otel-collector-config.yaml
    ports:
      - "4318:4318"  # OTLP HTTP

  tempo:
    image: grafana/tempo:latest
    command: ["-config.file=/etc/tempo.yaml"]
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml
    ports:
      - "3200:3200"

otel-collector-config.yaml

receivers:
  otlp:
    protocols:
      http:
        endpoint: 0.0.0.0:4318

processors:
  batch:
    timeout: 5s
    send_batch_size: 1000

exporters:
  otlp/tempo:
    endpoint: tempo:4317
    tls:
      insecure: true

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp/tempo]

Testing

# Run tests
composer test

# Run with coverage
composer test-coverage

# Static analysis
composer analyse

Troubleshooting

Self-Test Command

php artisan observability:test

Common Issues

1. No traces appearing in Tempo

Check OTLP endpoint is reachable: curl http://otel-collector:4318/v1/traces
Verify OBSERVABILITY_TRACING_ENABLED=true
Check sampling ratio is > 0

2. Missing trace_id in logs

Ensure TracingMiddleware is registered
Check if route is in ignored_routes
Verify logging processor is added to channel

3. High memory usage

Reduce max_body_size_bytes
Disable include_request_body and include_response_body
Increase sampling ratio to reduce trace volume

License

MIT

hsr-engineer / observability-laravel

Maintainers

Details