Skip to main content
Skip to main content
Edit this page

Monitoring Systemd Logs with ClickStack

TL;DR

This guide shows you how to monitor systemd journal logs with ClickStack by running the OpenTelemetry Collector with the journald receiver. You'll learn how to:

  • Deploy the OpenTelemetry Collector to read systemd journal entries
  • Send systemd logs to ClickStack via OTLP
  • Use a pre-built dashboard to visualize systemd log insights (service status, errors, authentication events)

A demo dataset with sample logs is available if you want to test the integration before configuring your production systems.

Time Required: 10-15 minutes

Integration with existing systems​

Monitor your existing Linux system's journald logs by running the OpenTelemetry Collector with the journald receiver to collect system logs and send them to ClickStack via OTLP.

If you want to test this integration first without modifying your existing setup, skip to the demo dataset section.

Prerequisites​
  • ClickStack instance running
  • Linux system with systemd (Ubuntu 16.04+, CentOS 7+, Debian 8+)
  • Docker or Docker Compose installed on the monitored system

Get ClickStack API key​

The OpenTelemetry Collector sends data to ClickStack's OTLP endpoint, which requires authentication.

  1. Open HyperDX at your ClickStack URL (e.g., http://localhost:8080)
  2. Create an account or log in if needed
  3. Navigate to Team Settings → API Keys
  4. Copy your Ingestion API Key
  1. Set it as an environment variable:
export CLICKSTACK_API_KEY=your-api-key-here

Verify systemd journal is running​

Ensure your system is using systemd and has journal logs:

# Check systemd version
systemctl --version

# View recent journal entries
journalctl -n 20

# Check journal disk usage
journalctl --disk-usage

If journal storage is in memory only, enable persistent storage:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Create OpenTelemetry Collector configuration​

Create a configuration file for the OpenTelemetry Collector:

cat > otel-config.yaml << 'EOF'
receivers:
  journald:
    directory: /var/log/journal
    priority: info
    units:
      - sshd
      - nginx
      - docker
      - containerd
      - systemd

processors:
  batch:
    timeout: 10s
    send_batch_size: 1024
  
  resource:
    attributes:
      - key: service.name
        value: systemd-logs
        action: insert
      - key: host.name
        from_attribute: _HOSTNAME
        action: upsert
  
  attributes:
    actions:
      - key: unit
        from_attribute: _SYSTEMD_UNIT
        action: upsert
      - key: priority
        from_attribute: PRIORITY
        action: upsert

exporters:
  otlphttp:
    endpoint: ${CLICKSTACK_ENDPOINT}
    headers:
      authorization: ${CLICKSTACK_API_KEY}

service:
  pipelines:
    logs:
      receivers: [journald]
      processors: [resource, attributes, batch]
      exporters: [otlphttp]
EOF

Deploy with Docker Compose​

Note

The journald receiver requires the journalctl binary to read journal files. The official otel/opentelemetry-collector-contrib image does not include journalctl by default.

For containerized deployments, you can either install the collector directly on the host or build a custom image with systemd utilities. See the troubleshooting section for details.

This example shows deploying the OTel Collector alongside ClickStack:

services:
  clickstack:
    image: clickhouse/clickstack-all-in-one:latest
    ports:
      - "8080:8080"
      - "4317:4317"
      - "4318:4318"
    networks:
      - monitoring
  
  otel-collector:
    image: otel/opentelemetry-collector-contrib:0.115.1
    depends_on:
      - clickstack
    environment:
      - CLICKSTACK_API_KEY=${CLICKSTACK_API_KEY}
      - CLICKSTACK_ENDPOINT=http://clickstack:4318
    volumes:
      - ./otel-config.yaml:/etc/otelcol/config.yaml:ro
      - /var/log/journal:/var/log/journal:ro
      - /run/log/journal:/run/log/journal:ro
      - /etc/machine-id:/etc/machine-id:ro
    command: ["--config=/etc/otelcol/config.yaml"]
    networks:
      - monitoring

networks:
  monitoring:
    driver: bridge

Start the services:

docker compose up -d

Verify logs in HyperDX​

Once configured, log into HyperDX and verify logs are flowing:

  1. Navigate to the Search view
  2. Set source to Logs
  3. Filter by service.name:systemd-logs
  4. You should see structured log entries with fields like unit, priority, MESSAGE, _HOSTNAME

Demo dataset​

For users who want to test the systemd logs integration before configuring their production systems, we provide a sample dataset of pre-generated systemd logs with realistic patterns.

Download the sample dataset​

Download the sample log file:

curl -O https://datasets-documentation.s3.eu-west-3.amazonaws.com/clickstack-integrations/systemd/systemd-demo.log

Create demo collector configuration​

Create a configuration file for the demo:

cat > systemd-demo.yaml << 'EOF'
receivers:
  filelog:
    include:
      - /tmp/systemd-demo/systemd-demo.log
    start_at: beginning
    operators:
      - type: regex_parser
        regex: '^(?P<timestamp>\S+) (?P<hostname>\S+) (?P<unit>\S+?)(?:\[(?P<pid>\d+)\])?: (?P<message>.*)$'
        parse_from: body
        parse_to: attributes
      - type: time_parser
        parse_from: attributes.timestamp
        layout: '%Y-%m-%dT%H:%M:%S%z'
      - type: add
        field: attributes.source
        value: "systemd-demo"

service:
  pipelines:
    logs/systemd-demo:
      receivers: [filelog]
      processors:
        - memory_limiter
        - transform
        - batch
      exporters:
        - clickhouse
EOF

Run ClickStack with demo data​

Start ClickStack with the demo logs:

docker run -d --name clickstack-demo \
  -p 8080:8080 -p 4317:4317 -p 4318:4318 \
  -e CUSTOM_OTELCOL_CONFIG_FILE=/etc/otelcol-contrib/custom.config.yaml \
  -v "$(pwd)/systemd-demo.yaml:/etc/otelcol-contrib/custom.config.yaml:ro" \
  -v "$(pwd)/systemd-demo.log:/tmp/systemd-demo/systemd-demo.log:ro" \
  clickhouse/clickstack-all-in-one:latest
Note

The demo uses the filelog receiver with text logs instead of journald to avoid requiring journalctl in the container.

Verify logs in HyperDX​

Once ClickStack is running:

  1. Open HyperDX and log in to your account
  2. Navigate to the Search view and set the source to Logs
  3. Set the time range to 2025-11-14 00:00:00 - 2025-11-17 00:00:00
Timezone Display

HyperDX displays timestamps in your browser's local timezone. The demo data spans 2025-11-15 00:00:00 - 2025-11-16 00:00:00 (UTC). The wide time range ensures you'll see the demo logs regardless of your location.

Dashboards and visualization​

To help you get started monitoring systemd logs with ClickStack, we provide essential visualizations for systemd journal data.

Download the dashboard configuration​

Import the pre-built dashboard​

  1. Open HyperDX and navigate to the Dashboards section
  2. Click Import Dashboard in the upper right corner under the ellipses
  1. Upload the systemd-logs-dashboard.json file and click Finish Import

View the dashboard​

The dashboard includes visualizations for:

  • Log volume over time
  • Top systemd units by log count
  • SSH authentication events
  • Service failures
  • Error rates
Note

For the demo dataset, set the time range to 2025-11-15 00:00:00 - 2025-11-16 00:00:00 (UTC) (adjust based on your local timezone).

Troubleshooting​

No logs appearing in HyperDX​

Check if logs are reaching ClickHouse:

docker exec clickstack clickhouse-client --query "
SELECT COUNT(*) as log_count
FROM otel_logs
WHERE ServiceName = 'systemd-logs'
"

If no results, check the collector logs:

docker logs otel-collector | grep -i "error\|journald" | tail -20

journalctl not found error​

If you see exec: "journalctl": executable file not found in $PATH:

The otel/opentelemetry-collector-contrib image does not include journalctl. You can either:

  1. Install the collector on the host:
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.115.0/otelcol-contrib_0.115.0_linux_amd64.tar.gz
tar -xzf otelcol-contrib_0.115.0_linux_amd64.tar.gz
sudo mv otelcol-contrib /usr/local/bin/
otelcol-contrib --config=otel-config.yaml
  1. Use the text export approach (like the demo) with the filelog receiver reading journald exports

Going to production​

This guide uses a separate OpenTelemetry Collector to read systemd logs and send them to ClickStack's OTLP endpoint, which is the recommended production pattern.

For production environments with multiple hosts, consider:

  • Deploying the collector as a DaemonSet in Kubernetes
  • Running the collector as a systemd service on each host
  • Using the OpenTelemetry Operator for automated deployment

See Ingesting with OpenTelemetry for production deployment patterns.