Skip to content

feat(loki.source.syslog): support raw format#5140

Merged
x1unix merged 32 commits intomainfrom
x1unix/feat/1711-syslog-fmt
Dec 23, 2025
Merged

feat(loki.source.syslog): support raw format#5140
x1unix merged 32 commits intomainfrom
x1unix/feat/1711-syslog-fmt

Conversation

@x1unix
Copy link
Member

@x1unix x1unix commented Dec 18, 2025

PR Description

This PR adds support of raw format to consume non-RFC5424 compliant logs such as CEF.
The format skips syslog message parsing, except non-transparent/octet count framing detection.

Which issue(s) this PR fixes

Fixes #1711

Notes to the Reviewer

PR Checklist

  • Documentation added
  • Tests updated
  • Config converters updated

BEGIN_COMMIT_OVERRIDE
feat(loki.source.syslog): Support raw format (#5140)
END_COMMIT_OVERRIDE

@x1unix x1unix requested review from a team and clayton-cornell as code owners December 18, 2025 06:30
@github-actions
Copy link
Contributor

github-actions bot commented Dec 18, 2025

💻 Deploy preview deleted (feat(loki.source.syslog): support raw format).

@github-actions
Copy link
Contributor

github-actions bot commented Dec 18, 2025

🔍 Dependency Review

github.com/leodido/go-syslog/v4 v4.2.0 -> v4.3.0 — ✅ Safe
  • Assessment: No breaking API changes identified for the symbols used by this repository; code changes are almost certainly not required.

  • Scope checked:

    • This project uses the following public APIs, all of which remain available in v4.3.0:
      • Packages: syslog, nontransparent, octetcounting, rfc3164, rfc5424
      • Types and functions:
        • type syslog.Result struct { Message syslog.Message; Error error }
        • type syslog.Base (methods: ComputeFromPriority, SeverityLevel, FacilityLevel)
        • type rfc3164.SyslogMessage
        • type rfc5424.SyslogMessage
        • nontransparent.NewParser, nontransparent.NewParserRFC3164
        • octetcounting.NewParser, octetcounting.NewParserRFC3164
  • Evidence of compatibility in this PR:

    • The code compiles against v4.3.0 while:
      • Using syslog.Base for raw-mode messages and severity/facility extraction.
      • Using the existing parsers for RFC3164/5424 streams:
        • nontransparent.NewParser(...), nontransparent.NewParserRFC3164(...)
        • octetcounting.NewParser(...), octetcounting.NewParserRFC3164(...)
      • Handling *rfc3164.SyslogMessage, *rfc5424.SyslogMessage, and *syslog.Base in the same flow.
    • No migration of imports or symbol renames required.
  • Relevant upstream notes:

    • Between v4.2.0 and v4.3.0, upstream changes are additive/bug-fix oriented for RFC3164/5424 and framing behavior. The public APIs consumed here remain stable.
    • No new minimum Go version requirement affecting this repository was introduced in v4.3.0.

No code changes suggested.

Notes

  • Only the syslog dependency replacement was bumped (v4.2.0 -> v4.3.0). No other module changes assessed.

@clayton-cornell clayton-cornell added the type/docs Docs Squad label across all Grafana Labs repos label Dec 18, 2025
@thampiotr thampiotr requested a review from Copilot December 18, 2025 09:37
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds support for a raw syslog format to consume non-RFC5424 compliant logs such as CEF (Common Event Format). The raw format skips syslog message parsing except for framing detection (octet counting or non-transparent).

Key changes:

  • Added new raw syslog format option that disables structured syslog parsing
  • Implemented raw message parser that detects framing type and extracts priority/severity when present
  • Added configuration validation to prevent incompatible options when using raw format

Reviewed changes

Copilot reviewed 25 out of 25 changed files in this pull request and generated 5 comments.

Show a summary per file
File Description
internal/loki/promtail/scrapeconfig/config.go Added SyslogFormatRaw constant and SyslogRawFormatOptions struct with delimiter configuration
internal/component/loki/source/syslog/config/config.go Added SyslogFormat validation, marshaling methods, and RawFormatOptions with delimiter support
internal/component/loki/source/syslog/types.go Updated ListenerConfig to support raw format options and added validation for format-specific fields
internal/component/loki/source/syslog/internal/syslogtarget/transport.go Modified TCP/UDP transports to handle raw format with custom delimiters
internal/component/loki/source/syslog/internal/syslogtarget/syslogtarget.go Added handleMessageRaw method and converted message handling to type switch
internal/component/loki/source/syslog/internal/syslogtarget/syslogparser/rawparser.go Implemented raw parser with octet counting and non-transparent framing detection
internal/converter/internal/promtailconvert/internal/build/syslog.go Updated converter to support raw format and options
docs/sources/reference/components/loki/loki.source.syslog.md Added documentation for raw format and configuration options
go.mod, dependency-replacements.yaml Updated go-syslog dependency to v4.3.0

@thampiotr
Copy link
Contributor

Nice work with the tests! I think some CI issues and a few copilot comments may be relevant.

@x1unix x1unix force-pushed the x1unix/feat/1711-syslog-fmt branch from 4ced942 to 1ab7ab5 Compare December 19, 2025 19:09
Copy link
Contributor

Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 25 out of 25 changed files in this pull request and generated 3 comments.

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
@x1unix x1unix merged commit 923d127 into main Dec 23, 2025
48 of 49 checks passed
@x1unix x1unix deleted the x1unix/feat/1711-syslog-fmt branch December 23, 2025 18:58
@grafana-alloybot grafana-alloybot bot mentioned this pull request Jan 5, 2026
jharvey10 pushed a commit that referenced this pull request Jan 5, 2026
This is a followup PR to a [previous PR](#5140) to extend `loki.source.syslog` component to enable parsing IOS-specific syslog fields.

This PR exposes existing cisco-specific go-syslog [parser options](https://github.com/leodido/go-syslog/tree/develop/rfc3164#cisco-device-configuration) to component arguments.
blewis12 pushed a commit that referenced this pull request Jan 6, 2026
This is a followup PR to a [previous PR](#5140) to extend `loki.source.syslog` component to enable parsing IOS-specific syslog fields.

This PR exposes existing cisco-specific go-syslog [parser options](https://github.com/leodido/go-syslog/tree/develop/rfc3164#cisco-device-configuration) to component arguments.
blewis12 pushed a commit that referenced this pull request Jan 6, 2026
This is a followup PR to a [previous PR](#5140) to extend `loki.source.syslog` component to enable parsing IOS-specific syslog fields.

This PR exposes existing cisco-specific go-syslog [parser options](https://github.com/leodido/go-syslog/tree/develop/rfc3164#cisco-device-configuration) to component arguments.
blewis12 pushed a commit that referenced this pull request Jan 7, 2026
This is a followup PR to a [previous PR](#5140) to extend `loki.source.syslog` component to enable parsing IOS-specific syslog fields.

This PR exposes existing cisco-specific go-syslog [parser options](https://github.com/leodido/go-syslog/tree/develop/rfc3164#cisco-device-configuration) to component arguments.
@github-actions github-actions bot locked as resolved and limited conversation to collaborators Jan 21, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

frozen-due-to-age type/docs Docs Squad label across all Grafana Labs repos

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Support more syslog message formats

4 participants