feat: Strip comments from normalized sql text in database_observability.postgres#5005
Conversation
cristiangreco
force-pushed
the
cristian/dbo11y-pg-remove-comments
branch
from
982e208 to
f3619fb
Compare
| c.entryHandler.Chan() <- database_observability.BuildLokiEntry( | ||
| logging.LevelInfo, | ||
| OP_QUERY_ASSOCIATION, | ||
| fmt.Sprintf(`queryid="%s" querytext=%q datname="%s" engine="postgres"`, queryID, queryText, databaseName), |
There was a problem hiding this comment.
Is there a reason we're removing the db engine? I understand that postgres is uniquely identified by the other values, I.E. queryid and datname. That said, having the engine listed doesn't increase cardinality, and isn't a huge increase in log line size either.
I don't have a strong opinion either way, just curious what the motivation is here.
There was a problem hiding this comment.
I understand that postgres is uniquely identified by the other values, I.E. queryid and datname. That said, having the engine listed doesn't increase cardinality, and isn't a huge increase in log line size either.
Yes that's correct, it doesn't really change the cardinality. It was added there at early development stage though and we've removed it from various places over time (this is likely the last place where it appears). Not a big deal, doing it just for consistency at this point.
|
|
cristiangreco
force-pushed
the
cristian/dbo11y-pg-remove-comments
branch
from
a180fcb to
c204afc
Compare
cristiangreco
changed the title
cristiangreco
changed the title
| } | ||
|
|
||
| func (c QueryDetails) tryTokenizeTableNames(sqlText string) ([]string, error) { | ||
| func TokenizeTableNames(normalizer *sqllexer.Normalizer, sqlText string) ([]string, error) { |
There was a problem hiding this comment.
Does this need to be exposed?
| return metadata.Tables, nil | ||
| } | ||
|
|
||
| func RemoveComments(normalizer *sqllexer.Normalizer, sqlText string) (string, error) { |
There was a problem hiding this comment.
Does this need to be exposed?
| _, metadata, err := normalizer.Normalize(sqlText, sqllexer.WithDBMS(sqllexer.DBMSPostgres)) | ||
| if err != nil { | ||
| return nil, fmt.Errorf("failed to extract table names: %w", err) | ||
| return sqlText, fmt.Errorf("failed to redact comments: %w", err) |
There was a problem hiding this comment.
nit: this is normalising, not redacting comments (which happens later)
Strip away comments from `pg_stat_statements` normalized sql queries, as postgres doesn't do that automatically. While at it: - move some stuff from `lexer.go` closer to postgres component - keep a normalizer instance around in `query_details.go` - polish some test cases formatting
cristiangreco
force-pushed
the
cristian/dbo11y-pg-remove-comments
branch
from
c204afc to
0275648
Compare
cristiangreco
changed the title
cristiangreco
changed the title
database_observability.postgres
4 participants
PR Description
Strip away comments from
pg_stat_statementsnormalized sql queries, as postgres doesn't do that automatically.While at it:
lexer.gocloser to postgres componentquery_details.goWhich issue(s) this PR fixes
n.a.
Notes to the Reviewer
PR Checklist