Skip to content

out_pgsql: loading libpg library as dynamic - #12633

Open
cosmo0920 wants to merge 2 commits into
masterfrom
cosmo0920-loading-libpg-library-as-dynamic
Open

cosmo0920 wants to merge 2 commits into
masterfrom
cosmo0920-loading-libpg-library-as-dynamic

Conversation

@cosmo0920

@cosmo0920 cosmo0920 commented Oct 22, 2026 •

Copy link
Copy Markdown
Contributor

Added a module-wide skip check in the PostgreSQL integration tests. It checks the selected binary before loading libpq or starting test servers.

Verification:

  • PGSQL enabled: 17 passed, normally and under strict Valgrind; no memory errors or leaks.
  • FLB_OUT_PGSQL=Off: 17 skipped, including with a nonexistent library path.
  • Restored the enabled build afterward.

Commands run:

tests/integration/.venv/bin/python -m pytest tests/integration/scenarios/out_pgsql -q
VALGRIND=1 VALGRIND_STRICT=1 tests/integration/.venv/bin/python -m pytest tests/integration/scenarios/out_pgsql -q
TEST_LIBPQ_PATH=/nonexistent/libpq.so tests/integration/.venv/bin/python -m pytest tests/integration/scenarios/out_pgsql -q -rs
VALGRIND=1 VALGRIND_STRICT=1 TEST_LIBPQ_PATH=/nonexistent/libpq.so tests/integration/.venv/bin/python -m pytest tests/integration/scenarios/out_pgsql -q -rs

Closes #12573.


Enter [N/A] in the box, if an item is not applicable to your change.

Testing
Before we can approve your change; please submit the following in a comment:

  • Example configuration file for the change
  • Debug log output from testing the change
  • Attached Valgrind output that shows no leaks or memory corruption was found

If this is a change to packaging of containers or native binaries then please confirm it works for all targets.

  • Run local packaging test showing all targets (including any new ones) build.
  • Set ok-package-test label to test for all targets (requires maintainer to do).

Documentation

  • Documentation required for this feature

Backporting

  • Backport to latest stable release.

Fluent Bit is licensed under Apache 2.0, by submitting this pull request I understand that this code will be released under the terms of that license.

Summary by CodeRabbit

  • New Features
    • PostgreSQL output can load its client library at runtime, using the system default or a configured library path. This removes the requirement for a link-time dependency on the PostgreSQL client library.
  • Bug Fixes
    • Improved cleanup when PostgreSQL library loading, table setup, or synchronous queries fail. Broken PostgreSQL client libraries no longer prevent unrelated pipelines from starting.

Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
Signed-off-by: Hiroshi Hatake <hiroshi@chronosphere.io>
@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review 🔄 Running since 2026-10-22T06:07:42.246539Z fc7979c PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Oct 22, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: defaults
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 3ee69d43-3927-419c-a73b-2e162f1a00ec
📥 Commits

Reviewing files that changed from the base of the PR and between cef5706 and fc7979c.

📒 Files selected for processing: 7
  • plugins/out_pgsql/CMakeLists.txt
  • plugins/out_pgsql/pgsql.c
  • plugins/out_pgsql/pgsql.h
  • plugins/out_pgsql/pgsql_connections.c
  • plugins/out_pgsql/pgsql_libpq.c
  • plugins/out_pgsql/pgsql_libpq.h
  • tests/integration/scenarios/out_pgsql/tests/test_libpq_loading.py

Included review availability: This review used your included allowance. Your plan provides up to 8 included reviews per hour; 7 remain after this review.

📝 Walkthrough

The PostgreSQL output now loads libpq at runtime and uses resolved function pointers for database operations. Integration tests cover library selection, query flushing, load failures, and operation without a libpq dynamic dependency.

Changes: PostgreSQL runtime library loading

Layer / File(s) Summary
Define and load the libpq interface
plugins/out_pgsql/CMakeLists.txt, plugins/out_pgsql/pgsql_libpq.h, plugins/out_pgsql/pgsql_libpq.c
The plugin now links with the platform dynamic-loading library and includes a libpq loader. The loader selects a platform filename by default, resolves required symbols, and clears its state when closed.
Use loaded functions in PostgreSQL operations
plugins/out_pgsql/pgsql.h, plugins/out_pgsql/pgsql.c, plugins/out_pgsql/pgsql_connections.c
Initialization reads the optional libpq_path property and loads libpq. Connection management, escaping, table creation, recovery, and query execution use loaded function pointers.
Verify runtime loading and database operations
tests/integration/scenarios/out_pgsql/tests/test_libpq_loading.py
Integration tests cover default and explicit library paths, sync and async flushes, multiple outputs, startup failures, and operation without a libpq dynamic dependency.

Priority: ➖ Normal

Estimated code review effort: 3 (Moderate) | ~25 minutes

Change: Feature · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant FluentBit
  participant PostgreSQLOutput
  participant LibpqLoader
  participant libpq
  participant PostgreSQLPeer
  FluentBit->>PostgreSQLOutput: Initialize output with libpq_path
  PostgreSQLOutput->>LibpqLoader: pgsql_libpq_open(path)
  LibpqLoader->>libpq: Open library and resolve required symbols
  LibpqLoader-->>PostgreSQLOutput: Store function pointers
  PostgreSQLOutput->>libpq: Create connection and execute queries
  libpq->>PostgreSQLPeer: PostgreSQL protocol operations
Loading
Merge Risk: ⚪ Minimal · up to fc797

No actionable merge-blocking issue remains; the change is ready for normal checks.

Pre-merge checks: Passed 4 · Failed 1

Failed Failed checks: 1 warning

Check name Status Explanation Resolution
Docstring Coverage Warning Docstring coverage is 5.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 20 functions across 6 files. (1 skipped: 1… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 4 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title identifies the main change: dynamic loading of the PostgreSQL client library. It contains a typo and awkward wording because it says "libpg" instead of "libpq," but it remains clear and rela…
Linked Issues check Passed Issue #12573 requires Fluent Bit to install when PostgreSQL 18 is present. The PR removes the -lpq link from plugins/out_pgsql/CMakeLists.txt and links only ${CMAKE_DL_LIBS}. pgsql_libpq.c loa…
Out of Scope Changes check Passed The changes stay within issue #12573. The source changes implement runtime libpq loading for the PostgreSQL output. The integration tests cover successful loading, failure handling, and removal of t…
Full details: Title check

Explanation

The title identifies the main change: dynamic loading of the PostgreSQL client library. It contains a typo and awkward wording because it says "libpg" instead of "libpq," but it remains clear and related to the changes.

Full details: Linked Issues check

Explanation

Issue #12573 requires Fluent Bit to install when PostgreSQL 18 is present. The PR removes the -lpq link from plugins/out_pgsql/CMakeLists.txt and links only ${CMAKE_DL_LIBS}. pgsql_libpq.c loads libpq.so.5 at runtime and resolves the required symbols. Integration tests cover runtime loading and assert that the Fluent Bit binary has no libpq NEEDED dependency. These changes address the reported RPM dependency conflict.

Full details: Out of Scope Changes check

Explanation

The changes stay within issue #12573. The source changes implement runtime libpq loading for the PostgreSQL output. The integration tests cover successful loading, failure handling, and removal of the link dependency. No unrelated product or test changes are present in the reviewed scope.

Full details: Docstring Coverage

Explanation

Docstring coverage is 5.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 20 functions across 6 files. (1 skipped: 1 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

✨ Finishing Touches: 💡 2
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
🛠️ Fix failing CI checks: 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@sxd

sxd commented Oct 23, 2026

Copy link
Copy Markdown
Member

Hi @cosmo0920

I have many questions about this, specially about why someone will want this, but I have to take a look more into this, specially about the version 5 on the lib name, since there has been discussions about bumping the version and that will make this fail.

I'm in conference until next week and I'll take some time to take into this.

Regards,

@cosmo0920

Copy link
Copy Markdown
Contributor Author

Hi sxd,

Thanks for responding this.
To be honest, I just wanted to change to load dynamically to resolve AlmaLinux 10 postgresql packaging issue.
This is because AlmaLinux has two packages to be able to install but just linking with -lpq option on compile, Fluent Bit package just links the specific dependent library at compilation and not to be able to install the different version of libpq.
But in #12573, users wanted to use libpq and postgresql which is the different version of compiled one.

This branch was previously deployed

1 inactive deployment
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Problematic dependency of fluent-bit-5.1.3 rpm on libpq.so.private16-5

2 participants