code_public/mod_reqin_log
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Refactor: thread-safe per-process state and add tests

Browse Source 
Major changes:
- Move child state from global variable to server config (reqin_log_server_conf_t)
- Add reqin_log_create_server_conf() for proper per-server initialization
- Fix thread safety for worker/event MPMs
- Add cmocka unit tests (test_module_real.c)
- Add Python integration tests (test_integration.py)
- Update CI workflow and Dockerfiles for test execution
- Fix: Remove child_exit hook (not in architecture.yml)

Tests:
- Unit tests: JSON escaping, ISO8601 formatting, header truncation
- Integration tests: basic_logging, header_limits, socket_unavailable, socket_loss

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
This commit is contained in:
Jacquin Antoine
2026-02-26 23:28:45 +01:00
parent 7cfd14fb65
commit 070c2a7bd2
7 changed files with 1425 additions and 340 deletions
Download Patch File Download Diff File Expand all files Collapse all files

202
architecture.yml
View File

@ -40,26 +40,38 @@ module:
- name: register_hooks
responsibilities:
- Register post_read_request hook for logging at request reception.
- Initialize per-process Unix socket connection if enabled.
- Register child_init hook for per-process state initialization.
- Initialize per-process server configuration structure.
- name: child_init
responsibilities:
- Initialize module state for each Apache child process.
- Reset per-process socket state (fd, timers, error flags).
- Attempt initial non-blocking connection to Unix socket if configured.
- name: child_exit
responsibilities:
- Cleanly close Unix socket file descriptor if open.
- name: post_read_request
responsibilities:
- Retrieve per-process server configuration (thread-safe).
- Ensure Unix socket is connected (with periodic reconnect).
- Build JSON log document for the request.
- Write JSON line to Unix socket using non-blocking I/O.
- Handle errors by dropping the current log line and rate-limiting
error reports into Apache error_log.
thread_safety:
model: per-process-state
description: >
Each Apache child process maintains its own socket state stored in the
server configuration structure (reqin_log_server_conf_t). This avoids
race conditions in worker and event MPMs where multiple threads share
a process.
implementation:
- State stored via ap_get_module_config(s->module_config)
- No global variables for socket state
- Each process has independent: socket_fd, connect timers, error timers
data_model:
json_line:
description: >
One JSON object per HTTP request, serialized on a single line and
terminated by "\n".
terminated by "\n". Uses flat structure with header fields at root level.
structure: flat
fields:
- name: time
type: string
@ -69,8 +81,9 @@ module:
type: integer
unit: nanoseconds
description: >
Monotonic or wall-clock based timestamp in nanoseconds since an
implementation-defined epoch (stable enough for ordering and latency analysis).
Wall-clock timestamp in nanoseconds since Unix epoch.
Note: apr_time_now() returns microseconds, multiplied by 1000 for nanoseconds.
example: 1708948770000000000
- name: src_ip
type: string
example: "192.0.2.10"
@ -95,15 +108,30 @@ module:
- name: http_version
type: string
example: "HTTP/1.1"
- name: headers
type: object
- name: header_<HeaderName>
type: string
description: >
Flattened headers from the configured header list. Keys are derived
from configured header names prefixed with 'header_'.
Flattened header fields at root level. For each configured header <H>,
a field 'header_<H>' is added directly to the JSON root object.
Headers are only included if present in the request.
key_pattern: "header_<configured_header_name>"
optional: true
example:
header_X-Request-Id: "abcd-1234"
header_User-Agent: "curl/7.70.0"
example_full:
time: "2026-02-26T11:59:30Z"
timestamp: 1708948770000000000
src_ip: "192.0.2.10"
src_port: 45678
dst_ip: "198.51.100.5"
dst_port: 443
method: "GET"
path: "/api/users"
host: "example.com"
http_version: "HTTP/1.1"
header_X-Request-Id: "abcd-1234"
header_User-Agent: "curl/7.70.0"
configuration:
scope: global
@ -258,44 +286,75 @@ constraints:
testing:
strategy:
unit_tests:
framework: cmocka
location: tests/unit/test_module_real.c
focus:
- JSON serialization with header truncation and header count limits.
- Directive parsing and configuration merging (global scope).
- Error-handling branches for non-blocking write and reconnect logic.
- Dynamic buffer operations (dynbuf_t) with resize handling.
- ISO8601 timestamp formatting.
- Header value truncation to JsonSockLogMaxHeaderValueLen.
- Control character escaping in JSON strings.
execution:
- docker build -f Dockerfile.tests .
- docker run --rm <image> ctest --output-on-failure
integration_tests:
framework: python3
location: tests/integration/test_integration.py
env:
server: apache-httpd 2.4
os: rocky-linux-8+
log_consumer: simple Unix socket server capturing JSON lines
log_consumer: Unix socket server (Python threading)
scenarios:
- name: basic_logging
description: >
With JsonSockLogEnabled On and valid socket, verify that each request
produces a valid JSON line with expected fields.
produces a valid JSON line with all required fields.
checks:
- All required fields present (time, timestamp, src_ip, dst_ip, method, path, host)
- Field types correct (timestamp is integer, time is ISO8601 string)
- Method matches HTTP request method
- name: header_limits
description: >
Configure more headers than JsonSockLogMaxHeaders and verify only
the first N are logged and values are truncated according to
JsonSockLogMaxHeaderValueLen.
the first N are logged and values are truncated.
checks:
- Header values truncated to JsonSockLogMaxHeaderValueLen (default: 256)
- Only configured headers appear in output
- name: socket_unavailable_on_start
description: >
Start Apache with JsonSockLogEnabled On but socket not yet created;
verify periodic reconnect attempts and throttled error logging.
checks:
- Requests succeed even when socket unavailable
- Module reconnects when socket becomes available
- name: runtime_socket_loss
description: >
Drop the Unix socket while traffic is ongoing; verify that log lines
are dropped, worker threads are not blocked, and reconnect attempts
resume once the socket reappears.
checks:
- Requests complete quickly (<2s) when socket is down
- Module recovers and logs again after socket restoration
execution:
- python3 tests/integration/test_integration.py --url http://localhost:8080
bash_tests:
location: scripts/run_integration_tests.sh
description: >
Legacy bash-based integration tests for simple validation.
Tests JSON format, required fields, header logging via curl and grep.
execution:
- bash scripts/run_integration_tests.sh
ci:
strategy:
description: >
All builds, tests and packaging are executed inside Docker containers.
The host only needs Docker and the CI runner.
The host only needs Docker and the CI runner (GitHub Actions).
tools:
orchestrator: "to-define (GitLab CI / GitHub Actions / autre)"
orchestrator: GitHub Actions
container_engine: docker
workflow_file: .github/workflows/ci.yml
stages:
- name: build
description: >
@ -305,6 +364,7 @@ ci:
- name: build-rocky-8
image: "rockylinux:8"
steps:
- checkout: actions/checkout@v4
- install_deps:
- gcc
- make
@ -314,41 +374,58 @@ ci:
- apr-util-devel
- rpm-build
- build_module:
command: "apxs -c -i src/mod_reqin_log.c"
command: "make APXS=/usr/bin/apxs"
- verify:
command: "ls -la modules/mod_reqin_log.so"
- upload_artifact: actions/upload-artifact@v4
- name: build-debian
image: "debian:stable"
steps:
- checkout: actions/checkout@v4
- install_deps:
- build-essential
- apache2
- apache2-dev
- debhelper
- build_module:
command: "apxs -c -i src/mod_reqin_log.c"
command: "make APXS=/usr/bin/apxs"
- verify:
command: "ls -la modules/mod_reqin_log.so"
- upload_artifact: actions/upload-artifact@v4
- name: test
description: >
Run unit tests (C) and integration tests (Apache + Unix socket consumer)
inside Docker containers.
Run unit tests (C with cmocka) inside Docker containers.
Integration tests require a running Apache instance.
jobs:
- name: unit-tests
image: "debian:stable"
steps:
- install_test_deps:
- build-essential
- cmake
- "test-framework (à définir: cmocka, criterion, ...)"
- run_tests:
command: "ctest || make test"
- name: integration-tests-rocky-8
image: "rockylinux:8"
steps:
- prepare_apache_and_module
- start_unix_socket_consumer
- run_http_scenarios:
description: >
Validate JSON logs, header limits, socket loss and reconnect
behaviour using curl/ab/siege or similar tools.
- checkout: actions/checkout@v4
- install_deps:
- gcc
- make
- httpd
- httpd-devel
- apr-devel
- apr-util-devel
- cmake
- git
- pkgconfig
- libxml2-devel
- build_cmocka:
description: "Build cmocka from source (not available in EPEL)"
command: |
cd /tmp && git clone https://git.cryptomilk.org/projects/cmocka.git
cd cmocka && git checkout cmocka-1.1.5
mkdir build && cd build
cmake .. -DCMAKE_INSTALL_PREFIX=/usr -DCMAKE_BUILD_TYPE=Release
make && make install && ldconfig
- build_tests:
command: |
mkdir -p build/tests
cd build/tests && cmake ../../ && make
- run_tests:
command: "cd build/tests && ctest --output-on-failure"
- name: package
description: >
@ -360,37 +437,48 @@ ci:
- install_deps:
- rpm-build
- rpmlint
- "build deps same as 'build-rocky-8'"
- gcc
- make
- httpd
- httpd-devel
- create_tarball:
command: "tar -czf mod_reqin_log-1.0.0.tar.gz --transform 's,^,mod_reqin_log-1.0.0/,' ."
- setup_rpmbuild:
command: |
mkdir -p ~/rpmbuild/{BUILD,RPMS,SOURCES,SPECS,SRPMS}
cp mod_reqin_log-1.0.0.tar.gz ~/rpmbuild/SOURCES/
cp packaging/rpm/mod_reqin_log.spec ~/rpmbuild/SPECS/
- build_rpm:
spec_file: "packaging/rpm/mod_reqin_log.spec"
command: "rpmbuild -ba packaging/rpm/mod_reqin_log.spec"
- artifacts:
paths:
- "dist/rpm/**/*.rpm"
command: "rpmbuild -ba ~/rpmbuild/SPECS/mod_reqin_log.spec"
- upload_artifact:
paths: "~/rpmbuild/RPMS/x86_64/*.rpm"
- name: deb-debian
image: "debian:stable"
steps:
- install_deps:
- devscripts
- build-essential
- apache2
- apache2-dev
- debhelper
- devscripts
- dpkg-dev
- "build deps same as 'build-debian'"
- build_deb:
- setup_package:
command: |
cd packaging/deb
debuild -us -uc
- artifacts:
paths:
- "dist/deb/**/*.deb"
cp -r packaging/deb/* ./debian/
# Create changelog
- build_deb:
command: "debuild -us -uc -b"
- upload_artifact:
paths: "../*.deb"
artifacts:
retention:
policy: "keep build logs and packages long enough for debugging (to define)"
policy: "Keep build logs and packages for 30 days for debugging"
outputs:
- type: module
path: "dist/modules/mod_reqin_log.so"
path: "modules/mod_reqin_log.so"
- type: rpm
path: "dist/rpm/"
path: "~/rpmbuild/RPMS/x86_64/"
- type: deb
path: "dist/deb/"
path: "../"