feat: ja4-platform monorepo — 5 services unified, tests & RPM builds standardized
Services: - ja4sentinel: TLS/JA4 fingerprint capture daemon (Go, libpcap) - logcorrelator: JA4 log correlation engine (Go, ClickHouse) - mod_reqin_log: Apache module (C, JSON request logging) - bot_detector: ML bot detection pipeline (Python) - dashboard: FastAPI/Streamlit analytics UI (Python) Shared libraries: - shared/go/ja4common: logger, config, shutdown, ipfilter (Go module) - shared/python/ja4_common: ClickHouseClient, ClickHouseSettings (Python package) - shared/clickhouse/: canonical SQL migrations (10 files) Build & packaging: - Unified 3-stage Dockerfile.package for Go RPMs (el8/el9/el10) - go.work workspace linking sentinel, correlator, ja4common - Makefile with test-all, build-all, rpm-* targets Fixes applied: - go.work: 1.21 → 1.24.6 (required by sentinel) - correlator Dockerfiles: golang:1.21 → golang:1.24 - replace directives in go.mod for ja4common local path - pyproject.toml: setuptools.backends → setuptools.build_meta - Removed static libpcap linking (unavailable on Rocky 9) - Fixed data races in output/writers_test.go (sync.Mutex + atomic.Int32) - Rewrote corrupted test files (logger_test.go × 2) Test coverage: - correlator: 67.1% total (unixsocket 80.5%, config 91.7%, app 83.3%, multi 87.7%, stdout 100%) - sentinel: all 10 packages pass (api, capture, config, fingerprint, ipfilter, logging, output, tlsparse) Documentation: - README.md + docs/ (architecture, development, 5 services, shared libs, DB schema & migrations) Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
This commit is contained in:
toto
284
services/mod-reqin-log/README.md
Normal file
284
services/mod-reqin-log/README.md
Normal file
@ -0,0 +1,284 @@
|
||||
# mod_reqin_log
|
||||
|
||||
Apache HTTPD 2.4 module for logging all incoming HTTP requests as JSON lines to a Unix domain socket.
|
||||
|
||||
## Features
|
||||
|
||||
- **Non-blocking I/O**: Logging never blocks worker processes
|
||||
- **Request-time logging**: Logs at `post_read_request` phase, capturing request data before application processing
|
||||
- **Configurable headers**: Select which HTTP headers to include in logs
|
||||
- **Header truncation**: Limit header value length to protect against oversized logs
|
||||
- **Automatic reconnection**: Reconnects to Unix socket on failure with configurable backoff
|
||||
- **Throttled error reporting**: Prevents error_log flooding on persistent failures
|
||||
- **MPM compatible**: Works with prefork, worker, and event MPMs
|
||||
- **Built-in security**: Sensitive headers (Authorization, Cookie, etc.) are automatically excluded
|
||||
- **RPM packaging**: Standard RPM packages for Rocky Linux 8/9 and AlmaLinux 10
|
||||
|
||||
## Requirements
|
||||
|
||||
### Runtime
|
||||
- Apache HTTPD 2.4+
|
||||
- GCC compiler
|
||||
- APR development libraries
|
||||
- Apache development headers (`httpd-devel` or `apache2-dev`)
|
||||
|
||||
### Packaging (RPM)
|
||||
- Docker (for reproducible builds)
|
||||
- rpmbuild (inside Docker)
|
||||
|
||||
## Installation
|
||||
|
||||
### Using Docker (recommended)
|
||||
|
||||
```bash
|
||||
# Build all RPM packages (el8, el9, el10)
|
||||
make package
|
||||
|
||||
# Test RPM package installation
|
||||
make test-package-rpm-el8 # Test el8 RPM (Rocky 8/RHEL 8)
|
||||
make test-package-rpm-el9 # Test el9 RPM (Rocky 9/RHEL 9)
|
||||
make test-package-rpm-el10 # Test el10 RPM (AlmaLinux 10/RHEL 10)
|
||||
make test-package # Test all RPM packages
|
||||
```
|
||||
|
||||
### Build from Source
|
||||
|
||||
```bash
|
||||
# Clone or extract the source
|
||||
cd mod_reqin_log
|
||||
|
||||
# Build the module
|
||||
make
|
||||
|
||||
# Install (requires root privileges)
|
||||
sudo make install
|
||||
```
|
||||
|
||||
## Configuration
|
||||
|
||||
Load the module and configure in your Apache configuration:
|
||||
|
||||
```apache
|
||||
# Load the module
|
||||
LoadModule reqin_log_module modules/mod_reqin_log.so
|
||||
|
||||
# Enable logging
|
||||
JsonSockLogEnabled On
|
||||
|
||||
# Unix socket path
|
||||
JsonSockLogSocket "/var/run/logcorrelator/http.socket"
|
||||
|
||||
# Headers to log (be careful not to log sensitive data)
|
||||
JsonSockLogHeaders X-Request-Id X-Trace-Id User-Agent Referer
|
||||
|
||||
# Maximum headers to log
|
||||
JsonSockLogMaxHeaders 10
|
||||
|
||||
# Maximum header value length
|
||||
JsonSockLogMaxHeaderValueLen 256
|
||||
|
||||
# Reconnect interval (seconds)
|
||||
JsonSockLogReconnectInterval 10
|
||||
|
||||
# Error report interval (seconds)
|
||||
JsonSockLogErrorReportInterval 10
|
||||
```
|
||||
|
||||
> **Important startup validation:** if `JsonSockLogEnabled On` is set without a valid `JsonSockLogSocket`, Apache startup fails with a configuration error.
|
||||
|
||||
### Configuration Directives
|
||||
|
||||
| Directive | Type | Default | Description |
|
||||
|-----------|------|---------|-------------|
|
||||
| `JsonSockLogEnabled` | On/Off | Off | Enable or disable logging |
|
||||
| `JsonSockLogSocket` | String | - | Unix domain socket path |
|
||||
| `JsonSockLogHeaders` | List | - | HTTP headers to include |
|
||||
| `JsonSockLogMaxHeaders` | Integer | 10 | Max headers to log |
|
||||
| `JsonSockLogMaxHeaderValueLen` | Integer | 256 | Max header value length |
|
||||
| `JsonSockLogReconnectInterval` | Integer | 10 | Reconnect delay (seconds) |
|
||||
| `JsonSockLogErrorReportInterval` | Integer | 10 | Error log throttle (seconds) |
|
||||
|
||||
## JSON Log Format
|
||||
|
||||
Each log entry is a single-line JSON object with a flat structure:
|
||||
|
||||
```json
|
||||
{
|
||||
"time": "2026-02-26T11:59:30Z",
|
||||
"timestamp": 1708948770000000000,
|
||||
"src_ip": "192.0.2.10",
|
||||
"src_port": 45678,
|
||||
"dst_ip": "198.51.100.5",
|
||||
"dst_port": 443,
|
||||
"method": "GET",
|
||||
"path": "/api/users",
|
||||
"host": "example.com",
|
||||
"http_version": "HTTP/1.1",
|
||||
"header_X-Request-Id": "abcd-1234",
|
||||
"header_User-Agent": "curl/7.70.0"
|
||||
}
|
||||
```
|
||||
|
||||
### Fields
|
||||
|
||||
| Field | Type | Description |
|
||||
|-------|------|-------------|
|
||||
| `time` | String | ISO8601 timestamp with timezone |
|
||||
| `timestamp` | Integer | Microseconds since epoch (expressed as nanoseconds for compatibility) |
|
||||
| `src_ip` | String | Client IP address |
|
||||
| `src_port` | Integer | Client port |
|
||||
| `dst_ip` | String | Server IP address |
|
||||
| `dst_port` | Integer | Server port |
|
||||
| `method` | String | HTTP method |
|
||||
| `path` | String | Request path |
|
||||
| `host` | String | Host header value |
|
||||
| `http_version` | String | HTTP protocol version |
|
||||
| `header_<Name>` | String | Flattened HTTP headers (e.g., `header_X-Request-Id`) |
|
||||
|
||||
**Note:** Headers are logged as flat fields at the root level (not nested). Sensitive headers are automatically excluded. The `timestamp` field has microsecond precision (APR's `apr_time_now()` returns microseconds, multiplied by 1000 for nanosecond representation).
|
||||
|
||||
## Unix Socket Consumer
|
||||
|
||||
Create a Unix socket listener to receive log entries:
|
||||
|
||||
```python
|
||||
#!/usr/bin/env python3
|
||||
import socket
|
||||
import os
|
||||
import json
|
||||
|
||||
SOCKET_PATH = os.environ.get("MOD_REQIN_LOG_SOCKET", "/var/run/logcorrelator/http.socket")
|
||||
|
||||
# Remove existing socket file
|
||||
if os.path.exists(SOCKET_PATH):
|
||||
os.remove(SOCKET_PATH)
|
||||
|
||||
# Create Unix socket server
|
||||
server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
|
||||
server.bind(SOCKET_PATH)
|
||||
server.listen(5)
|
||||
# Set secure permissions: owner and group only (not world-writable)
|
||||
os.chmod(SOCKET_PATH, 0o660)
|
||||
|
||||
print(f"Listening on {SOCKET_PATH}")
|
||||
|
||||
while True:
|
||||
conn, addr = server.accept()
|
||||
data = b""
|
||||
while True:
|
||||
chunk = conn.recv(4096)
|
||||
if not chunk:
|
||||
break
|
||||
data += chunk
|
||||
if b"\n" in data:
|
||||
for line in data.decode().strip().split("\n"):
|
||||
if line:
|
||||
log_entry = json.loads(line)
|
||||
print(json.dumps(log_entry, indent=2))
|
||||
data = b""
|
||||
conn.close()
|
||||
```
|
||||
|
||||
**Note:** Ensure the Apache user is in the socket file's group to allow connections.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Built-in Sensitive Headers Blacklist
|
||||
|
||||
⚠️ **The module automatically blocks logging of sensitive headers:**
|
||||
|
||||
The following headers are **always excluded** from logs to prevent credential leakage:
|
||||
- `Authorization`
|
||||
- `Cookie`, `Set-Cookie`
|
||||
- `X-Api-Key`, `X-Auth-Token`
|
||||
- `Proxy-Authorization`
|
||||
- `WWW-Authenticate`
|
||||
|
||||
These headers are silently skipped (logged at DEBUG level only).
|
||||
|
||||
### Socket Security
|
||||
|
||||
- **Socket permissions**: Default to `0o660` (owner and group only)
|
||||
- **Recommended path**: `/var/run/logcorrelator/http.socket` (not `/tmp`)
|
||||
- **Environment variable**: Use `MOD_REQIN_LOG_SOCKET` to configure path
|
||||
- **Group membership**: Ensure Apache user is in the socket's group
|
||||
|
||||
### Additional Hardening
|
||||
|
||||
- **Socket path length**: Validated against system limit (108 bytes max)
|
||||
- **JSON size limit**: 64KB max per log line (prevents memory DoS)
|
||||
- **NULL pointer checks**: All connection/request fields validated
|
||||
- **Thread safety**: Mutex protects socket FD in worker/event MPMs
|
||||
- **Error logging**: Generic messages in error_log, details at DEBUG level
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Module not loading
|
||||
|
||||
```
|
||||
AH00534: mod_reqin_log: Unable to load module
|
||||
```
|
||||
|
||||
Ensure the module path is correct and the file exists:
|
||||
```bash
|
||||
ls -la /usr/lib/apache2/modules/mod_reqin_log.so
|
||||
```
|
||||
|
||||
### Socket connection failures
|
||||
|
||||
```
|
||||
[mod_reqin_log] Unix socket connect failed: /var/run/logcorrelator/http.socket
|
||||
```
|
||||
|
||||
- Ensure the socket consumer is running
|
||||
- Check socket file permissions
|
||||
- Verify SELinux/AppArmor policies if applicable
|
||||
|
||||
### No logs appearing
|
||||
|
||||
1. Verify `JsonSockLogEnabled On` is set
|
||||
2. Verify `JsonSockLogSocket` path is configured
|
||||
3. Check Apache error log for module errors
|
||||
4. Ensure socket consumer is listening
|
||||
|
||||
## Testing
|
||||
|
||||
### Run Unit Tests
|
||||
|
||||
```bash
|
||||
# Using Docker (recommended)
|
||||
docker build -f Dockerfile.tests -t mod_reqin_log:tests .
|
||||
docker run --rm mod_reqin_log:tests ctest --output-on-failure
|
||||
|
||||
# Or locally with cmocka
|
||||
sudo dnf install cmocka-devel # Rocky Linux
|
||||
sudo apt install libcmocka-dev # Debian/Ubuntu
|
||||
mkdir build && cd build
|
||||
cmake ..
|
||||
make test
|
||||
```
|
||||
|
||||
### Build and Test Packages
|
||||
|
||||
```bash
|
||||
# Build all RPM packages (el8, el9, el10)
|
||||
make package
|
||||
|
||||
# Test RPM package installation
|
||||
make test-package-rpm-el8 # Test el8 RPM in Docker
|
||||
make test-package-rpm-el9 # Test el9 RPM in Docker
|
||||
make test-package-rpm-el10 # Test el10 RPM in Docker
|
||||
make test-package # Test all RPM packages
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
Apache License 2.0
|
||||
|
||||
## Contributing
|
||||
|
||||
1. Fork the repository
|
||||
2. Create a feature branch
|
||||
3. Make your changes
|
||||
4. Run tests
|
||||
5. Submit a pull request
|
||||
Reference in New Issue
Block a user