code_public/mod_reqin_log
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Initial commit: mod_reqin_log Apache module

Browse Source 
Features:
- JSON logging of HTTP requests to Unix domain socket
- Configurable HTTP headers logging (flat JSON structure)
- Header value truncation and count limits
- Automatic reconnect on socket disconnection
- Error reporting with throttling

Configuration directives:
- JsonSockLogEnabled: Enable/disable logging
- JsonSockLogSocket: Unix socket path
- JsonSockLogHeaders: List of headers to log
- JsonSockLogMaxHeaders: Maximum headers to log
- JsonSockLogMaxHeaderValueLen: Max header value length
- JsonSockLogReconnectInterval: Reconnect delay
- JsonSockLogErrorReportInterval: Error log throttle

Includes:
- Module source code (src/)
- Unit and integration tests (tests/, scripts/)
- Documentation (README.md, architecture.yml)
- Build configuration (CMakeLists.txt, Makefile)
- Packaging (deb/rpm)

Co-authored-by: Qwen-Coder <qwen-coder@alibabacloud.com>
This commit is contained in:
Jacquin Antoine
2026-02-26 13:55:07 +01:00
commit 66549acf5c
27 changed files with 3550 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

259
README.md Normal file
View File

@ -0,0 +1,259 @@
# mod_reqin_log
Apache HTTPD 2.4 module for logging all incoming HTTP requests as JSON lines to a Unix domain socket.
## Features
- **Non-blocking I/O**: Logging never blocks worker processes
- **Request-time logging**: Logs at `post_read_request` phase, capturing request data before application processing
- **Configurable headers**: Select which HTTP headers to include in logs
- **Header truncation**: Limit header value length to protect against oversized logs
- **Automatic reconnection**: Reconnects to Unix socket on failure with configurable backoff
- **Throttled error reporting**: Prevents error_log flooding on persistent failures
- **MPM compatible**: Works with prefork, worker, and event MPMs
## Requirements
- Apache HTTPD 2.4+
- GCC compiler
- APR development libraries
- Apache development headers (`httpd-devel` or `apache2-dev`)
## Installation
### Build from Source
```bash
# Clone or extract the source
cd mod_reqin_log
# Build the module
make
# Install (requires root privileges)
sudo make install
```
### Using Package Manager
#### RPM (Rocky Linux 8+)
```bash
# Build RPM package
rpmbuild -ba packaging/rpm/mod_reqin_log.spec
# Install the package
sudo rpm -ivh ~/rpmbuild/RPMS/x86_64/mod_reqin_log-1.0.0-1.el8.x86_64.rpm
```
#### DEB (Debian/Ubuntu)
```bash
# Build DEB package
cd packaging/deb
debuild -us -uc
# Install the package
sudo dpkg -i ../libapache2-mod-reqin-log_1.0.0_amd64.deb
```
## Configuration
Load the module and configure in your Apache configuration:
```apache
# Load the module
LoadModule reqin_log_module modules/mod_reqin_log.so
# Enable logging
JsonSockLogEnabled On
# Unix socket path
JsonSockLogSocket "/var/run/mod_reqin_log.sock"
# Headers to log (be careful not to log sensitive data)
JsonSockLogHeaders X-Request-Id X-Trace-Id User-Agent Referer
# Maximum headers to log
JsonSockLogMaxHeaders 10
# Maximum header value length
JsonSockLogMaxHeaderValueLen 256
# Reconnect interval (seconds)
JsonSockLogReconnectInterval 10
# Error report interval (seconds)
JsonSockLogErrorReportInterval 10
```
### Configuration Directives
| Directive | Type | Default | Description |
|-----------|------|---------|-------------|
| `JsonSockLogEnabled` | On/Off | Off | Enable or disable logging |
| `JsonSockLogSocket` | String | - | Unix domain socket path |
| `JsonSockLogHeaders` | List | - | HTTP headers to include |
| `JsonSockLogMaxHeaders` | Integer | 10 | Max headers to log |
| `JsonSockLogMaxHeaderValueLen` | Integer | 256 | Max header value length |
| `JsonSockLogReconnectInterval` | Integer | 10 | Reconnect delay (seconds) |
| `JsonSockLogErrorReportInterval` | Integer | 10 | Error log throttle (seconds) |
## JSON Log Format
Each log entry is a single-line JSON object with a flat structure:
```json
{
"time": "2026-02-26T11:59:30Z",
"timestamp": 1708948770000000000,
"src_ip": "192.0.2.10",
"src_port": 45678,
"dst_ip": "198.51.100.5",
"dst_port": 443,
"method": "GET",
"path": "/api/users",
"host": "example.com",
"http_version": "HTTP/1.1",
"header_X-Request-Id": "abcd-1234",
"header_User-Agent": "curl/7.70.0"
}
}
```
### Fields
| Field | Type | Description |
|-------|------|-------------|
| `time` | String | ISO8601 timestamp with timezone |
| `timestamp` | Integer | Nanoseconds since epoch |
| `src_ip` | String | Client IP address |
| `src_port` | Integer | Client port |
| `dst_ip` | String | Server IP address |
| `dst_port` | Integer | Server port |
| `method` | String | HTTP method |
| `path` | String | Request path |
| `host` | String | Host header value |
| `http_version` | String | HTTP protocol version |
| `headers` | Object | Configured HTTP headers |
## Unix Socket Consumer
Create a Unix socket listener to receive log entries:
```python
#!/usr/bin/env python3
import socket
import os
import json
SOCKET_PATH = "/var/run/mod_reqin_log.sock"
# Remove existing socket file
if os.path.exists(SOCKET_PATH):
os.remove(SOCKET_PATH)
# Create Unix socket server
server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
server.bind(SOCKET_PATH)
server.listen(5)
os.chmod(SOCKET_PATH, 0o666)
print(f"Listening on {SOCKET_PATH}")
while True:
conn, addr = server.accept()
data = b""
while True:
chunk = conn.recv(4096)
if not chunk:
break
data += chunk
if b"\n" in data:
for line in data.decode().strip().split("\n"):
if line:
log_entry = json.loads(line)
print(json.dumps(log_entry, indent=2))
data = b""
conn.close()
```
## Security Considerations
⚠️ **Important**: This module logs all HTTP requests transparently.
- **Do not log sensitive headers**: Avoid including `Authorization`, `Cookie`, `X-Api-Key`, or other sensitive headers in `JsonSockLogHeaders`
- **Socket permissions**: Ensure the Unix socket has appropriate file permissions
- **Log consumer security**: Protect the socket consumer from unauthorized access
## Troubleshooting
### Module not loading
```
AH00534: mod_reqin_log: Unable to load module
```
Ensure the module path is correct and the file exists:
```bash
ls -la /usr/lib/apache2/modules/mod_reqin_log.so
```
### Socket connection failures
```
[mod_reqin_log] Unix socket connect failed: /var/run/mod_reqin_log.sock
```
- Ensure the socket consumer is running
- Check socket file permissions
- Verify SELinux/AppArmor policies if applicable
### No logs appearing
1. Verify `JsonSockLogEnabled On` is set
2. Verify `JsonSockLogSocket` path is configured
3. Check Apache error log for module errors
4. Ensure socket consumer is listening
## Testing
### Run Unit Tests
```bash
# Install test dependencies
sudo dnf install cmocka-devel # Rocky Linux
sudo apt install libcmocka-dev # Debian/Ubuntu
# Build and run tests
mkdir build && cd build
cmake ..
make test
```
### Integration Testing
```bash
# Start socket consumer
python3 scripts/socket_consumer.py &
# Start Apache with module enabled
sudo systemctl start httpd
# Send test requests
curl -H "X-Request-Id: test-123" http://localhost/
# Check consumer output
```
## License
Apache License 2.0
## Contributing
1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run tests
5. Submit a pull request