mod_reqin_log/README.md

# mod_reqin_log

Apache HTTPD 2.4 module for logging all incoming HTTP requests as JSON lines to a Unix domain socket.

## Features

- **Non-blocking I/O**: Logging never blocks worker processes
- **Request-time logging**: Logs at `post_read_request` phase, capturing request data before application processing
- **Configurable headers**: Select which HTTP headers to include in logs
- **Header truncation**: Limit header value length to protect against oversized logs
- **Automatic reconnection**: Reconnects to Unix socket on failure with configurable backoff
- **Throttled error reporting**: Prevents error_log flooding on persistent failures
- **MPM compatible**: Works with prefork, worker, and event MPMs

## Requirements

- Apache HTTPD 2.4+
- GCC compiler
- APR development libraries
- Apache development headers (`httpd-devel` or `apache2-dev`)

## Installation

### Using Docker (recommended)

```bash
# Build all packages (DEB + RPMs for el7, el8, el9)
make package

# Test package installation
make test-package-deb       # Test DEB in Docker container
make test-package-rpm-el7   # Test el7 RPM (CentOS 7/RHEL 7)
make test-package-rpm-el8   # Test el8 RPM (Rocky 8/RHEL 8)
make test-package-rpm-el9   # Test el9 RPM (Rocky 9/RHEL 9)
make test-package           # Test all packages
```

### Build from Source

```bash
# Clone or extract the source
cd mod_reqin_log

# Build the module
make

# Install (requires root privileges)
sudo make install
```

## Configuration

Load the module and configure in your Apache configuration:

```apache
# Load the module
LoadModule reqin_log_module modules/mod_reqin_log.so

# Enable logging
JsonSockLogEnabled On

# Unix socket path
JsonSockLogSocket "/var/run/mod_reqin_log.sock"

# Headers to log (be careful not to log sensitive data)
JsonSockLogHeaders X-Request-Id X-Trace-Id User-Agent Referer

# Maximum headers to log
JsonSockLogMaxHeaders 10

# Maximum header value length
JsonSockLogMaxHeaderValueLen 256

# Reconnect interval (seconds)
JsonSockLogReconnectInterval 10

# Error report interval (seconds)
JsonSockLogErrorReportInterval 10
```

> **Important startup validation:** if `JsonSockLogEnabled On` is set without a valid `JsonSockLogSocket`, Apache startup fails with a configuration error.

### Configuration Directives

| Directive | Type | Default | Description |
|-----------|------|---------|-------------|
| `JsonSockLogEnabled` | On/Off | Off | Enable or disable logging |
| `JsonSockLogSocket` | String | - | Unix domain socket path |
| `JsonSockLogHeaders` | List | - | HTTP headers to include |
| `JsonSockLogMaxHeaders` | Integer | 10 | Max headers to log |
| `JsonSockLogMaxHeaderValueLen` | Integer | 256 | Max header value length |
| `JsonSockLogReconnectInterval` | Integer | 10 | Reconnect delay (seconds) |
| `JsonSockLogErrorReportInterval` | Integer | 10 | Error log throttle (seconds) |

## JSON Log Format

Each log entry is a single-line JSON object with a flat structure:

```json
{
  "time": "2026-02-26T11:59:30Z",
  "timestamp": 1708948770000000000,
  "src_ip": "192.0.2.10",
  "src_port": 45678,
  "dst_ip": "198.51.100.5",
  "dst_port": 443,
  "method": "GET",
  "path": "/api/users",
  "host": "example.com",
  "http_version": "HTTP/1.1",
  "header_X-Request-Id": "abcd-1234",
  "header_User-Agent": "curl/7.70.0"
}
```

### Fields

| Field | Type | Description |
|-------|------|-------------|
| `time` | String | ISO8601 timestamp with timezone |
| `timestamp` | Integer | Nanoseconds since epoch |
| `src_ip` | String | Client IP address |
| `src_port` | Integer | Client port |
| `dst_ip` | String | Server IP address |
| `dst_port` | Integer | Server port |
| `method` | String | HTTP method |
| `path` | String | Request path |
| `host` | String | Host header value |
| `http_version` | String | HTTP protocol version |
| `header_<Name>` | String | Flattened HTTP headers (e.g., `header_X-Request-Id`) |

**Note:** Headers are logged as flat fields at the root level (not nested). Sensitive headers are automatically excluded.

## Unix Socket Consumer

Create a Unix socket listener to receive log entries:

```python
#!/usr/bin/env python3
import socket
import os
import json

SOCKET_PATH = os.environ.get("MOD_REQIN_LOG_SOCKET", "/var/run/mod_reqin_log.sock")

# Remove existing socket file
if os.path.exists(SOCKET_PATH):
    os.remove(SOCKET_PATH)

# Create Unix socket server
server = socket.socket(socket.AF_UNIX, socket.SOCK_STREAM)
server.bind(SOCKET_PATH)
server.listen(5)
# Set secure permissions: owner and group only (not world-writable)
os.chmod(SOCKET_PATH, 0o660)

print(f"Listening on {SOCKET_PATH}")

while True:
    conn, addr = server.accept()
    data = b""
    while True:
        chunk = conn.recv(4096)
        if not chunk:
            break
        data += chunk
        if b"\n" in data:
            for line in data.decode().strip().split("\n"):
                if line:
                    log_entry = json.loads(line)
                    print(json.dumps(log_entry, indent=2))
            data = b""
    conn.close()
```

**Note:** Ensure the Apache user is in the socket file's group to allow connections.

## Security Considerations

### Built-in Sensitive Headers Blacklist

⚠️ **The module automatically blocks logging of sensitive headers:**

The following headers are **always excluded** from logs to prevent credential leakage:
- `Authorization`
- `Cookie`, `Set-Cookie`
- `X-Api-Key`, `X-Auth-Token`
- `Proxy-Authorization`
- `WWW-Authenticate`

These headers are silently skipped (logged at DEBUG level only).

### Socket Security

- **Socket permissions**: Default to `0o660` (owner and group only)
- **Recommended path**: `/var/run/mod_reqin_log.sock` (not `/tmp`)
- **Environment variable**: Use `MOD_REQIN_LOG_SOCKET` to configure path
- **Group membership**: Ensure Apache user is in the socket's group

### Additional Hardening

- **Socket path length**: Validated against system limit (108 bytes max)
- **JSON size limit**: 64KB max per log line (prevents memory DoS)
- **NULL pointer checks**: All connection/request fields validated
- **Thread safety**: Mutex protects socket FD in worker/event MPMs
- **Error logging**: Generic messages in error_log, details at DEBUG level

## Troubleshooting

### Module not loading

```
AH00534: mod_reqin_log: Unable to load module
```

Ensure the module path is correct and the file exists:
```bash
ls -la /usr/lib/apache2/modules/mod_reqin_log.so
```

### Socket connection failures

```
[mod_reqin_log] Unix socket connect failed: /var/run/mod_reqin_log.sock
```

- Ensure the socket consumer is running
- Check socket file permissions
- Verify SELinux/AppArmor policies if applicable

### No logs appearing

1. Verify `JsonSockLogEnabled On` is set
2. Verify `JsonSockLogSocket` path is configured
3. Check Apache error log for module errors
4. Ensure socket consumer is listening

## Testing

### Run Unit Tests

```bash
# Using Docker (recommended)
docker build -f Dockerfile.tests -t mod_reqin_log:tests .
docker run --rm mod_reqin_log:tests ctest --output-on-failure

# Or locally with cmocka
sudo dnf install cmocka-devel  # Rocky Linux
sudo apt install libcmocka-dev # Debian/Ubuntu
mkdir build && cd build
cmake ..
make test
```

### Integration Testing

```bash
# Using GitLab CI (recommended)
# All integration tests run automatically in CI

# Or manually with the Python test suite
python3 tests/integration/test_integration.py --url http://localhost:8080
```

### Build and Test Packages

```bash
# Build all packages (DEB + RPMs for el7, el8, el9)
make package

# Test package installation
make test-package-deb       # Test DEB in Docker
make test-package-rpm-el7   # Test el7 RPM in Docker
make test-package-rpm-el8   # Test el8 RPM in Docker
make test-package-rpm-el9   # Test el9 RPM in Docker
make test-package           # Test all packages
```

## License

Apache License 2.0

## Contributing

1. Fork the repository
2. Create a feature branch
3. Make your changes
4. Run tests
5. Submit a pull request