docs: add gosearch integration design spec
Add comprehensive design document for integrating gosearch (kafka) metasearch engine into the NixOS VPS configuration. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
This commit is contained in:
parent
bd3295e918
commit
ea372d056e
1 changed files with 210 additions and 0 deletions
210
docs/superpowers/specs/2026-03-21-gosearch-integration-design.md
Normal file
210
docs/superpowers/specs/2026-03-21-gosearch-integration-design.md
Normal file
|
|
@ -0,0 +1,210 @@
|
||||||
|
# gosearch Integration Design
|
||||||
|
|
||||||
|
**Date:** 2026-03-21
|
||||||
|
**Status:** Approved
|
||||||
|
**Author:** Claude Code
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Integrate [gosearch](https://git.ashisgreat.xyz/penal-colony/gosearch) (kafka), a privacy-respecting Go-based metasearch engine, into the NixOS VPS configuration. The service will run as a hardened systemd service, connect to the existing Valkey cache, and be exposed via Nginx reverse proxy.
|
||||||
|
|
||||||
|
## Requirements
|
||||||
|
|
||||||
|
- Deploy gosearch using its native NixOS module
|
||||||
|
- Expose on `search2.ashisgreat.xyz`
|
||||||
|
- Connect to existing Valkey instance (shared with SearXNG)
|
||||||
|
- Enable Brave Search with API key
|
||||||
|
- Run on port `8889` (localhost only)
|
||||||
|
- Harden the service following systemd security best practices
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
```
|
||||||
|
┌─────────────────────────────────────────────────────────────┐
|
||||||
|
│ User Request Flow │
|
||||||
|
│ Browser → search2.ashisgreat.xyz → Nginx → localhost:8889 │
|
||||||
|
└─────────────────────────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────┐
|
||||||
|
│ systemd kafka service │
|
||||||
|
│ - User: kafka | Port: 8889 | Hardened systemd │
|
||||||
|
└─────────────────────────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────┐
|
||||||
|
│ Configuration Layer │
|
||||||
|
│ /etc/kafka/config.toml (generated via sops template) │
|
||||||
|
└─────────────────────────────────────────────────────────────┘
|
||||||
|
│
|
||||||
|
▼
|
||||||
|
┌─────────────────────────────────────────────────────────────┐
|
||||||
|
│ Dependencies │
|
||||||
|
│ - Valkey: 127.0.0.1:6379, DB index 1 │
|
||||||
|
│ - Brave API: from SOPS │
|
||||||
|
└─────────────────────────────────────────────────────────────┘
|
||||||
|
```
|
||||||
|
|
||||||
|
## Implementation
|
||||||
|
|
||||||
|
### 1. Add Flake Input
|
||||||
|
|
||||||
|
```nix
|
||||||
|
# flake.nix
|
||||||
|
inputs.gosearch.url = "git+ssh://forgejo@git.ashisgreat.xyz:2222/penal-colony/gosearch.git";
|
||||||
|
```
|
||||||
|
|
||||||
|
### 2. Import Module
|
||||||
|
|
||||||
|
```nix
|
||||||
|
# configuration.nix
|
||||||
|
imports = [
|
||||||
|
...
|
||||||
|
inputs.gosearch.nixosModules.default
|
||||||
|
];
|
||||||
|
```
|
||||||
|
|
||||||
|
### 3. Service Configuration
|
||||||
|
|
||||||
|
```nix
|
||||||
|
# configuration.nix
|
||||||
|
services.kafka = {
|
||||||
|
enable = true;
|
||||||
|
port = 8889;
|
||||||
|
baseUrl = "https://search2.ashisgreat.xyz";
|
||||||
|
config = config.sops.templates."kafka-config.toml".path;
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
### 4. SOPS Template
|
||||||
|
|
||||||
|
```nix
|
||||||
|
# configuration.nix
|
||||||
|
sops.templates."kafka-config.toml" = {
|
||||||
|
content = ''
|
||||||
|
[server]
|
||||||
|
port = 8889
|
||||||
|
http_timeout = "10s"
|
||||||
|
base_url = "https://search2.ashisgreat.xyz"
|
||||||
|
|
||||||
|
[engines]
|
||||||
|
local_ported = ["wikipedia", "arxiv", "crossref", "braveapi", "qwant", "duckduckgo", "github", "reddit", "bing"]
|
||||||
|
|
||||||
|
[engines.brave]
|
||||||
|
api_key = "${config.sops.placeholder.openclaw_brave_api_key}"
|
||||||
|
|
||||||
|
[cache]
|
||||||
|
address = "127.0.0.1:6379"
|
||||||
|
db = 1
|
||||||
|
default_ttl = "5m"
|
||||||
|
|
||||||
|
[cors]
|
||||||
|
allowed_origins = ["*"]
|
||||||
|
|
||||||
|
[rate_limit]
|
||||||
|
requests = 30
|
||||||
|
window = "1m"
|
||||||
|
cleanup_interval = "5m"
|
||||||
|
|
||||||
|
[global_rate_limit]
|
||||||
|
requests = 0
|
||||||
|
window = "1m"
|
||||||
|
|
||||||
|
[burst_rate_limit]
|
||||||
|
burst = 0
|
||||||
|
burst_window = "5s"
|
||||||
|
sustained = 0
|
||||||
|
sustained_window = "1m"
|
||||||
|
'';
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
### 5. Systemd Hardening
|
||||||
|
|
||||||
|
```nix
|
||||||
|
# configuration.nix
|
||||||
|
systemd.services.kafka.serviceConfig = {
|
||||||
|
# Capability bounds
|
||||||
|
CapabilityBoundingSet = [ "" ];
|
||||||
|
AmbientCapabilities = [ "" ];
|
||||||
|
|
||||||
|
# Filesystem
|
||||||
|
ProtectSystem = "strict";
|
||||||
|
ProtectHome = true;
|
||||||
|
ReadWritePaths = [ "/var/lib/kafka" ];
|
||||||
|
PrivateTmp = true;
|
||||||
|
|
||||||
|
# Network
|
||||||
|
PrivateDevices = true;
|
||||||
|
RestrictAddressFamilies = [ "AF_INET" "AF_INET6" "AF_UNIX" ];
|
||||||
|
|
||||||
|
# Process isolation
|
||||||
|
ProtectProc = "invisible";
|
||||||
|
ProcSubset = "pid";
|
||||||
|
NoNewPrivileges = true;
|
||||||
|
ProtectClock = true;
|
||||||
|
ProtectHostname = true;
|
||||||
|
|
||||||
|
# System call filtering
|
||||||
|
SystemCallFilter = [ "@system-service" "~@privileged" ];
|
||||||
|
SystemCallArchitectures = "native";
|
||||||
|
|
||||||
|
# Memory
|
||||||
|
MemoryDenyWriteExecute = true;
|
||||||
|
RestrictRealtime = true;
|
||||||
|
RestrictSUIDSGID = true;
|
||||||
|
|
||||||
|
# Resource limits
|
||||||
|
RestrictNamespaces = true;
|
||||||
|
LockPersonality = true;
|
||||||
|
RemoveIPC = true;
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
### 6. Nginx Configuration
|
||||||
|
|
||||||
|
Add to `myModules.nginx.domains`:
|
||||||
|
|
||||||
|
```nix
|
||||||
|
"search2.ashisgreat.xyz" = {
|
||||||
|
port = 8889;
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
### 7. Valkey Host Port Forward
|
||||||
|
|
||||||
|
Since gosearch runs as a systemd service (not in podman network), add a host port forward for Valkey:
|
||||||
|
|
||||||
|
```nix
|
||||||
|
# modules/searxng.nix - modify valkey container
|
||||||
|
virtualisation.oci-containers.containers."searxng-valkey" = {
|
||||||
|
...
|
||||||
|
ports = [ "127.0.0.1:6379:6379" ];
|
||||||
|
};
|
||||||
|
```
|
||||||
|
|
||||||
|
## Security Considerations
|
||||||
|
|
||||||
|
1. **Network Exposure**: Service binds to localhost only; Nginx is the only external access
|
||||||
|
2. **Secrets Management**: Brave API key from existing SOPS secret
|
||||||
|
3. **Cache Isolation**: Uses DB index 1 (SearXNG uses 0)
|
||||||
|
4. **Process Isolation**: Comprehensive systemd hardening applied
|
||||||
|
5. **Supply Chain**: Binary built from source via Nix flakes
|
||||||
|
|
||||||
|
## Files to Modify
|
||||||
|
|
||||||
|
1. `flake.nix` - Add gosearch input
|
||||||
|
2. `configuration.nix` - Import module, configure service, add SOPS template, add Nginx domain, add systemd hardening
|
||||||
|
3. `modules/searxng.nix` - Add Valkey host port forward
|
||||||
|
4. `secrets/secrets.yaml` - No changes needed (reusing existing Brave key)
|
||||||
|
|
||||||
|
## Testing Checklist
|
||||||
|
|
||||||
|
- [ ] Configuration builds: `nixos-rebuild build`
|
||||||
|
- [ ] Service starts: `systemctl status kafka`
|
||||||
|
- [ ] Service is hardened: `systemctl show kafka | grep -i protect`
|
||||||
|
- [ ] Web interface loads: `curl -I http://localhost:8889`
|
||||||
|
- [ ] Nginx proxy works: `curl -I https://search2.ashisgreat.xyz`
|
||||||
|
- [ ] Valkey connection works: check service logs
|
||||||
|
- [ ] Search returns results
|
||||||
|
- [ ] Health check: `curl http://localhost:8889/healthz`
|
||||||
Loading…
Add table
Add a link
Reference in a new issue