kafka/CLAUDE.md

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

kafka is a privacy-respecting metasearch engine written in Go. It provides a SearXNG-compatible /search API and an HTML frontend (HTMX + Go templates). 9 engines are implemented natively in Go; unlisted engines can be proxied to an upstream SearXNG instance. Responses from multiple engines are merged into a single JSON/CSV/RSS/HTML response.

Build & Run Commands

# Enter Nix dev shell (provides Go 1.24 toolchain + curl)
nix develop

# Run all tests
go test ./...

# Run a single test
go test -run TestWikipedia ./internal/engines/

# Run tests in a specific package with verbose output
go test -v ./internal/engines/

# Run the server (requires config.toml)
go run ./cmd/searxng-go -config config.toml

There is no Makefile. There is no linter configured.

Architecture

Request flow: HTTP request -> middleware chain (global rate limit -> burst rate limit -> per-IP rate limit -> CORS) -> HTTP handler -> search.Service (cache check) -> engines.Planner (splits into local vs upstream) -> parallel local engine execution + upstream proxy -> MergeResponses -> cache write -> serialize (JSON/CSV/RSS/HTML).

Key packages:

• internal/contracts — Shared types: SearchRequest, SearchResponse, MainResult, OutputFormat. MainResult preserves unknown JSON keys from upstream via a raw map[string]any field and round-trips them faithfully.
• internal/config — TOML-based configuration with env var fallbacks. Load(path) reads config.toml; env vars override zero-value fields. See config.example.toml for all settings.
• internal/engines — Engine interface and all 9 Go-native implementations. factory.go registers engines via NewDefaultPortedEngines(). planner.go routes engines to local or upstream based on LOCAL_PORTED_ENGINES env var.
• internal/search — Service orchestrates the pipeline: cache check, planning, parallel engine execution via goroutines/WaitGroup, upstream proxying, response merging. Individual engine failures are reported as unresponsive_engines rather than aborting the search. Qwant has fallback logic to upstream on empty results.
• internal/httpapi — HTTP handlers for /, /search, /healthz, /opensearch.xml. Detects HTMX requests via HX-Request header to return fragments instead of full pages.
• internal/upstream — Client that proxies requests to an upstream SearXNG instance via POST.
• internal/cache — Valkey/Redis-backed cache with SHA-256 cache keys. No-op if unconfigured.
• internal/middleware — Three rate limiters (per-IP sliding window, burst+sustained, global) and CORS. All disabled by default.
• internal/views — HTML templates and static files embedded via //go:embed. Renders full pages or HTMX fragments. Templates: base.html, index.html, results.html, results_inner.html, result_item.html.
• cmd/searxng-go — Entry point. Loads TOML config, seeds env vars for engine code, wires up middleware chain, starts HTTP server.

Engine interface (internal/engines/engine.go):

type Engine interface {
    Name() string
    Search(ctx context.Context, req contracts.SearchRequest) (contracts.SearchResponse, error)
}

Adding a new engine:

1. Create a new struct implementing the Engine interface in internal/engines/ (single file, e.g., newengine.go)
2. Add a test file alongside it (use roundTripperFunc and httpResponse helpers in http_mock_test.go for mocking HTTP)
3. Register it in NewDefaultPortedEngines() in factory.go
4. Add its name to defaultPortedEngines in planner.go
5. Add category mappings in inferFromCategories() if applicable

Configuration

Config is loaded from config.toml (see config.example.toml). All fields can be overridden via environment variables (env vars take precedence over zero-value TOML fields). Key sections: [server], [upstream], [engines], [cache], [cors], [rate_limit], [global_rate_limit], [burst_rate_limit].

Conventions

• Module path: github.com/metamorphosis-dev/kafka
• Tests use shared mock helpers in internal/engines/http_mock_test.go (roundTripperFunc, httpResponse)
• Engine implementations are single files under internal/engines/ (e.g., wikipedia.go, duckduckgo.go)
• Response merging de-duplicates by engine|title|url key; suggestions/corrections are merged as sets
• MainResult uses custom UnmarshalJSON/MarshalJSON to preserve unknown upstream JSON keys
• HTML templates and static files are embedded at build time via //go:embed in internal/views/
• Structured logging via log/slog with JSON handler