+ {{.Title}} +
+ {{if .Content}} +{{.Content}}
+ {{end}} + {{if .Engine}} +diff --git a/.forgejo/workflows/test.yml b/.forgejo/workflows/test.yml index 6b09ea7..bd05693 100644 --- a/.forgejo/workflows/test.yml +++ b/.forgejo/workflows/test.yml @@ -16,7 +16,7 @@ jobs: - name: Set up Go uses: https://github.com/actions/setup-go@v5 with: - go-version: '1.24' + go-version-file: go.mod - name: Test run: go test -race -v ./... diff --git a/CLAUDE.md b/CLAUDE.md index bba67e1..b7f254e 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -4,7 +4,7 @@ This file provides guidance to Claude Code (claude.ai/code) when working with co ## Project Overview -kafka is a privacy-respecting metasearch engine written in Go. It provides a SearXNG-compatible `/search` API and an HTML frontend (HTMX + Go templates). 9 engines are implemented natively in Go; unlisted engines can be proxied to an upstream SearXNG instance. Responses from multiple engines are merged into a single JSON/CSV/RSS/HTML response. +kafka is a privacy-respecting metasearch engine written in Go. It provides a SearXNG-compatible `/search` API and an HTML frontend (HTMX + Go templates). 9 engines are implemented natively in Go; unlisted engines can be proxied to an upstream metasearch instance. Responses from multiple engines are merged into a single JSON/CSV/RSS/HTML response. ## Build & Run Commands @@ -22,7 +22,7 @@ go test -run TestWikipedia ./internal/engines/ go test -v ./internal/engines/ # Run the server (requires config.toml) -go run ./cmd/searxng-go -config config.toml +go run ./cmd/kafka -config config.toml ``` There is no Makefile. There is no linter configured. @@ -37,12 +37,13 @@ There is no Makefile. There is no linter configured. - `internal/config` — TOML-based configuration with env var fallbacks. `Load(path)` reads `config.toml`; env vars override zero-value fields. See `config.example.toml` for all settings. - `internal/engines` — `Engine` interface and all 9 Go-native implementations. `factory.go` registers engines via `NewDefaultPortedEngines()`. `planner.go` routes engines to local or upstream based on `LOCAL_PORTED_ENGINES` env var. - `internal/search` — `Service` orchestrates the pipeline: cache check, planning, parallel engine execution via goroutines/WaitGroup, upstream proxying, response merging. Individual engine failures are reported as `unresponsive_engines` rather than aborting the search. Qwant has fallback logic to upstream on empty results. -- `internal/httpapi` — HTTP handlers for `/`, `/search`, `/healthz`, `/opensearch.xml`. Detects HTMX requests via `HX-Request` header to return fragments instead of full pages. -- `internal/upstream` — Client that proxies requests to an upstream SearXNG instance via POST. +- `internal/autocomplete` — Fetches search suggestions. Proxies to upstream `/autocompleter` if configured, falls back to Wikipedia OpenSearch API otherwise. +- `internal/httpapi` — HTTP handlers for `/`, `/search`, `/autocompleter`, `/healthz`, `/opensearch.xml`. Detects HTMX requests via `HX-Request` header to return fragments instead of full pages. +- `internal/upstream` — Client that proxies requests to an upstream metasearch instance via POST. - `internal/cache` — Valkey/Redis-backed cache with SHA-256 cache keys. No-op if unconfigured. - `internal/middleware` — Three rate limiters (per-IP sliding window, burst+sustained, global) and CORS. All disabled by default. - `internal/views` — HTML templates and static files embedded via `//go:embed`. Renders full pages or HTMX fragments. Templates: `base.html`, `index.html`, `results.html`, `results_inner.html`, `result_item.html`. -- `cmd/searxng-go` — Entry point. Loads TOML config, seeds env vars for engine code, wires up middleware chain, starts HTTP server. +- `cmd/kafka` — Entry point. Loads TOML config, seeds env vars for engine code, wires up middleware chain, starts HTTP server. **Engine interface** (`internal/engines/engine.go`): ```go diff --git a/Dockerfile b/Dockerfile index c41b5a1..e21960f 100644 --- a/Dockerfile +++ b/Dockerfile @@ -11,7 +11,7 @@ RUN go mod download # Copy source and build COPY . . -RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /kafka ./cmd/searxng-go +RUN CGO_ENABLED=0 GOOS=linux go build -ldflags="-s -w" -o /kafka ./cmd/kafka # Runtime stage FROM alpine:3.21 diff --git a/README.md b/README.md index 2f0868f..c03019e 100644 --- a/README.md +++ b/README.md @@ -24,8 +24,8 @@ A privacy-respecting, open metasearch engine written in Go. SearXNG-compatible A ```bash git clone https://git.ashisgreat.xyz/penal-colony/gosearch.git cd kafka -go build ./cmd/searxng-go -./searxng-go -config config.toml +go build ./cmd/kafka +./kafka -config config.toml ``` ### Docker Compose @@ -76,7 +76,7 @@ sudo nixos-rebuild switch --flake .# ```bash nix develop go test ./... -go run ./cmd/searxng-go -config config.toml +go run ./cmd/kafka -config config.toml ``` ## Endpoints @@ -138,7 +138,7 @@ Copy `config.example.toml` to `config.toml` and edit. All settings can also be o ### Key Sections - **`[server]`** — port, timeout, public base URL for OpenSearch -- **`[upstream]`** — optional upstream SearXNG proxy for unported engines +- **`[upstream]`** — optional upstream metasearch proxy for unported engines - **`[engines]`** — which engines run locally, engine-specific settings - **`[cache]`** — Valkey/Redis address, password, TTL - **`[cors]`** — allowed origins and methods @@ -152,7 +152,7 @@ Copy `config.example.toml` to `config.toml` and edit. All settings can also be o |---|---| | `PORT` | Listen port (default: 8080) | | `BASE_URL` | Public URL for OpenSearch XML | -| `UPSTREAM_SEARXNG_URL` | Upstream SearXNG instance URL | +| `UPSTREAM_SEARXNG_URL` | Upstream instance URL | | `LOCAL_PORTED_ENGINES` | Comma-separated local engine list | | `HTTP_TIMEOUT` | Upstream request timeout | | `BRAVE_API_KEY` | Brave Search API key | @@ -177,7 +177,7 @@ See `config.example.toml` for the full list including rate limiting and CORS var | Reddit | Reddit JSON API | Discussions | | Bing | Bing RSS | General web | -Engines not listed in `engines.local_ported` are proxied to an upstream SearXNG instance if `upstream.url` is configured. +Engines not listed in `engines.local_ported` are proxied to an upstream metasearch instance if `upstream.url` is configured. ## Architecture diff --git a/cmd/searxng-go/main.go b/cmd/kafka/main.go similarity index 95% rename from cmd/searxng-go/main.go rename to cmd/kafka/main.go index dac6258..90c750d 100644 --- a/cmd/searxng-go/main.go +++ b/cmd/kafka/main.go @@ -53,9 +53,10 @@ func main() { } svc := search.NewService(search.ServiceConfig{ - UpstreamURL: cfg.Upstream.URL, - HTTPTimeout: cfg.HTTPTimeout(), - Cache: searchCache, + UpstreamURL: cfg.Upstream.URL, + HTTPTimeout: cfg.HTTPTimeout(), + Cache: searchCache, + EnginesConfig: cfg, }) acSvc := autocomplete.NewService(cfg.Upstream.URL, cfg.HTTPTimeout()) @@ -103,7 +104,7 @@ func main() { }, logger)(handler) addr := fmt.Sprintf(":%d", cfg.Server.Port) - logger.Info("searxng-go starting", + logger.Info("kafka starting", "addr", addr, "cache", searchCache.Enabled(), "rate_limit", cfg.RateLimit.Requests > 0, diff --git a/config.example.toml b/config.example.toml index df77184..34f60a6 100644 --- a/config.example.toml +++ b/config.example.toml @@ -15,14 +15,14 @@ http_timeout = "10s" base_url = "" [upstream] -# URL of an upstream SearXNG instance for unported engines (env: UPSTREAM_SEARXNG_URL) +# URL of an upstream metasearch instance for unported engines (env: UPSTREAM_SEARXNG_URL) # Leave empty to run without an upstream proxy. url = "" [engines] # Comma-separated list of engines to execute locally in Go (env: LOCAL_PORTED_ENGINES) -# Engines not listed here will be proxied to upstream SearXNG. -local_ported = ["wikipedia", "arxiv", "crossref", "braveapi", "qwant", "duckduckgo", "github", "reddit", "bing"] +# Engines not listed here will be proxied to the upstream instance. +local_ported = ["wikipedia", "arxiv", "crossref", "braveapi", "qwant", "duckduckgo", "github", "reddit", "bing", "google", "youtube"] [engines.brave] # Brave Search API key (env: BRAVE_API_KEY) @@ -35,6 +35,10 @@ access_token = "" category = "web-lite" results_per_page = 10 +[engines.youtube] +# YouTube Data API v3 key (env: YOUTUBE_API_KEY) +api_key = "" + [cache] # Valkey/Redis cache for search results. # Leave address empty to disable caching entirely. diff --git a/docs/superpowers/plans/2026-03-22-settings-ui.md b/docs/superpowers/plans/2026-03-22-settings-ui.md new file mode 100644 index 0000000..cf34df5 --- /dev/null +++ b/docs/superpowers/plans/2026-03-22-settings-ui.md @@ -0,0 +1,747 @@ +# Settings UI Implementation Plan + +> **For agentic workers:** REQUIRED SUB-SKILL: Use superpowers:subagent-driven-development (recommended) or superpowers:executing-plans to implement this plan task-by-task. Steps use checkbox (`- [ ]`) syntax for tracking. + +**Goal:** A preferences popover panel (top-right on desktop, bottom sheet on mobile) that lets users set theme, enabled engines, safe search, and default format. All changes auto-save to `localStorage` and apply immediately to the DOM. + +**Architecture:** Pure client-side JS + CSS added alongside existing templates. No Go changes. Settings persist via `localStorage` key `kafka_prefs`. Theme applies via `data-theme` attribute on ``. + +**Tech Stack:** Vanilla JS (no framework), existing `kafka.css` custom properties, HTMX for search. + +--- + +## File Map + +| Action | File | +|--------|------| +| Create | `internal/views/static/js/settings.js` | +| Modify | `internal/views/static/css/kafka.css` | +| Modify | `internal/views/templates/base.html` | +| Modify | `internal/views/templates/index.html` | +| Modify | `internal/views/templates/results.html` | +| Modify | `internal/views/views.go` | + +**Key insight on engine preferences:** `ParseSearchRequest` reads `engines` as a CSV form value (`r.FormValue("engines")`). The search forms in `index.html` and `results.html` will get a hidden `#engines-input` field that is kept in sync with localStorage. On submit, the engines preference is sent as a normal form field. HTMX `hx-include="this"` already includes the form element, so the hidden input is automatically included in the request. + +--- + +## Task 1: CSS — Popover, toggles, bottom sheet + +**Files:** +- Modify: `internal/views/static/css/kafka.css` + +- [ ] **Step 1: Add CSS for popover, triggers, toggles, bottom sheet** + +Append the following to `kafka.css`: + +```css +/* ============================================ + Settings Panel + ============================================ */ + +/* Header */ +.site-header { + display: flex; + align-items: center; + justify-content: space-between; + padding: 0.6rem 1rem; + background: var(--color-header-background); + border-bottom: 1px solid var(--color-header-border); +} +.site-title { + font-size: 1rem; + font-weight: 600; + color: var(--color-base-font); +} + +/* Gear trigger button */ +.settings-trigger { + background: none; + border: none; + font-size: 1.1rem; + cursor: pointer; + padding: 0.3rem 0.5rem; + border-radius: var(--radius); + color: var(--color-base-font); + opacity: 0.7; + transition: opacity 0.2s, background 0.2s; + line-height: 1; +} +.settings-trigger:hover, +.settings-trigger[aria-expanded="true"] { + opacity: 1; + background: var(--color-sidebar-background); +} + +/* Popover panel */ +.settings-popover { + position: absolute; + top: 100%; + right: 0; + width: 280px; + max-height: 420px; + overflow-y: auto; + background: var(--color-base-background); + border: 1px solid var(--color-sidebar-border); + border-radius: var(--radius); + box-shadow: 0 8px 24px rgba(0, 0, 0, 0.12); + z-index: 200; + display: none; + flex-direction: column; +} +.settings-popover[data-open="true"] { + display: flex; + animation: settings-slide-in 0.2s ease; +} +@keyframes settings-slide-in { + from { opacity: 0; transform: translateY(-8px); } + to { opacity: 1; transform: translateY(0); } +} + +.settings-popover-header { + display: flex; + justify-content: space-between; + align-items: center; + padding: 0.75rem 1rem; + border-bottom: 1px solid var(--color-sidebar-border); + font-weight: 600; + font-size: 0.9rem; + flex-shrink: 0; +} +.settings-popover-close { + background: none; + border: none; + font-size: 1.2rem; + cursor: pointer; + color: var(--color-base-font); + opacity: 0.6; + padding: 0 0.25rem; + line-height: 1; +} +.settings-popover-close:hover { opacity: 1; } + +.settings-popover-body { + padding: 0.8rem; + display: flex; + flex-direction: column; + gap: 1rem; +} + +.settings-section-title { + font-size: 0.7rem; + font-weight: 600; + text-transform: uppercase; + letter-spacing: 0.05em; + color: var(--color-suggestion); + margin-bottom: 0.5rem; +} + +/* Theme buttons */ +.theme-buttons { + display: flex; + gap: 0.4rem; +} +.theme-btn { + flex: 1; + padding: 0.35rem 0.5rem; + border: 1px solid var(--color-sidebar-border); + border-radius: var(--radius); + background: var(--color-btn-background); + color: var(--color-base-font); + cursor: pointer; + font-size: 0.75rem; + text-align: center; + transition: background 0.15s, border-color 0.15s; +} +.theme-btn:hover { background: var(--color-btn-hover); } +.theme-btn.active { + background: var(--color-link); + color: #fff; + border-color: var(--color-link); +} + +/* Engine toggles — 2-column grid */ +.engine-grid { + display: grid; + grid-template-columns: 1fr 1fr; + gap: 0.4rem; +} +.engine-toggle { + display: flex; + align-items: center; + gap: 0.4rem; + padding: 0.3rem 0.5rem; + border-radius: var(--radius); + background: var(--color-sidebar-background); + font-size: 0.78rem; + cursor: pointer; +} +.engine-toggle input[type="checkbox"] { + width: 15px; + height: 15px; + margin: 0; + cursor: pointer; + accent-color: var(--color-link); +} +.engine-toggle span { + flex: 1; + min-width: 0; + overflow: hidden; + text-overflow: ellipsis; + white-space: nowrap; +} + +/* Search defaults */ +.setting-row { + display: flex; + align-items: center; + justify-content: space-between; + gap: 0.5rem; + margin-top: 0.4rem; +} +.setting-row label { + font-size: 0.85rem; + flex: 1; +} +.setting-row select { + width: 110px; + padding: 0.3rem 0.4rem; + font-size: 0.8rem; + border: 1px solid var(--color-sidebar-border); + border-radius: var(--radius); + background: var(--color-base-background); + color: var(--color-base-font); + cursor: pointer; +} + +/* Mid-search notice */ +.settings-notice { + font-size: 0.72rem; + color: var(--color-suggestion); + margin-top: 0.3rem; + font-style: italic; +} + +/* Dark theme via data-theme attribute */ +html[data-theme="dark"] { + --color-base: #222; + --color-base-font: #dcdcdc; + --color-base-background: #2b2b2b; + --color-header-background: #333; + --color-header-border: #444; + --color-search-border: #555; + --color-search-focus: #5dade2; + --color-result-url: #8ab4f8; + --color-result-url-visited: #b39ddb; + --color-result-content: #b0b0b0; + --color-result-title: #8ab4f8; + --color-result-title-visited: #b39ddb; + --color-result-engine: #999; + --color-result-border: #3a3a3a; + --color-link: #5dade2; + --color-link-visited: #b39ddb; + --color-sidebar-background: #333; + --color-sidebar-border: #444; + --color-infobox-background: #333; + --color-infobox-border: #444; + --color-pagination-current: #5dade2; + --color-pagination-border: #444; + --color-error: #e74c3c; + --color-error-background: #3b1a1a; + --color-suggestion: #999; + --color-footer: #666; + --color-btn-background: #333; + --color-btn-border: #555; + --color-btn-hover: #444; +} + +/* Mobile: Bottom sheet + FAB trigger */ +@media (max-width: 768px) { + /* Hide desktop trigger, show FAB */ + .settings-trigger-desktop { + display: none; + } + .settings-trigger-mobile { + display: block; + } + .settings-popover { + position: fixed; + top: auto; + bottom: 0; + left: 0; + right: 0; + width: 100%; + max-height: 70vh; + border-radius: var(--radius) var(--radius) 0 0; + border-bottom: none; + } + /* FAB: fixed bottom-right button visible only on mobile */ + .settings-trigger-mobile { + display: block; + position: fixed; + bottom: 1.5rem; + right: 1.5rem; + width: 48px; + height: 48px; + border-radius: 50%; + background: var(--color-link); + color: #fff; + border: none; + box-shadow: 0 4px 12px rgba(0,0,0,0.2); + font-size: 1.2rem; + z-index: 199; + opacity: 1; + } +} +``` + +Note: The existing `:root` and `@media (prefers-color-scheme: dark)` blocks provide the "system" theme. `html[data-theme="dark"]` overrides only apply when the user explicitly picks dark mode. When `theme === 'system'`, the `data-theme` attribute is removed and the browser's `prefers-color-scheme` media query kicks in via the existing CSS. + +- [ ] **Step 2: Verify existing tests still pass** + +Run: `go test ./...` +Expected: all pass + +- [ ] **Step 3: Commit** + +```bash +git add internal/views/static/css/kafka.css +git commit -m "feat(settings): add popover, toggle, and bottom-sheet CSS" +``` + +--- + +## Task 2: JS — Settings logic + +**Files:** +- Create: `internal/views/static/js/settings.js` + +- [ ] **Step 1: Write the settings JS module** + +Create `internal/views/static/js/settings.js`: + +```javascript +'use strict'; + +var ALL_ENGINES = [ + 'wikipedia', 'arxiv', 'crossref', 'braveapi', + 'qwant', 'duckduckgo', 'github', 'reddit', 'bing' +]; + +var DEFAULT_PREFS = { + theme: 'system', + engines: ALL_ENGINES.slice(), + safeSearch: 'moderate', + format: 'html' +}; + +var STORAGE_KEY = 'kafka_prefs'; + +// ── Persistence ────────────────────────────────────────────────────────────── + +function loadPrefs() { + try { + var raw = localStorage.getItem(STORAGE_KEY); + if (!raw) return { theme: DEFAULT_PREFS.theme, engines: DEFAULT_PREFS.engines.slice(), safeSearch: DEFAULT_PREFS.safeSearch, format: DEFAULT_PREFS.format }; + var saved = JSON.parse(raw); + return { theme: saved.theme || DEFAULT_PREFS.theme, engines: saved.engines || DEFAULT_PREFS.engines.slice(), safeSearch: saved.safeSearch || DEFAULT_PREFS.safeSearch, format: saved.format || DEFAULT_PREFS.format }; + } catch (e) { + return { theme: DEFAULT_PREFS.theme, engines: DEFAULT_PREFS.engines.slice(), safeSearch: DEFAULT_PREFS.safeSearch, format: DEFAULT_PREFS.format }; + } +} + +function savePrefs(prefs) { + try { + localStorage.setItem(STORAGE_KEY, JSON.stringify({ theme: prefs.theme, engines: prefs.engines, safeSearch: prefs.safeSearch, format: prefs.format })); + } catch (e) { /* quota or private mode */ } +} + +// ── Theme application ──────────────────────────────────────────────────────── + +function applyTheme(theme) { + if (theme === 'system') { + document.documentElement.removeAttribute('data-theme'); + } else { + document.documentElement.setAttribute('data-theme', theme); + } +} + +// ── Engine input sync ───────────────────────────────────────────────────────── + +function syncEngineInput(prefs) { + var input = document.getElementById('engines-input'); + if (input) input.value = prefs.engines.join(','); +} + +// ── Panel open / close ──────────────────────────────────────────────────────── + +function closePanel() { + var panel = document.getElementById('settings-popover'); + var trigger = document.getElementById('settings-trigger'); + if (!panel) return; + panel.setAttribute('data-open', 'false'); + if (trigger) trigger.setAttribute('aria-expanded', 'false'); + if (trigger) trigger.focus(); +} + +function openPanel() { + var panel = document.getElementById('settings-popover'); + var trigger = document.getElementById('settings-trigger'); + if (!panel) return; + panel.setAttribute('data-open', 'true'); + if (trigger) trigger.setAttribute('aria-expanded', 'true'); + var focusable = panel.querySelector('button, input, select'); + if (focusable) focusable.focus(); +} + +// ── Escape key ─────────────────────────────────────────────────────────────── + +document.addEventListener('keydown', function(e) { + if (e.key !== 'Escape') return; + var panel = document.getElementById('settings-popover'); + if (!panel || panel.getAttribute('data-open') !== 'true') return; + closePanel(); +}); + +// ── Click outside ───────────────────────────────────────────────────────────── + +document.addEventListener('click', function(e) { + var panel = document.getElementById('settings-popover'); + var trigger = document.getElementById('settings-trigger'); + if (!panel || panel.getAttribute('data-open') !== 'true') return; + if (!panel.contains(e.target) && (!trigger || !trigger.contains(e.target))) { + closePanel(); + } +}); + +// ── Focus trap ──────────────────────────────────────────────────────────────── + +document.addEventListener('keydown', function(e) { + if (e.key !== 'Tab') return; + var panel = document.getElementById('settings-popover'); + if (!panel || panel.getAttribute('data-open') !== 'true') return; + var focusable = Array.prototype.slice.call(panel.querySelectorAll('button, input, select, [tabindex]:not([tabindex="-1"])')); + if (!focusable.length) return; + var first = focusable[0]; + var last = focusable[focusable.length - 1]; + if (e.shiftKey) { + if (document.activeElement === first) { e.preventDefault(); last.focus(); } + } else { + if (document.activeElement === last) { e.preventDefault(); first.focus(); } + } +}); + +// ── Render ──────────────────────────────────────────────────────────────────── + +function escapeHtml(str) { + return String(str).replace(/&/g, '&').replace(//g, '>'); +} + +function renderPanel(prefs) { + var panel = document.getElementById('settings-popover'); + if (!panel) return; + var body = panel.querySelector('.settings-popover-body'); + if (!body) return; + + var themeBtns = ''; + ['light', 'dark', 'system'].forEach(function(t) { + var icons = { light: '\u2600', dark: '\u263D', system: '\u2318' }; + var labels = { light: 'Light', dark: 'Dark', system: 'System' }; + var active = prefs.theme === t ? ' active' : ''; + themeBtns += ''; + }); + + var engineToggles = ''; + ALL_ENGINES.forEach(function(name) { + var checked = prefs.engines.indexOf(name) !== -1 ? ' checked' : ''; + engineToggles += ''; + }); + + var ssOptions = [ + { val: 'moderate', label: 'Moderate' }, + { val: 'strict', label: 'Strict' }, + { val: 'off', label: 'Off' } + ]; + var fmtOptions = [ + { val: 'html', label: 'HTML' }, + { val: 'json', label: 'JSON' }, + { val: 'csv', label: 'CSV' }, + { val: 'rss', label: 'RSS' } + ]; + var ssOptionsHtml = ''; + var fmtOptionsHtml = ''; + ssOptions.forEach(function(o) { + var sel = prefs.safeSearch === o.val ? ' selected' : ''; + ssOptionsHtml += ''; + }); + fmtOptions.forEach(function(o) { + var sel = prefs.format === o.val ? ' selected' : ''; + fmtOptionsHtml += ''; + }); + + body.innerHTML = + '
Engine changes apply to your next search.
' + + 'No results found.
diff --git a/internal/views/templates/video_item.html b/internal/views/templates/video_item.html new file mode 100644 index 0000000..0ca0109 --- /dev/null +++ b/internal/views/templates/video_item.html @@ -0,0 +1,22 @@ +{{define "video_item"}} +{{.Content}}
+ {{end}} + {{if .Engine}} +