penal-colony/kafka
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
6 commits 4 branches 0 tags 26 MiB
Find a file
Franz Kafka 385a7acab7 feat: concurrent engine execution with graceful degradation 
- Run all local engines in parallel using goroutines + sync.WaitGroup
- Individual engine failures are captured as unresponsive_engines entries
  instead of aborting the entire search request
- Context cancellation is respected: cancelled engines report as unresponsive
- Upstream proxy failure is also gracefully handled (single unresponsive entry)
- Extract unresponsiveResponse() and emptyResponse() helpers for consistency
- Add comprehensive tests:
  - ConcurrentEngines: verifies parallelism (2x100ms engines complete in ~100ms)
  - GracefulDegradation: one engine fails, one succeeds, both represented
  - AllEnginesFail: no error returned, all engines in unresponsive_engines
  - ContextCancellation: engine respects context timeout, reports unresponsive
2026-03-21 15:39:00 +00:00
cmd/searxng-go feat: migrate from env vars to config.toml 2026-03-21 14:56:00 +00:00
internal feat: concurrent engine execution with graceful degradation 2026-03-21 15:39:00 +00:00
.gitignore feat: migrate from env vars to config.toml 2026-03-21 14:56:00 +00:00
config.example.toml feat: migrate from env vars to config.toml 2026-03-21 14:56:00 +00:00
flake.lock feat: build Go-based SearXNG-compatible search service 2026-03-20 20:34:08 +01:00
flake.nix feat: build Go-based SearXNG-compatible search service 2026-03-20 20:34:08 +01:00
go.mod feat: concurrent engine execution with graceful degradation 2026-03-21 15:39:00 +00:00
go.sum feat: concurrent engine execution with graceful degradation 2026-03-21 15:39:00 +00:00
README.md feat: build Go-based SearXNG-compatible search service 2026-03-20 20:34:08 +01:00
searxng-go feat: concurrent engine execution with graceful degradation 2026-03-21 15:39:00 +00:00

README.md

gosearch (SearXNG rewrite in Go)

This repository contains a standalone Go HTTP service that implements a SearXNG-compatible API-first /search endpoint and proxies unported engines to an upstream SearXNG instance.

Endpoints

  • GET /healthz -> OK
  • GET|POST /search
    • Required form/body parameter: q
    • Optional: format (json | csv | rss; default: json)

Supported format=...

  • json: SearXNG-style JSON response (query, number_of_results, results, answers, corrections, infoboxes, suggestions, unresponsive_engines)
  • csv: CSV with header title,url,content,host,engine,score,type
  • rss: RSS 2.0 feed based on the opensearch_response_rss.xml template fields

Request parameters

The server accepts SearXNG form parameters (both GET query string and POST form-encoded):

  • q (required): search query
  • format (optional): json/csv/rss
  • pageno (optional, default 1): positive integer
  • safesearch (optional, default 0): integer 0..2
  • time_range (optional): day|week|month|year (or omitted/None)
  • timeout_limit (optional): float, seconds (or omitted/None)
  • language (optional, default auto): auto or a BCP-47-ish language code
  • engines (optional): comma-separated engine names (e.g. wikipedia,arxiv)
  • categories / category_<name> (optional): used for selecting the initial ported subset
  • engine_data-<engine>-<key>=<value> (optional): per-engine custom parameters

Environment variables

  • PORT (optional, default 8080)
  • UPSTREAM_SEARXNG_URL (optional for now, but required if you expect unported engines)
    • When set, unported engines are proxied to ${UPSTREAM_SEARXNG_URL}/search with format=json.
  • LOCAL_PORTED_ENGINES (optional, default wikipedia,arxiv,crossref,braveapi,qwant)
    • Controls which engine names are executed locally (Go-native adapters).
  • HTTP_TIMEOUT (optional, default 10s)
    • Timeout for both local engine API calls and upstream proxy calls.
  • Brave Search API:
    • BRAVE_API_KEY (optional): enables the braveapi engine when set
    • BRAVE_ACCESS_TOKEN (optional): if set, requests must include a token (header Authorization: Bearer <token>, X-Search-Token, X-Brave-Access-Token, or form field token)

Ported vs proxied strategy

  1. The service plans which engines should run locally vs upstream using LOCAL_PORTED_ENGINES.
  2. It executes local ported engines using Go-native adapters:
    • wikipedia, arxiv, crossref
  3. Any remaining requested engines are proxied to upstream SearXNG (format=json).
  4. Responses are merged:
    • results are de-duplicated by engine|title|url
    • suggestions/corrections are treated as sets
    • other arrays are concatenated

Running with Nix

This repo uses flake.nix to provide the Go toolchain.

nix develop
go test ./...
go run ./cmd/searxng-go

Example:

export UPSTREAM_SEARXNG_URL="http://127.0.0.1:8888"
export PORT="8080"
nix develop -c go run ./cmd/searxng-go