penal-colony/kafka
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
12 commits 4 branches 0 tags 26 MiB
Find a file
Franz Kafka a8ab29b23a fix: fix DDG and Bing parsers — verified with live tests 
DuckDuckGo:
- Fixed parser to handle single-quoted class attributes (class='result-link')
- Decode DDG tracking URLs (uddg= parameter) to extract real URLs
- Match snippet extraction to actual DDG Lite HTML structure (</td> terminator)

Bing:
- Switched from HTML scraping (blocked by JS detection) to RSS endpoint
  (?format=rss) which returns parseable XML
- Added JSON API response parsing as fallback
- Returns graceful unresponsive_engines entry when blocked

Live test results:
- DuckDuckGo: 9 results ✅
- GitHub: 10 results (14,768 total) ✅
- Bing: 10 results via RSS ✅
- Reddit: skipped (403 from sandbox, needs browser-like context)
2026-03-21 16:57:02 +00:00
cmd/searxng-go feat: HTMX + Go Templates HTML frontend 2026-03-21 16:10:42 +00:00
internal fix: fix DDG and Bing parsers — verified with live tests 2026-03-21 16:57:02 +00:00
.gitignore feat: add multi-stage Dockerfile and docker-compose.yml 2026-03-21 15:46:33 +00:00
config.example.toml feat: add DuckDuckGo, GitHub, Reddit, and Bing engines 2026-03-21 16:52:11 +00:00
docker-compose.yml feat: add multi-stage Dockerfile and docker-compose.yml 2026-03-21 15:46:33 +00:00
Dockerfile feat: add multi-stage Dockerfile and docker-compose.yml 2026-03-21 15:46:33 +00:00
flake.lock feat: build Go-based SearXNG-compatible search service 2026-03-20 20:34:08 +01:00
flake.nix feat: build Go-based SearXNG-compatible search service 2026-03-20 20:34:08 +01:00
go.mod feat: Valkey cache for search results 2026-03-21 15:43:47 +00:00
go.sum feat: Valkey cache for search results 2026-03-21 15:43:47 +00:00
README.md feat: build Go-based SearXNG-compatible search service 2026-03-20 20:34:08 +01:00

README.md

gosearch (SearXNG rewrite in Go)

This repository contains a standalone Go HTTP service that implements a SearXNG-compatible API-first /search endpoint and proxies unported engines to an upstream SearXNG instance.

Endpoints

  • GET /healthz -> OK
  • GET|POST /search
    • Required form/body parameter: q
    • Optional: format (json | csv | rss; default: json)

Supported format=...

  • json: SearXNG-style JSON response (query, number_of_results, results, answers, corrections, infoboxes, suggestions, unresponsive_engines)
  • csv: CSV with header title,url,content,host,engine,score,type
  • rss: RSS 2.0 feed based on the opensearch_response_rss.xml template fields

Request parameters

The server accepts SearXNG form parameters (both GET query string and POST form-encoded):

  • q (required): search query
  • format (optional): json/csv/rss
  • pageno (optional, default 1): positive integer
  • safesearch (optional, default 0): integer 0..2
  • time_range (optional): day|week|month|year (or omitted/None)
  • timeout_limit (optional): float, seconds (or omitted/None)
  • language (optional, default auto): auto or a BCP-47-ish language code
  • engines (optional): comma-separated engine names (e.g. wikipedia,arxiv)
  • categories / category_<name> (optional): used for selecting the initial ported subset
  • engine_data-<engine>-<key>=<value> (optional): per-engine custom parameters

Environment variables

  • PORT (optional, default 8080)
  • UPSTREAM_SEARXNG_URL (optional for now, but required if you expect unported engines)
    • When set, unported engines are proxied to ${UPSTREAM_SEARXNG_URL}/search with format=json.
  • LOCAL_PORTED_ENGINES (optional, default wikipedia,arxiv,crossref,braveapi,qwant)
    • Controls which engine names are executed locally (Go-native adapters).
  • HTTP_TIMEOUT (optional, default 10s)
    • Timeout for both local engine API calls and upstream proxy calls.
  • Brave Search API:
    • BRAVE_API_KEY (optional): enables the braveapi engine when set
    • BRAVE_ACCESS_TOKEN (optional): if set, requests must include a token (header Authorization: Bearer <token>, X-Search-Token, X-Brave-Access-Token, or form field token)

Ported vs proxied strategy

  1. The service plans which engines should run locally vs upstream using LOCAL_PORTED_ENGINES.
  2. It executes local ported engines using Go-native adapters:
    • wikipedia, arxiv, crossref
  3. Any remaining requested engines are proxied to upstream SearXNG (format=json).
  4. Responses are merged:
    • results are de-duplicated by engine|title|url
    • suggestions/corrections are treated as sets
    • other arrays are concatenated

Running with Nix

This repo uses flake.nix to provide the Go toolchain.

nix develop
go test ./...
go run ./cmd/searxng-go

Example:

export UPSTREAM_SEARXNG_URL="http://127.0.0.1:8888"
export PORT="8080"
nix develop -c go run ./cmd/searxng-go