penal-colony/kafka
2
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions
No description
11 commits 4 branches 0 tags 26 MiB
Find a file
Franz Kafka df8fe9474b feat: add DuckDuckGo, GitHub, Reddit, and Bing engines 
- DuckDuckGo: scrapes Lite HTML endpoint for results
  - Language-aware region mapping (de→de-de, ja→jp-jp, etc.)
  - HTML parser extracts result links and snippets from DDG Lite markup
  - Shared html_helpers.go with extractAttr, stripHTML, htmlUnescape

- GitHub: uses public Search API (repos, sorted by stars)
  - No auth required (10 req/min unauthenticated)
  - Shows stars, language, topics, last updated date
  - Paginated via GitHub's page parameter

- Reddit: uses public JSON search API
  - Respects safesearch (skips over_18 posts)
  - Shows subreddit, score, comment count
  - Links self-posts to the thread URL

- Bing: scrapes web search HTML (b_algo containers)
  - Extracts titles, URLs, and snippets from Bing's result markup
  - Handles Bing's tracking URL encoding

- Updated factory, config defaults, and config.example.toml
- Full test suite: unit tests for all engines, HTML parsing tests,
  region mapping tests, live request tests (skipped in short mode)

9 engines total: wikipedia, arxiv, crossref, braveapi, qwant,
duckduckgo, github, reddit, bing
2026-03-21 16:52:11 +00:00
cmd/searxng-go feat: HTMX + Go Templates HTML frontend 2026-03-21 16:10:42 +00:00
internal feat: add DuckDuckGo, GitHub, Reddit, and Bing engines 2026-03-21 16:52:11 +00:00
.gitignore feat: add multi-stage Dockerfile and docker-compose.yml 2026-03-21 15:46:33 +00:00
config.example.toml feat: add DuckDuckGo, GitHub, Reddit, and Bing engines 2026-03-21 16:52:11 +00:00
docker-compose.yml feat: add multi-stage Dockerfile and docker-compose.yml 2026-03-21 15:46:33 +00:00
Dockerfile feat: add multi-stage Dockerfile and docker-compose.yml 2026-03-21 15:46:33 +00:00
flake.lock feat: build Go-based SearXNG-compatible search service 2026-03-20 20:34:08 +01:00
flake.nix feat: build Go-based SearXNG-compatible search service 2026-03-20 20:34:08 +01:00
go.mod feat: Valkey cache for search results 2026-03-21 15:43:47 +00:00
go.sum feat: Valkey cache for search results 2026-03-21 15:43:47 +00:00
README.md feat: build Go-based SearXNG-compatible search service 2026-03-20 20:34:08 +01:00

README.md

gosearch (SearXNG rewrite in Go)

This repository contains a standalone Go HTTP service that implements a SearXNG-compatible API-first /search endpoint and proxies unported engines to an upstream SearXNG instance.

Endpoints

  • GET /healthz -> OK
  • GET|POST /search
    • Required form/body parameter: q
    • Optional: format (json | csv | rss; default: json)

Supported format=...

  • json: SearXNG-style JSON response (query, number_of_results, results, answers, corrections, infoboxes, suggestions, unresponsive_engines)
  • csv: CSV with header title,url,content,host,engine,score,type
  • rss: RSS 2.0 feed based on the opensearch_response_rss.xml template fields

Request parameters

The server accepts SearXNG form parameters (both GET query string and POST form-encoded):

  • q (required): search query
  • format (optional): json/csv/rss
  • pageno (optional, default 1): positive integer
  • safesearch (optional, default 0): integer 0..2
  • time_range (optional): day|week|month|year (or omitted/None)
  • timeout_limit (optional): float, seconds (or omitted/None)
  • language (optional, default auto): auto or a BCP-47-ish language code
  • engines (optional): comma-separated engine names (e.g. wikipedia,arxiv)
  • categories / category_<name> (optional): used for selecting the initial ported subset
  • engine_data-<engine>-<key>=<value> (optional): per-engine custom parameters

Environment variables

  • PORT (optional, default 8080)
  • UPSTREAM_SEARXNG_URL (optional for now, but required if you expect unported engines)
    • When set, unported engines are proxied to ${UPSTREAM_SEARXNG_URL}/search with format=json.
  • LOCAL_PORTED_ENGINES (optional, default wikipedia,arxiv,crossref,braveapi,qwant)
    • Controls which engine names are executed locally (Go-native adapters).
  • HTTP_TIMEOUT (optional, default 10s)
    • Timeout for both local engine API calls and upstream proxy calls.
  • Brave Search API:
    • BRAVE_API_KEY (optional): enables the braveapi engine when set
    • BRAVE_ACCESS_TOKEN (optional): if set, requests must include a token (header Authorization: Bearer <token>, X-Search-Token, X-Brave-Access-Token, or form field token)

Ported vs proxied strategy

  1. The service plans which engines should run locally vs upstream using LOCAL_PORTED_ENGINES.
  2. It executes local ported engines using Go-native adapters:
    • wikipedia, arxiv, crossref
  3. Any remaining requested engines are proxied to upstream SearXNG (format=json).
  4. Responses are merged:
    • results are de-duplicated by engine|title|url
    • suggestions/corrections are treated as sets
    • other arrays are concatenated

Running with Nix

This repo uses flake.nix to provide the Go toolchain.

nix develop
go test ./...
go run ./cmd/searxng-go

Example:

export UPSTREAM_SEARXNG_URL="http://127.0.0.1:8888"
export PORT="8080"
nix develop -c go run ./cmd/searxng-go