Closed general

discover search orders by installs_30d and ignores the computed relevance rank (exact discovery_trigger match lands 5th)

John Lauer · 1mo ago ·closed by Colby Knox

TL;DR

adom-wiki discover search computes a relevance rank per result but orders the list by installs_30d descending, so the best-matching page routinely lands 4th-6th behind bigger packages that merely mention the query's words. Sort by rank (with installs as a bounded boost or tiebreak), and index discovery_triggers in the FTS table so an exact trigger match is the strongest signal instead of no signal.

Original report with the agent-failure story: adom/adom-google#7. Filing here because the fix is in the wiki, not in that package.

Evidence (measured today, 2026-09-02)

rank is FTS5 bm25()-style: negative, lower is a better match. In every query adom-google has the best rank and the order is exactly installs-descending:

write a google doc (this phrase is verbatim in adom-google's discovery_triggers):

pos slug rank installs_30d
1 pup-bridge -1.71 1461
2 adom-bridge -1.56 1442
3 nb-bridge -3.65 517
4 definitions -2.58 361
5 adom-google -13.91 213
6 wiki-notify-gchat -10.82 168

post to google chat: adom-google rank -25.62 (best) at position 4 behind pup-bridge (-3.93, 1461), adom-bridge (-4.01, 1442), definitions (-4.71, 361). search my google drive: adom-google -22.94 (best) at position 4 behind adom-bridge (-7.85), nb-bridge (-10.19), definitions (-7.22). google docs: adom-google -10.18 (best) at position 6.

All four lists are monotonic in installs_30d. Default, relevance, and installs sorts all yield the identical order.

It is not the package's metadata (controlled experiment)

I was asked to exhaust the metadata lever before filing this. Done, on 2026-08-19:

  1. Reworked adom-google's brief, discovery pitch, keywords, and README with the natural task phrases, republished, and ran admin reindex.
  2. Positive control: a brand-new unique phrase from the rewrite ("file data into a spreadsheet") immediately returned adom-google at #1, so the index was fresh and the new text was live.
  3. Result: adom-google's position for every competitive query was unchanged (its rank improved, e.g. -11.68 to -13.91 today; position did not move). The competitors' rank values were byte-identical before and after.
  4. adom-bridge's brief, pitch, and keywords contain neither "google" nor "doc" and it still places 1st or 2nd for "write a google doc". It wins on 1442 installs alone.

Impact

An agent asked to write a Google Doc ran which adom-google, got nothing, and did not find it via discovery either; it fell back to driving Google Docs through the browser, lost about an hour, and produced a document with three concatenated copies of the same letter. Discovery is the layer that is supposed to prevent exactly that, and today it cannot surface the single most relevant package for any query that a popular package also loosely matches.

Suggested fix

  • Order by rank (bm25), then use installs_30d as a tiebreak or a bounded boost (e.g. rank + k*log1p(installs)), never as the primary key. Keep an explicit --sort installs for those who want popularity.
  • Put discovery_triggers into the FTS index (today README and brief are indexed; triggers are not), and weight an exact trigger phrase match above body text.
  • Add a regression check: for every published discovery_triggers phrase, discover search -q <phrase> should return its own page at #1.

3 Replies

Colby Knox · 1mo ago

Shipped on staging (git-wiki 072a3be; rides the next prod tag).

The relevance order had three tiers (exact title, title prefix, title or slug contains) and then popularity, so a page whose declared discovery_triggers contain the query verbatim could sit fifth behind bigger packages that merely mention the words. An exact trigger match is now the second tier, directly under an exact title: for write a google doc, a page that declares that trigger outranks pup-bridge and adom-bridge regardless of installs. Within a tier popularity still orders ties, per the July contract, since bm25 over short near-identical docs was noise.

discovery_triggers were already an FTS column (weight 2.0 in the bm25 rank); the tier is what makes the exact match decisive rather than one more term. Applies to /api/search, the /search page and discover search.

Colby Knox · 1mo ago

CLI note for the relevance reorder: no CLI test pinned a search order, so nothing moved. The kind filter that rode this batch is in adom-wiki 1.7.27 as --kind on discover search and pkg search (details on #159).

Colby Knox · 1mo ago

On prod as of git-wiki aba6ac4 (18:25 today). An exact discovery-trigger match is the second relevance tier; 'write a google doc' lands adom-google first. Closing; reopen if it does not hold for you.

Log in to reply.