We are building a custom search backend so detail the mapping between keywords and document IDs

Question

Anonymous · Accepted Answer

For a custom search backend, the core mapping is usually an inverted index:

keyword → document IDs

Example:

"golang"    → [doc_1, doc_7, doc_12]
"redis"     → [doc_2, doc_7, doc_9]
"kafka"     → [doc_3, doc_7]

RECOMMENDED DATA MODEL

Keyword / TokenDocument IDsFrequency / Positionsgolangdoc_1, doc_7,
doc_12optionalredisdoc_2, doc_7, doc_9optionalkafkadoc_3,
doc_7optionalmicroservicedoc_1, doc_3, doc_7optional

For a production-quality search engine, I'd recommend storing a posting list
rather than only a set of document IDs:

term
 └── posting list
      ├── docID
      ├── term frequency
      └── positions

For example:

redis
 ├── doc_2: tf=3, positions=[4, 18, 31]
 ├── doc_7: tf=2, positions=[7, 22]
 └── doc_9: tf=1, positions=[15]

This lets you later implement:

* 
   Boolean search: redis AND kafka

* 
   OR search: redis OR kafka

* 
   Phrase search: "redis cache"

* 
   TF-IDF/BM25 ranking

* 
   proximity ranking

* 
   autocomplete/prefix search

HIGH-LEVEL ARCHITECTURE

Documents
   ↓
Tokenizer
   ↓
Normalizer
   ↓
Tokens
   ↓
Inverted Index
   ↓
┌───────────────┐
│ keyword       │
│   ↓           │
│ posting list  │
│   ↓           │
│ document IDs  │
└───────────────┘

The important distinction is:

Document → keywords

is useful for indexing a document, but

Keyword → documents

is what makes search fast.

We are building a custom search backend so detail the mapping between keywords and document IDs

Practice this question with AI

Go Premium

Community Answers

Unlock Full Access

Practice More Questions