Bibhash Mitra

Bibhash Mitra

Senior Data Scientist & AI/ML Expert with 6+ years of experience in Generative AI, NLP, Computer Vision, and Quantum Computing.

Search for a Bengali book by typing its name in English. Or in Bengali. Or misspell it a little. It still works.

Bibyutatsu Ebooks is an open-access digital library of 1,439 Bengali literary works with instant client-side search, cover galleries, and direct multi-format downloads. Runs entirely on GitHub Pages.1

Why this exists

Bengali digital books are scattered everywhere: forums, Telegram groups, personal drives, community sites. Finding a specific title means hopping between three or four sources, guessing file formats, and hoping somebody tagged the metadata correctly.

The reading experience once you have the file is fine. EPUB readers and Kindle have been good for years. The bottleneck is discovery. There is no single place where you can type “sonar kella” in English, find Satyajit Ray’s Feluda novel, pick a format, and download it. That is what this project is for.

The search problem: two scripts, many spellings

Bengali literature has a naming problem for English speakers. শরদিন্দু বন্দ্যোপাধ্যায় becomes “Sharadindu Bandyopadhyay” or “Sharadindu Banerjee” or “Saradindu Bandopadhyay” depending on who romanized it. ব্যোমকেশ is “Byomkesh” to some readers and “Bomkesh” to others. A plain text search breaks on the first query.

You need a system that knows “feluda” maps to “ফেলুদা”, that “humayan” is a misspelling of “humayun”, and that “satyajit roy” and “সত্যজিৎ রায়” are the same person. The solution is to compute all plausible romanized spellings at index time and store them in a pre-computed search_text field per book. The browser then does plain string matching, nothing fancier.

How the transliteration engine works

transliteration.py converts Bengali Unicode text into multiple plausible romanized spellings. It maps every consonant, vowel, vowel sign, and conjunct cluster to its romanized equivalents:

Bengali Romanized variants
ক k
ছ chh, ch
ফ ph, f
ভ bh, v
ক্ষ kkh, ksh, x
জ্ঞ ggo, gyan, jn

The transliterator walks through each Unicode character, handles conjunct sequences (like ক্ষ or জ্ঞ) with priority over single characters, and generates a cartesian product of all plausible spellings. A single Bengali word can produce up to 12 romanized variants.

def transliterate_word(word: str) -> list[str]:
    """Returns a list of plausible romanized spellings."""
    normalized = unicodedata.normalize('NFC', word)
    variants = [""]

    while i < length:
        # Try conjuncts first (2-3 char sequences)
        for c_len in (3, 2):
            sub = normalized[i:i + c_len]
            if sub in CONJUNCTS:
                opts = CONJUNCTS[sub]
                variants = [v + o for v in variants for o in opts]
                # ... handle following vowel signs
                break

        # Then individual consonants with inherent vowel logic
        if ch in CONSONANTS:
            # If no following vowel sign: add both 'o' and 'a' endings
            new_variants.append(v + co + 'o')
            new_variants.append(v + co + 'a')

The critical design decision is how to handle the inherent vowel. In Bengali, a consonant without an explicit vowel sign carries an inherent “অ” sound, which romanizes as either ‘o’ or ‘a’ depending on dialect and position. Generating both variants is why “bomkesh” finds books tagged with “byomkesh”.

The catalog builder pipeline

build_catalog.py transforms raw book files into a normalized catalog.json that the frontend loads wholesale. The output is 3.1MB and runs as a batch process whenever books are added.

What it does: parse Calibre OPF files or internal EPUB metadata for title, creator, description, and publisher. Map Bengali author names to English equivalents using a hand-curated database of 270+ authors in author_mapping.py (each entry carries the Bengali name, English name, known aliases, and genre tags). Automatically tag books in popular series (Feluda, Byomkesh, Shonku, Kakababu, Himu, Masud Rana, Tin Goyenda) by matching keywords against titles. Run every title and author through the transliteration engine and concatenate all romanized variants into search_text. Convert cover images to WebP thumbnails. The 1,430+ covers total about 28MB.

app.js fetches catalog.json on page load and runs all search and filtering in the browser. No server involved. The matching algorithm has three tiers, tried in order:

  1. Exact substring: the query token appears inside a search index token.
  2. Prefix: the query token is a prefix of a search index token.
  3. Levenshtein fuzzy: if the query token is at least 4 characters long and within edit distance 2 of a target token, it matches.
function matchesToken(qTok, targetTokens, maxDist = 2) {
  for (let i = 0; i < targetTokens.length; i++) {
    const t = targetTokens[i];
    if (t.includes(qTok) || t.startsWith(qTok)) return true;
    if (qTok.length >= 4 && Math.abs(qTok.length - t.length) <= maxDist) {
      if (levenshtein(qTok, t) <= maxDist) return true;
    }
  }
  return false;
}

The length guard (Math.abs(qTok.length - t.length) <= maxDist) filters out obviously non-matching pairs before running the O(n * m) distance calculation. Multi-word queries use AND logic: every token in the query must independently match. So “humayun ahmed” returns only Humayun Ahmed’s books, not every book that mentions either word.

Five themes and a particle canvas

The UI has five themes: Dark, Light, Batman, Cyberpunk, and Ocean. Each one applies a coordinated color palette across the interface and the Three.js particle canvas running in the background.

The canvas renders 75 floating nodes (40 on mobile) connected by proximity-based edges. Theme switches propagate to the particle colors in real time:

const THEME_COLORS = {
  dark:      { n1: 0x00ccff, n2: 0x8855ff, p: 0xffffff },
  light:     { n1: 0x0055cc, n2: 0x7722cc, p: 0x0055cc },
  batman:    { n1: 0xFFE919, n2: 0xff4444, p: 0xFFE919 },
  cyberpunk: { n1: 0xff0080, n2: 0x00ffcc, p: 0xff0080 },
  ocean:     { n1: 0x00e5b0, n2: 0x0099ff, p: 0x00e5b0 },
};

The Batman theme uses the Bangers font with yellow and red particles. Completely unnecessary. Worth it.

Bibyutatsu BookStore homepage showing the dark theme with glassmorphic book cards and particle canvas

Download distribution: GitHub Releases as a CDN

Hosting 1,439 books in the Git repository itself is not practical. That much binary content would blow past GitHub’s file size limits and make every git clone painful.

The workaround: sync_releases.py uploads book files to tagged GitHub Releases in batches of 75 files per tag (e.g., v1.0-batch-01, v1.0-batch-02). Each file gets a permanent CDN URL that catalog.json stores per format. The tool is idempotent: it checks what is already uploaded and skips those assets. Running it twice on the same catalog uploads nothing. That matters when you are pushing 20 new books into an existing catalog of 1,400+.

python3 tools/sync_releases.py --catalog ./catalog.json

Each book can have up to four format options: .epub, .kfx (Kindle), .mobi, and .pdf. The frontend renders download buttons per available format with human-readable file sizes from the catalog.

Two ingestion toolchains: contemporary and classical

The 1,439 books come from two sources with very different technical characteristics.

KindleBangla (tools/kindlebangla/)

The bulk of the contemporary collection came from KindleBangla, a community project preserving popular Bengali fiction. The toolchain for it: downloader.py crawls book detail pages, extracts download links, and saves files with metadata; verify_downloads.py extracts nested .rar archives and validates file integrity. The book_details_links.json and books_metadata.json files in that directory are the sitemap and raw metadata dump from the initial scrape.

BongBoi (tools/bongboi/)

200+ classical, out-of-copyright Bengali works came from the eedeidk/bongboi archive. Public-domain EPUBs are a different problem from modern downloads.

Many of them have no dedicated cover image. sync_bongboi.py inspects each EPUB archive, extracts embedded illustration pages if they exist, and otherwise renders a typographic title cover using PIL. Historical author names also come in archaic phonetic forms. The pipeline normalizes names for Ishwar Chandra Vidyasagar, Abanindranath Tagore, Michael Madhusudan Dutt, Begum Rokeya, Rakhaldas Bandyopadhyay, and others into canonical author entries that the transliteration and alias systems can then handle uniformly.

python3 tools/bongboi/sync_bongboi.py

53 search benchmarks at 100% recall

verify_catalog.py runs 53 benchmark queries using the same fuzzy matching logic the frontend uses. Every query must return at least the minimum expected count, or the entire suite fails.

Category Example queries Min expected
Iconic characters feluda, byomkesh, shonku, himu, kakababu 3-5 books each
Phonetic variants bomkesh, humayan, satyajit roy Same as canonical
Bengali authors sharadindu bandyopadhyay, sunil ganguly 5+ books
English aliases sharadindu banerjee, atin banerjee Same as Bengali
Classical figures vidyasagar, abanindranath, begum rokeya, madhusudan dutt 1-8 books
World literature agatha christie, jules verne, asimov 1-5 books
Bengali script ফেলুদা, ব্যোমকেশ, হুমায়ূন আহমেদ, বিদ্যাসাগর Same as English

The cross-script cases are the most useful ones to look at. “vidyasagar” and “ঈশ্বরচন্দ্র বিদ্যাসাগর” must return the same classical volumes. If the transliteration engine and the author aliasing pipeline are not aligned, one of those queries fails while the other passes. The test catches that drift automatically.

python3 tests/verify_catalog.py ./catalog.json
# ✓ Validated stats: 1439 books, 271 authors.
# ✓ Verified 100% of 1439 books conform to normalized schema.
# Benchmark Results: 53/53 passed (100.0%)
# ✓ All search and transliteration benchmarks PASSED with 100% recall!

What a catalog entry looks like

{
  "id": "sonar-kella-satyajit-ray",
  "title": "সোনার কেল্লা",
  "title_en": "Sonar Kella",
  "author": "সত্যজিৎ রায়",
  "author_en": "Satyajit Ray",
  "genres": ["Mystery", "Thriller"],
  "series": "Feluda",
  "cover": "assets/covers/sonar-kella-satyajit-ray.webp",
  "formats": {
    "epub": {
      "filename": "Sonar Kella - Satyajit Ray.epub",
      "size_bytes": 245760,
      "size_formatted": "240.0 KB",
      "url": "https://github.com/Bibyutatsu/ebooks/releases/download/v1.0-batch-03/..."
    }
  },
  "search_text": "sonar kella সোনার কেল্লা sonarkella satyajit ray সত্যজিৎ রায় feluda..."
}

The search_text field concatenates every searchable variant: Bengali title, English title, all transliterated romanizations, Bengali author name, English author name, known aliases, series name, and genre tags. That single string is what the browser tokenizes and matches against.

Limits

The 3.1MB catalog.json loads into browser memory on page open. On a fast connection or modern device this is instant. On a very slow mobile connection there is a noticeable pause before the grid appears. Splitting the catalog into chunks could fix that, but the tradeoff in code complexity is not worth it yet.

The transliteration engine handles standard modern Bengali script well. Regional dialects, historical spellings, and author-specific romanization preferences are a different story. The author mapping file covers the most common cases and the benchmark suite catches regressions, but unusual historical figures still slip through occasionally.

Cover quality varies. Some files have crisp 300x400 artwork. Others have 120x160 thumbnails stretched to fill the card. WebP conversion reduces file size but cannot fix the source.

GitHub Releases distribution uses 33 batch tags right now. GitHub does not document hard limits on asset count per repository, and 33 tags with 75 files each has worked fine. At 10,000 books it would be time to move to actual object storage.

Running it locally

git clone https://github.com/Bibyutatsu/ebooks.git
cd ebooks
python3 -m http.server 8000
# then open http://localhost:8000

Adding books:

# 1. Drop book folders into downloads/<Author>/<Title>/
python3 tools/build_catalog.py --downloads ../downloads --output .
python3 tests/verify_catalog.py ./catalog.json
python3 tools/sync_releases.py --catalog ./catalog.json
git add catalog.json assets/covers/
git commit -m "feat: add new books"
git push origin main

References

  1. Bibyutatsu/ebooks - Open-source Bengali ebook library, MIT licensed. ↩

You may also enjoy

Repolect: Reasoning-based code intelligence

11 minute read

Stop searching for code. Start repolecting about it 😉. Repolect indexes your codebase into a hierarchical semantic tree and knowledge graph, then navigates i...