# TCG Card Recognizer

By Munawar Khel / Independent Practice / Freelance UI/UX Designer / 2026 / 9 min read

Published at https://munawarkhel.com/work/independent-practice/tcg-ocr-recognizer/

Delivered a working card recognizer to the Girudo team, unblocking a capability their own attempt had stalled on, built and handed over as an independent project alongside the platform design engagement

- Company: Independent Practice
- My role: Freelance UI/UX Designer
- Timeline: 2026
- Platform: Web. Browser SPA (React 19 and Vite) with a Python recognition backend
- Category: Computer Vision / OCR / Trading Card Games
- Status: New design, delivered
- Tools: Claude Code, Python, Flask, OpenCV, PyTorch, ONNX Runtime, Apple Vision, RapidOCR, Tesseract, Google Gemini, React 19, Vite, TypeScript
- Team: Solo build (concept, design, full-stack)

## Context

Trading card play has moved onto camera. Players stream their games, play remotely against a webcam pointed at the table, and increasingly expect the software watching that table to understand the game rather than just relay it. The reference point for that behaviour is SpellTable, where clicking a card in a video feed tells you which card it is.

Girudo is a trading card game community and creator platform I had already designed end to end as a separate engagement. Its team had been trying to build the card recognition the platform depends on and had run into an accuracy wall they could not get past. I took it on as an independent project, built it separately from the platform design work, and delivered the working system to them.

The build targets Magic: The Gathering first, with the full catalog of 27,663 cards carrying complete visual signatures, and the interface prepared for four further games behind the same pipeline.

## The problem

Girudo's premise is that people play with their actual physical cards online, streaming their table to opponents and spectators. That premise only holds if the software knows what is on the table. A camera feed of a card game is just video, and video is what Twitch already does better than a startup can. The thing incumbents structurally cannot offer is hosting the game itself, and hosting the game starts with recognising the cards in it. Recognition was not one feature among the platform's modules. It was the capability that the differentiating half of the product rested on.

Everything downstream needs the same question answered first: which card is that. A spectator following a match cannot read a card held up to a webcam across a room, so the card played has to be identified on screen for the viewing experience to work at all. An opponent playing remotely needs the same thing to make a decision without asking someone to hold a card closer to the lens. Decklists and state tracking need to know what entered play. Without recognition, every one of those becomes manual typing, and a player who has to stop and type is a player who goes back to Discord and a webcam. The feature that justifies the platform quietly becomes the feature nobody uses.

That need had gone unmet because the problem is much harder than it looks from outside. Reading text from an image is solved; reading a trading card on a table is not the same problem. Foils throw glare across the exact band the title sits in. Borderless and showcase printings move or remove the title band entirely. Cards on a real table are angled, partially shadowed, and lit by whatever lamp is in the room. Older frames use type treatments no modern OCR model was trained on. Every one of these is normal rather than exceptional, and each degrades the single signal the obvious approach depends on.

The second constraint only appears at scale, and it is the one that decides whether the feature is usable. When the reference catalog holds tens of thousands of cards, a weak reading does not fail visibly. It produces a confident wrong answer, because somewhere among 27,663 candidates something always plausibly matches a bad read. A system that names the wrong card is worse than no system, because a player who cannot trust it has to verify every result by hand, at which point the tool costs more attention than it saves. That is where the effort had stalled: not at getting it working on clean cards, but at getting it trustworthy enough that a player would leave it switched on.

## What I did

The core move is to stop treating this as an OCR problem. Instead of one signal read well, the recognizer reads four independent signals and fuses them: OCR on the title band, a 256-bit perceptual hash of the artwork window verified by normalised cross-correlation against stored thumbnails, ORB keypoints with RANSAC homography verification, and a coarse colour and region feature vector. Each runs across four orientations, so a rotated or upside-down card costs nothing. A whole-card OCR rescue pass handles borderless and showcase frames that have no standard title band to read.

The signals fail in different ways, which is the point. Glare that ruins the title band leaves the artwork hash intact. A borderless frame with no title still has distinctive art and keypoints. Fusion scores agreement between signals higher than any single strong reading, and applies an explicit penalty to a result carried by one signal alone once the index grows past a few thousand cards, which is exactly the condition that produces confident wrong answers.

Underneath the fusion sits a stated principle: a wrong answer is far worse than no answer. The system carries a confidence floor and abstains below it, returning ranked candidates rather than a bare failure, so an uncertain read surfaces as a short list the player resolves in one click instead of a false certainty they have to catch themselves.

Above it sits an escalation tier. When the local pipeline is not confident, the click escalates to Google Gemini as a vision fallback with a structured output schema. Gemini's answer is then grounded against the local catalog by fuzzy name match and written back into the local index, so the same physical card matches locally and instantly on every future sighting. The system gets faster and cheaper the more it is used, and the cloud dependency shrinks with use rather than growing.

## Outcomes

- Delivered a working card recognizer to the Girudo team, unblocking a capability their own attempt had stalled on, built and handed over as an independent project alongside the platform design engagement
- Built a recognition pipeline that fuses four independent signals across four orientations, so the failure modes that defeat OCR alone, foils, glare, angled cards, and borderless frames, are covered by signals that do not fail together
- Indexed the complete Magic: The Gathering catalog of 27,663 cards with full visual signature coverage, searchable as a vectorised bit-distance lookup rather than a scan
- Demonstrated a self-improving escalation model in which cloud vision handles only what the local pipeline cannot, and every escalated answer is written back so the same card resolves locally and instantly thereafter
- Kept the live camera loop entirely client-side at roughly 5 ms per frame, reducing a recognition to a single request carrying one crop, with indexed cards returning a verdict in 0.1 to 0.4 seconds
- Made the recognizer's reasoning observable through a live activity trace and ranked alternatives, so an uncertain result is legible to the player rather than presented as a false certainty
- Produced a reusable evaluation apparatus, a synthetic scene generator with lighting and glare simulation plus a benchmark harness that reports which signal carried each identification, so future changes can be measured rather than guessed at

## Decisions

### Multi-signal fusion instead of better OCR

What: Read four independent signals per card, OCR, artwork perceptual hash, ORB keypoints, and a colour feature vector, across four orientations, and fuse them with agreement scoring rather than picking the single best reading.

Alternatives: Investing in a stronger OCR engine or fine-tuning a text model on card fonts, which is the direction the original attempt had taken.

Why: The failure modes that matter are foils, glare, angle, and borderless frames, and every one of them attacks the title band specifically. No text engine survives a title that is physically obscured. Signals that fail independently can cover for each other; a single signal, however good, cannot.

### Abstain rather than guess

What: Set an explicit confidence floor, return ranked candidates below it, and treat the abstain rate as a first-class metric in the benchmark harness alongside hits and misses.

Alternatives: Always returning the top match, which is what a naive nearest-neighbour lookup does by construction.

Why: At catalog scale a weak signal always finds something plausible, so the failure mode is not a miss but a confident wrong answer. A player who cannot trust the output has to verify every result, which costs more time than the tool saves. Surfacing three candidates the player resolves in one click preserves trust; naming the wrong card destroys it.

### Cloud as an escalation tier that teaches the local index

What: Route only low-confidence clicks to Gemini, ground its answer against the local catalog, and write the result back into the local signature index so the next sighting of that card matches locally.

Alternatives: Sending every click to a cloud vision API, or refusing any cloud dependency and accepting the misses.

Why: Cloud-first is slower per click, costs money per click, and needs connectivity at a game table. Making it the exception rather than the rule keeps the common path fast and free, and the write-back means the dependency shrinks with use instead of compounding.

### Detector choice exposed as a user control

What: Shipped both the classical OpenCV detector and the neural segmentation model as a visible toggle in the interface, with their tradeoff stated, rather than silently selecting one.

Alternatives: Picking the more accurate model and hiding the classical path, or auto-switching based on device capability.

Why: The two occupy genuinely different points on a speed and precision tradeoff, and which one is right depends on the machine and the lighting in front of it. The neural path is also still weak on out-of-distribution objects like hands and keyboards. Exposing the choice let the recipient tune to their own setup instead of inheriting my assumptions about it.

## What I learned

The accuracy wall was not where the team thought it was. They were trying to read cards better, and the answer was to stop depending on reading them at all. Reframing a recognition problem as a signal fusion problem rather than a text extraction problem is what unlocked it, and that reframing was worth more than any amount of tuning on the original approach. It is a reminder that when a technical effort stalls, the constraint is often in how the problem was framed rather than in the execution.

Designing for uncertainty is a product decision, not an engineering detail. The choice to abstain and offer candidates rather than assert a best guess shaped the confidence floor, the fusion penalty, the results panel, and the correction dialog, all of which exist because the system is allowed to be unsure. Building a product that admits doubt turned out to be what made it trustworthy enough to use.

Measurement changed the architecture more than intuition did. Testing under synthetic glare and low light, rather than on clean scans, is what revealed that crop precision matters more than crop convenience, and that keypoint matching stays discriminative exactly where perceptual hashing degrades. Neither was obvious in advance, and both moved real structure in the system. The open items are honest ones: two adjacent cards can still merge into a single detection, and the neural detector needs a retrain with hard negatives before it stops firing on hands and keyboards.

Tags: AI Product Design, Claude Code, Full-Stack