Flagship Methodology · Framework v1.0
How we test: the Global Language Quality Benchmark Framework
Every score on LocRank comes from a fixed public corpus, a deterministic scoring script, and archived raw outputs. No sponsored placements, no star ratings, no estimates. If a tool has not been run, its page says “test pending” and explains exactly why.
Principles
Measured, never sponsored
Vendors cannot pay for a score, a ranking or a review. Tool links on this site are plain official links; LocRank earns no affiliate commission from them (status as of 2026-11-21, disclosed on every page).
Append-only evidence
Each benchmark run is published as a dated, permanent archive page with the full raw outputs. Runs are never edited after publication; corrections and re-tests ship as new runs. Historical data is the asset — it cannot be recreated retroactively.
Reproducible by anyone
The corpus, the term lists, the formulas and the scoring weights are published on this page. Anyone with the same inputs gets the same numbers.
Conflicts, disclosed
LocRank is published by HankLoc, which sells localization services. That is exactly why the method is mechanical: a deterministic script scores archived outputs, and no commercial relationship enters the formula.
The framework: one architecture, many language benchmarks
A benchmark is a (language, corpus, rule set, scoring script) bundle with its own version number. The framework defines the shared metrics — script purity, wording density, terminology consistency — and each language benchmark supplies its own corpus and norm references. The Traditional Chinese Quality Benchmark (TCQ) is the flagship and the first live instance; Simplified Chinese is in development, and Japanese, Korean and Spanish slots are reserved in the same architecture. A language with no completed run publishes no scores.
Traditional Chinese Quality Benchmark — v1.0 specification
Norm reference: Traditional Chinese as written in Taiwan. Input: 24 fixed English segments across four content types (marketing, UI strings, game dialogue, customer support), chosen to contain known cross-strait wording traps and repeated key terms. Access: each engine is called through its public or API access path, once per segment, with no human editing; the access path is recorded on each tool page and in the run archive.
The three metrics
| Metric | Definition | Formula | Weight |
|---|---|---|---|
| Script purity | Share of CJK characters in the output that are simplified-only characters, detected character-by-character with the OpenCC s2t conversion table. | max(0, 100 − residue% × 4) | 40% |
| Wording | Density of mainland-China wording, matched greedily (longest-first) against a 131-term rule list — the same list that powers this site's free wording checker. | max(0, 100 − hits per 100 CJK chars × 10) | 35% |
| Terminology | For each tagged term occurrence: the preferred Taiwan rendering scores 1, a mainland variant scores 0, any other rendering is excluded and disclosed. | preferred / (preferred + mainland) × 100 | 25% |
Composite score = 0.40 × purity + 0.35 × wording + 0.25 × terminology, rounded to the nearest integer. Grades: A ≥ 90 · B ≥ 80 · C ≥ 70 · D ≥ 60 · F < 60.
Worked example (segment M1, run of 2026-11-21)
| Engine | Raw output (archived) | Flags |
|---|---|---|
| Google Translate | 觀看視頻,了解我們的軟體如何幫助您的團隊在線上工作。 | 視頻 (mainland term for “video”), 在線 (mainland phrasing) |
| Gemini | 觀看影片,了解我們的軟體如何協助您的團隊進行線上協作。 | None |
Full per-segment evidence: Benchmark run 2026-11-21 (v1.0).
The corpus (v1.0, frozen)
| ID | Type | English source segment | Tagged terms |
|---|---|---|---|
| M1 | marketing | Watch the video to see how our software helps your team work online. | video, software, online |
| M2 | marketing | High-quality sound and unbeatable value for money — download the app today. | quality, value for money |
| M3 | marketing | Get instant message notifications and free shipping on every order. | message |
| M4 | marketing | Our network covers the whole city, so you can take a taxi or a bus to get anywhere. | network |
| M5 | marketing | Sign up now and print your boarding pass before the countdown ends. | |
| M6 | marketing | This smartphone has a bigger screen and a smarter default search. | screen, default, search |
| U1 | ui | Settings | settings |
| U2 | ui | Log in to your account | log in |
| U3 | ui | Search settings and message notifications | search, settings, message |
| U4 | ui | Download the software update | software |
| U5 | ui | Watch the video in high quality | video, quality |
| U6 | ui | Default network settings | default, network, settings |
| G1 | game | Watch the opening video, then open settings to adjust the screen brightness. | video, settings, screen |
| G2 | game | Collect 100 coins to level up and unlock the secret level. | — |
| G3 | game | The final countdown begins! Your team must protect the crystal at all costs. | — |
| G4 | game | Check your messages — the guild leader has sent you a new quest. | message |
| G5 | game | This boss drops high-quality gear with excellent value for money. | quality, value for money |
| G6 | game | Download the update to play online with friends around the world. | online |
| S1 | support | Please contact our online support team if you cannot log in. | online, log in |
| S2 | support | You can print your receipt from the account settings page. | print, settings |
| S3 | support | We will send you a text message to verify your phone number. | text message |
| S4 | support | If the video will not play, check your network connection. | video, network |
| S5 | support | Your software license expires in 7 days. Download the latest version now. | software |
| S6 | support | Thank you for your message. Our support team will reply within 24 hours. | message |
Terminology checks
| English term | Preferred (Taiwan) | Mainland variants |
|---|---|---|
| default | 預設 | 默認 / 默认 |
| log in | 登入 | 登錄 / 登录 |
| message | 訊息 | 信息 |
| network | 網路 | 網絡 / 网络 |
| online | 線上 | 在線 / 在线 |
| 列印 | 打印 | |
| quality | 品質 | 質量 / 质量 |
| screen | 螢幕 | 屏幕 |
| search | 搜尋 | 搜索 |
| settings | 設定 | 設置 / 设置 |
| software | 軟體 | 軟件 / 软件 |
| text message | 簡訊 | 短信 |
| value for money | CP 值 / CP值 | 性價比 / 性价比 |
| video | 影片 | 視頻 / 视频 |
How platforms (TMS) are scored
A translation management system does not itself translate: it routes work to MT engines and humans. Scoring a TMS therefore means running its built-in or default pre-translation path on the same corpus, inside a real account, and scoring what comes out — with the access path disclosed. Until that account-based run is executed, a TMS entry shows “test pending”. We do not borrow the score of an engine a TMS can connect to; connections are configuration, not output.
Limitations, stated plainly
- One run per engine per benchmark version; engines change without notice, so scores are dated and re-tested quarterly.
- The corpus is 24 segments. It is designed to discriminate script and wording quality, not to grade literary translation.
- LLM scores belong to the exact model version and prompt archived in the run — a different prompt is a different configuration.
- Wording hits are machine flags from a published rule list; a human reviewer can dispute a flag, and disputes are resolved in the next run, never by editing a published one.
Cite this benchmark
LocRank, Traditional Chinese Quality Benchmark, v1.0 (2026-11-21). Corpus, method and run archive: locrank.com/directory/methodology/ and locrank.com/directory/benchmarks/zh-hant/2026-11-21/. Linking and citation are welcome; scores consumed programmatically should follow the API terms (attribution required).
Get the full benchmark dataset — and every future score — by email
The public pages show the headline scores. The email edition carries the complete dataset behind them:
- The full per-segment benchmark dataset for every run (all 24 segments, every engine, every flag)
- New-tool test alerts — the day a pending tool gets its first measured score
- Quarterly re-test results, including score changes and what caused them
Free. One list, no spam, unsubscribe anytime. You will receive a confirmation email first.