Clean a TMX for MT training data
A translation memory is parallel data — and if you fine-tune an MT or LLM on it (you will), every junk segment is a noisy training example. Clean the TM before it becomes training data.
━━ the problem
Training an MT model on segments that a previous MT produced teaches the model its own mistakes. Filtering machine-origin units by their creationid signature breaks the loop.
"café" stored as "café" from a misread encoding looks fine in a spot-check but degrades the model. A mojibake detector flags source/target with encoding corruption so you drop or fix them.
A segment repeated 400 times over-weights that example. Deduplicating to one instance per source gives the model a balanced view of your data.
━━ how TM Cleaner handles it
Run the TM through dedup + junk + MT-origin + mojibake + placeholder-mismatch classifiers — plus, since October 2026, number-mismatch, wrong-script and punctuation-variant checks — preview what's removed, and export a clean TMX, a source⟍target TSV or a JSONL file, with a held-out test set if you want one.
- ✓MT-origin, mojibake, placeholder-mismatch and length-outlier classifiers target exactly the noise that hurts training
- ✓deduplication balances the example distribution instead of over-weighting repeated segments
- ✓TSV and JSONL exports give you a clean parallel corpus in the formats training tooling expects
- ✓a held-out test set (up to 20 %), chosen by source sentence so the same sentence never sits in both train and test
- ✓audit CSV documents every dropped pair — reproducible, defensible dataset hygiene
━━ frequently asked
can I export cleaned data as plain parallel text?+
yes. alongside the cleaned TMX you can export a TSV of source⟍target pairs — the shape most MT/LLM fine-tuning pipelines expect. the audit CSV records every pair that was dropped and why.
how do you detect machine-translated units?+
by their metadata origin — the creationid/changeid stamped by MT engines and pretranslation steps. you supply or confirm the MT-signature patterns; matching units are flagged as junk_mt_origin.
does cleaning alter the parallel alignment?+
no. cleaning removes or keeps whole units (both sides together) and never re-aligns anything. the one time text changes is if you turn on repair broken characters: encoding damage like “café” is then fixed in place. otherwise every kept pair is exactly as it was in your TM.
━━ related pages
free · no account up to 50 MB · files up to 2 GB · try the demo