diff --git a/README.md b/README.md index 572a69e..8fd858d 100644 --- a/README.md +++ b/README.md @@ -326,7 +326,7 @@ Open the **Workflow** tab and choose **Wolf RPG (WolfDawn)** from the engine sel |------|--------| | **0 Project** | Select the game folder, **Unpack** the `.wolf` archives into a loose `Data/` folder, then **Extract text** (maps, common events, databases, `Game.dat`, external event text, and the project-wide name glossary). Everything is staged into `files/` for translation. **Set up git tracking** by copying the `gameupdate/` folder (updater scripts, `patch.sh`/`patch.ps1`, and a `.gitignore` that excludes the original game files) into the game root, so the translation project can be tracked with git and later shipped as a git-based patch players apply themselves. Finally, **Format extracted JSON** (dazedformat) normalises `wolf_json/` and `files/` to the same layout the translator writes back (`json.dump`, indent 4) — run it once and commit it as your baseline so later injects produce clean, line-level git diffs instead of reformatting whole files. | | **1 Glossary** | Build both glossaries before translating. **1a - Character & Worldbuilding Glossary (`vocab.txt`):** copy the WOLF-tailored prompt into Cursor/Copilot with the extracted `files/` JSON, let it discover character names, speech registers, and lore terms, then paste the result into the in-tab editor and save (the shared `vocab.txt` is used by every translation batch to keep names and voice consistent). **1b - Name Value Glossary (`names.json`):** translate the name glossary first, since WOLF references item/skill/enemy names by value across many files. | -| **2 Translate** | Pick the **Translation mode** (Normal or Batch - Batch uses the Anthropic Batches API, ~50% cheaper, Claude only), then run the `Wolf RPG (WolfDawn)` module over `files/`. The chosen mode also applies to the Step 1b name glossary run. Only the `text` fields are filled in; `source` is preserved so injection can verify each line. | +| **2 Translate** | Pick the **Translation mode** (Normal or Batch - Batch uses the Anthropic Batches API, ~50% cheaper, Claude only), then run the `Wolf RPG (WolfDawn)` module over `files/`. The chosen mode also applies to the Step 1b name glossary run. Under **Speaker handling**, toggle which speaker formats are reshaped: WolfDawn tags who speaks on each line, and for lines where the name is baked into the first line (a nameplate) the translator reshapes them into the `[Speaker]: line` convention the prompt understands (translating the speaker tag), then restores WOLF's native `Speaker⏎line` layout on inject. Turn a format off if a game's first lines are not really speaker names. Only the `text` fields are filled in; `source` is preserved so injection can verify each line. | | **3 Inject** | Write the translations back into the game's `Data/` binaries **and** refresh the git-tracked `wolf_json/` with the translated JSON, so the game project's git diff shows both the JSON source and the injected binaries. Toggle `--en-punct` (convert Japanese punctuation to ASCII) and `--allow-code-drift` (relax the inline-code guard) as needed, and use **Check name consistency** to catch names translated differently across files. **Quick inject** lists the files you've translated so far (present in `translated/`) so you can tick just a few, inject them straight into `Data/`, and review the git diff / test in-game - ideal for iterating. **Inject all translations** writes every document and re-applies the name glossary across `Data/` for the final build. | | **4 Package** | Either run from the loose `Data/` folder (backs up `Data.wolf` → `Data.wolf.bak`), or **Repack** a fresh `Data.wolf`, inheriting the original archive's encryption. | | **5 Saves** | *(Optional)* Update existing `.sav` files so old Japanese saves load cleanly in the translated build. Originals are backed up automatically. | diff --git a/gui/wolf_workflow_tab.py b/gui/wolf_workflow_tab.py index 2504fbb..8354c7f 100644 --- a/gui/wolf_workflow_tab.py +++ b/gui/wolf_workflow_tab.py @@ -816,6 +816,7 @@ class WolfWorkflowTab(QWidget): )) self._add_tl_mode_selector(layout) + self._add_speaker_options(layout) btn = self._register(_make_btn("Translate all files now", "#00a86b")) btn.clicked.connect(lambda: self._navigate_to_translation(auto_start=True)) @@ -825,6 +826,47 @@ class WolfWorkflowTab(QWidget): open_btn.clicked.connect(lambda: self._navigate_to_translation(auto_start=False)) layout.addWidget(open_btn) + def _add_speaker_options(self, layout: QVBoxLayout): + """Toggles for which speaker formats are reshaped into '[Speaker]: line'.""" + from util import wolf_speakers + + layout.addWidget(_make_hr()) + layout.addWidget(self._subheading("Speaker handling")) + layout.addWidget(self._desc( + "WolfDawn tags who is speaking on each line. For lines where the name is baked " + "into the first line (a nameplate), the translator reshapes them into the " + "\"[Speaker]: line\" format the prompt understands, translates the speaker tag, then " + "restores WOLF's native \"Speaker⏎line\" layout on inject. Turn the formats off if a " + "game's first lines are not really speaker names." + )) + + cfg = wolf_speakers.load_config() + + self._speaker_hi_cb = QCheckBox( + "High-confidence nameplates (a face window precedes the line)" + ) + self._speaker_hi_cb.setChecked(bool(cfg.get("literal_line1", True))) + self._speaker_hi_cb.stateChanged.connect(self._save_speaker_options) + layout.addWidget(self._speaker_hi_cb) + + self._speaker_lo_cb = QCheckBox( + "Low-confidence first-line guesses (short first line, no preceding face window)" + ) + self._speaker_lo_cb.setChecked(bool(cfg.get("literal_line1_lowconf", True))) + self._speaker_lo_cb.stateChanged.connect(self._save_speaker_options) + layout.addWidget(self._speaker_lo_cb) + + def _save_speaker_options(self): + from util import wolf_speakers + + try: + wolf_speakers.save_config({ + "literal_line1": self._speaker_hi_cb.isChecked(), + "literal_line1_lowconf": self._speaker_lo_cb.isChecked(), + }) + except Exception as exc: + self._log(f"❌ Could not save speaker options: {exc}") + def _add_tl_mode_selector(self, layout: QVBoxLayout): """Normal vs Batch selector; applies to the name glossary (1b) and full runs.""" row = QHBoxLayout() diff --git a/modules/wolfdawn.py b/modules/wolfdawn.py index e4fb09e..897099c 100644 --- a/modules/wolfdawn.py +++ b/modules/wolfdawn.py @@ -17,6 +17,13 @@ Only entries whose ``source`` contains target-language (Japanese by default) text are sent to the model; everything else keeps ``text == source`` so inject is a no-op for it. Translated text is written back verbatim (no re-wrapping) to keep WolfDawn's inline-code and byte-exact guards happy. + +Speakers: WolfDawn tags each line with ``speaker`` / ``speaker_src``. For the +first-line formats (``literal_line1`` / ``literal_line1_lowconf``) the speaker +name is baked into line 1 of ``source``. Those lines are reshaped into the shared +``[Speaker]: line`` convention (which the prompt already translates) and restored +to WOLF's native ``Speaker\nline`` layout on write-back. See ``util.wolf_speakers``; +which formats are reshaped is configurable from the workflow. """ import json @@ -36,6 +43,7 @@ from util.translation import ( getPricingConfig, calculateCost, ) +from util import wolf_speakers # Globals (mirror the other engine modules; populated from .env at import time) MODEL = os.getenv("model") @@ -54,6 +62,12 @@ FILENAME = None # Regex - default matches Japanese (kanji, kana, full-width forms). LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+" +# Speaker handling: for first-line-speaker formats, reshape the line into the +# shared "[Speaker]: line" transport before translating and restore WOLF's +# native "Speaker\nline" layout on write-back. Which formats are reshaped is +# configurable from the workflow (data/wolf_speakers.json). +SPEAKER_CONFIG = wolf_speakers.load_config() + # Pricing / batching from the configured model PRICING_CONFIG = getPricingConfig(MODEL) INPUTAPICOST = PRICING_CONFIG["inputAPICost"] @@ -172,7 +186,22 @@ def parseDocument(data, filename): if not translatable: return [data, totalTokens, None] - sources = [e["source"] for e in translatable] + # Reshape first-line-speaker lines into the shared "[Speaker]: line" + # transport format. plans[i] carries what is needed to restore each + # entry after translation. + sources = [] + plans = [] # (entry, prefix, has_speaker) + for entry in translatable: + src = entry["source"] + split = wolf_speakers.split_source(src, entry.get("speaker_src", ""), SPEAKER_CONFIG) + if split is not None: + prefix, speaker, body = split + sources.append(wolf_speakers.to_prefixed(speaker, body)) + plans.append((entry, prefix, True)) + else: + sources.append(src) + plans.append((entry, "", False)) + try: response = translateAI(sources, []) except Exception as e: @@ -183,9 +212,18 @@ def parseDocument(data, filename): totalTokens[1] += tokens[1] # Write translations back (skip in estimate mode: translated == sources). - if not ESTIMATE and isinstance(translated, list) and len(translated) == len(translatable): - for entry, text in zip(translatable, translated): - if isinstance(text, str): + if not ESTIMATE and isinstance(translated, list) and len(translated) == len(plans): + for (entry, prefix, has_speaker), text in zip(plans, translated): + if not isinstance(text, str): + continue + if has_speaker: + speaker_en, body_en = wolf_speakers.parse_prefixed(text) + if speaker_en is not None: + entry["text"] = wolf_speakers.restore_source(prefix, speaker_en, body_en) + else: + # Model dropped the [Speaker]: prefix; keep its output as-is. + entry["text"] = prefix + text + else: entry["text"] = text return [data, totalTokens, None] diff --git a/tests/test_wolf_speakers.py b/tests/test_wolf_speakers.py new file mode 100644 index 0000000..56fb9d8 --- /dev/null +++ b/tests/test_wolf_speakers.py @@ -0,0 +1,106 @@ +#!/usr/bin/env python3 +"""Unit tests for util/wolf_speakers.py (first-line speaker reshaping).""" + +from __future__ import annotations + +import os +import sys +import unittest +from pathlib import Path + +ROOT = Path(__file__).resolve().parents[1] +os.chdir(ROOT) +sys.path.insert(0, str(ROOT)) + +from util import wolf_speakers as ws # noqa: E402 + +ALL_ON = {"literal_line1": True, "literal_line1_lowconf": True} +ALL_OFF = {"literal_line1": False, "literal_line1_lowconf": False} + + +class TestSplitSource(unittest.TestCase): + def test_splits_lowconf_when_enabled(self): + out = ws.split_source("市民\nおはよう\n元気？", "literal_line1_lowconf", ALL_ON) + self.assertEqual(out, ("", "市民", "おはよう\n元気？")) + + def test_splits_highconf_when_enabled(self): + out = ws.split_source("セルリア\nふふふ", "literal_line1", ALL_ON) + self.assertEqual(out, ("", "セルリア", "ふふふ")) + + def test_disabled_format_returns_none(self): + self.assertIsNone(ws.split_source("市民\nおはよう", "literal_line1_lowconf", ALL_OFF)) + + def test_non_firstline_src_returns_none(self): + for src in ("narration", "ui", "choice", "string_var", ""): + self.assertIsNone(ws.split_source("市民\nおはよう", src, ALL_ON)) + + def test_no_body_returns_none(self): + self.assertIsNone(ws.split_source("市民", "literal_line1", ALL_ON)) + + def test_preserves_window_option_prefix(self): + out = ws.split_source("@2\n市民\nおはよう", "literal_line1", ALL_ON) + self.assertEqual(out, ("@2\n", "市民", "おはよう")) + + +class TestPrefixedRoundTrip(unittest.TestCase): + def test_to_prefixed(self): + self.assertEqual(ws.to_prefixed("市民", "おはよう"), "[市民]: おはよう") + + def test_parse_prefixed(self): + self.assertEqual(ws.parse_prefixed("[Citizen]: Good morning"), ("Citizen", "Good morning")) + + def test_parse_prefixed_multiline_body(self): + spk, body = ws.parse_prefixed("[Celria]: Wave.\nSmile.") + self.assertEqual(spk, "Celria") + self.assertEqual(body, "Wave.\nSmile.") + + def test_parse_prefixed_no_prefix(self): + self.assertEqual(ws.parse_prefixed("Just narration"), (None, "Just narration")) + + def test_restore_source(self): + self.assertEqual(ws.restore_source("", "Citizen", "Good morning"), "Citizen\nGood morning") + + def test_restore_source_with_window_prefix(self): + self.assertEqual( + ws.restore_source("@2\n", "Citizen", "Hi"), "@2\nCitizen\nHi" + ) + + def test_full_round_trip_structure_preserved(self): + source = "市民\nおはよう\n元気？" + prefix, speaker, body = ws.split_source(source, "literal_line1_lowconf", ALL_ON) + transport = ws.to_prefixed(speaker, body) + self.assertEqual(transport, "[市民]: おはよう\n元気？") + # Simulate a translation that keeps the [Speaker]: format. + spk_en, body_en = ws.parse_prefixed("[Citizen]: Morning\nDoing well?") + restored = ws.restore_source(prefix, spk_en, body_en) + self.assertEqual(restored, "Citizen\nMorning\nDoing well?") + # Same number of newlines as the original layout (speaker on its own line). + self.assertEqual(source.count("\n"), restored.count("\n")) + + +class TestConfigIO(unittest.TestCase): + def test_load_defaults_when_missing(self): + orig = ws.CONFIG_PATH + try: + ws.CONFIG_PATH = ROOT / "tests" / "_nonexistent_wolf_speakers.json" + self.assertEqual(ws.load_config(), ws.DEFAULT_CONFIG) + finally: + ws.CONFIG_PATH = orig + + def test_save_and_load_roundtrip(self): + import tempfile + + orig = ws.CONFIG_PATH + try: + with tempfile.TemporaryDirectory() as td: + ws.CONFIG_PATH = Path(td) / "wolf_speakers.json" + ws.save_config({"literal_line1": False, "literal_line1_lowconf": True}) + loaded = ws.load_config() + self.assertFalse(loaded["literal_line1"]) + self.assertTrue(loaded["literal_line1_lowconf"]) + finally: + ws.CONFIG_PATH = orig + + +if __name__ == "__main__": + unittest.main(verbosity=2) diff --git a/tests/test_wolfdawn.py b/tests/test_wolfdawn.py index 5afe285..b4c8a16 100644 --- a/tests/test_wolfdawn.py +++ b/tests/test_wolfdawn.py @@ -154,6 +154,96 @@ class TestTranslationWriteback(unittest.TestCase): self.assertEqual(data["scenes"][0]["lines"][0]["text"], "こんにちは") +import re # noqa: E402 + + +def _mock_translate_speaker(text, history=None, history_ctx=None): + """Emulate a model that keeps the [Speaker]: format when present.""" + def one(t): + m = re.match(r"^\[([^\]]*)\]:\s*(.*)$", t, re.DOTALL) + if m: + return f"[EN_{m.group(1)}]: EN_{m.group(2)}" + return f"EN_{t}" + + if isinstance(text, list): + return [[one(t) for t in text], [1, 1]] + return [one(text), [1, 1]] + + +SPEAKER_MAP_DOC = { + "file": "OP.mps", + "kind": "map", + "scenes": [ + { + "event": 0, "name": "ev", + "lines": [ + {"cmd": 26, "str": 0, "speaker": "市民", "speaker_src": "literal_line1_lowconf", + "source": "市民\nおはよう\n元気？", "text": "市民\nおはよう\n元気？"}, + {"cmd": 27, "str": 0, "speaker": "セルリア", "speaker_src": "literal_line1", + "source": "セルリア\nふふふ", "text": "セルリア\nふふふ"}, + {"cmd": 28, "str": 0, "speaker": "Narration", "speaker_src": "narration", + "source": "むかしむかし", "text": "むかしむかし"}, + ], + } + ], +} + + +class _SpeakerHarness: + """Run parseDocument with the speaker-aware mock and a chosen speaker config.""" + + def __init__(self, config): + self.config = config + self.captured = [] + + def run(self, data, filename="OP.mps.json"): + def translate(text, history=None, history_ctx=None): + self.captured.append(copy.deepcopy(text)) + return _mock_translate_speaker(text, history, history_ctx) + + orig_t, orig_est, orig_cfg = wd.translateAI, wd.ESTIMATE, wd.SPEAKER_CONFIG + wd.translateAI = translate + wd.ESTIMATE = False + wd.SPEAKER_CONFIG = self.config + try: + result = wd.parseDocument(copy.deepcopy(data), filename) + return result, self.captured + finally: + wd.translateAI, wd.ESTIMATE, wd.SPEAKER_CONFIG = orig_t, orig_est, orig_cfg + + +class TestSpeakerReshaping(unittest.TestCase): + def test_firstline_speakers_reshaped_and_restored(self): + cfg = {"literal_line1": True, "literal_line1_lowconf": True} + (data, _t, err), captured = _SpeakerHarness(cfg).run(SPEAKER_MAP_DOC) + self.assertIsNone(err) + # Model saw the [Speaker]: transport for the two nameplate lines. + self.assertEqual( + captured[0], + ["[市民]: おはよう\n元気？", "[セルリア]: ふふふ", "むかしむかし"], + ) + lines = data["scenes"][0]["lines"] + # Restored to WOLF's native Speaker\nbody layout. + self.assertEqual(lines[0]["text"], "EN_市民\nEN_おはよう\n元気？") + self.assertEqual(lines[1]["text"], "EN_セルリア\nEN_ふふふ") + # Narration was translated as a plain blob. + self.assertEqual(lines[2]["text"], "EN_むかしむかし") + # Sources are preserved for the inject drift guard. + self.assertEqual(lines[0]["source"], "市民\nおはよう\n元気？") + # Original layout (newline count) preserved on the reshaped lines. + self.assertEqual(lines[0]["source"].count("\n"), lines[0]["text"].count("\n")) + + def test_disabled_format_sends_raw_blob(self): + cfg = {"literal_line1": True, "literal_line1_lowconf": False} + (data, _t, err), captured = _SpeakerHarness(cfg).run(SPEAKER_MAP_DOC) + self.assertIsNone(err) + # Low-confidence line is sent as the raw source (no reshaping). + self.assertIn("市民\nおはよう\n元気？", captured[0]) + self.assertIn("[セルリア]: ふふふ", captured[0]) + lines = data["scenes"][0]["lines"] + self.assertEqual(lines[0]["text"], "EN_市民\nおはよう\n元気？") + + class TestOpenFiles(unittest.TestCase): def test_rejects_unknown_kind(self): with tempfile.TemporaryDirectory() as td: diff --git a/util/wolf_speakers.py b/util/wolf_speakers.py new file mode 100644 index 0000000..09493cf --- /dev/null +++ b/util/wolf_speakers.py @@ -0,0 +1,121 @@ +"""Speaker handling for the WolfDawn translation module. + +WolfDawn attributes a speaker to every extracted line (``speaker`` / +``speaker_src`` fields). For the two "first-line" formats the speaker name is +baked into line 1 of the ``source`` text, e.g.:: + + "市民\nおぉっ！来た！帰ってきたぞ！" speaker_src = literal_line1_lowconf + "セルリア\nほーら、ローザも手を振って。" speaker_src = literal_line1 + +The shared translation prompt (``data/prompt.txt``) is built around the RPG +Maker ``[Speaker]: line`` convention and already knows to translate the speaker +tag ("Always translate speaker tags to English: ``[クロネ]:`` -> ``[Kurone]:``"). +So when translating we reshape those lines into ``[Speaker]: body`` and, on +write-back, restore WOLF's native ``Speaker\nbody`` structure so injection stays +byte-faithful to the original layout. + +Which formats are reshaped is configurable (``data/wolf_speakers.json``) so the +workflow can toggle the high-confidence nameplate format and the lower-confidence +first-line guess independently. +""" + +from __future__ import annotations + +import json +import re + +from util.paths import DATA_DIR +from util.speaker_prefix import SPEAKER_TAG_RE, strip_speaker_prefix + +# speaker_src values whose name is baked into line 1 of the source text. +FIRSTLINE_SRCS = ("literal_line1", "literal_line1_lowconf") + +# Default: reshape both first-line formats. WolfDawn already gates the +# low-confidence one (short line 1, no control codes, no sentence punctuation), +# so it is safe enough to enable by default. +DEFAULT_CONFIG = { + "literal_line1": True, + "literal_line1_lowconf": True, +} + +CONFIG_PATH = DATA_DIR / "wolf_speakers.json" + +# Optional leading window-option prefix (``@