Proper wrapping and speaker catching
This commit is contained in:
parent
e3a46cb8e6
commit
d14819700b
5 changed files with 218 additions and 19 deletions
|
|
@ -326,7 +326,7 @@ Open the **Workflow** tab and choose **Wolf RPG (WolfDawn)** from the engine sel
|
|||
|------|--------|
|
||||
| **0 Project** | Select the game folder, **Unpack** the `.wolf` archives into a loose `Data/` folder, then **Extract text** (maps, common events, databases, `Game.dat`, external event text, and the project-wide name glossary). Everything is staged into `files/` for translation. **Set up git tracking** by copying the `gameupdate/` folder (updater scripts, `patch.sh`/`patch.ps1`, and a `.gitignore` that excludes the original game files) into the game root, so the translation project can be tracked with git and later shipped as a git-based patch players apply themselves. Finally, **Format extracted JSON** (dazedformat) normalises `wolf_json/` and `files/` to the same layout the translator writes back (`json.dump`, indent 4) — run it once and commit it as your baseline so later injects produce clean, line-level git diffs instead of reformatting whole files. |
|
||||
| **1 Glossary** | Build both glossaries before translating. **1a - Character & Worldbuilding Glossary (`vocab.txt`):** copy the WOLF-tailored prompt into Cursor/Copilot with the extracted `files/` JSON, let it discover character names, speech registers, and lore terms, then paste the result into the in-tab editor and save (the shared `vocab.txt` is used by every translation batch to keep names and voice consistent). **1b - Name Value Glossary (`names.json`):** translate the name glossary first, since WOLF references item/skill/enemy names by value across many files. |
|
||||
| **2 Translate** | Pick the **Translation mode** (Normal or Batch - Batch uses the Anthropic Batches API, ~50% cheaper, Claude only), then run the `Wolf RPG (WolfDawn)` module over `files/`. The chosen mode also applies to the Step 1b name glossary run. Under **Speaker handling**, toggle which speaker formats are reshaped: WolfDawn tags who speaks on each line, and for lines where the name is baked into the first line (a nameplate) the translator reshapes them into the `[Speaker]: line` convention the prompt understands (translating the speaker tag), then restores WOLF's native `Speaker⏎line` layout on inject. Turn a format off if a game's first lines are not really speaker names. Only the `text` fields are filled in; `source` is preserved so injection can verify each line. |
|
||||
| **2 Translate** | Pick the **Translation mode** (Normal or Batch - Batch uses the Anthropic Batches API, ~50% cheaper, Claude only), then run the `Wolf RPG (WolfDawn)` module over `files/`. The chosen mode also applies to the Step 1b name glossary run. Under **Speaker handling**, toggle which speaker formats are reshaped: WolfDawn tags who speaks on each line, and for lines where the name is baked into the first line (a nameplate) the translator reshapes them into the `[Speaker]: line` convention the prompt understands (translating the speaker tag), then restores WOLF's native `Speaker⏎line` layout on inject. Turn a format off if a game's first lines are not really speaker names. Under **Text wrap width**, set whether/where translated dialogue re-wraps to fit WOLF's message box (same `dazedwrap` engine the RPG Maker workflow uses, saved to `.env` as `wolfWrap` / `wolfWidth`); only the dialogue body is wrapped - a speaker's nameplate line is always kept separate so the `⏎` after a name is never folded into the text. Only the `text` fields are filled in; `source` is preserved so injection can verify each line. |
|
||||
| **3 Inject** | Write the translations back into the game's `Data/` binaries **and** refresh the git-tracked `wolf_json/` with the translated JSON, so the game project's git diff shows both the JSON source and the injected binaries. Toggle `--en-punct` (convert Japanese punctuation to ASCII) and `--allow-code-drift` (relax the inline-code guard) as needed, and use **Check name consistency** to catch names translated differently across files. **Quick inject** lists the files you've translated so far (present in `translated/`) so you can tick just a few, inject them straight into `Data/`, and review the git diff / test in-game - ideal for iterating. **Inject all translations** writes every document and re-applies the name glossary across `Data/` for the final build. |
|
||||
| **4 Package** | Either run from the loose `Data/` folder (backs up `Data.wolf` → `Data.wolf.bak`), or **Repack** a fresh `Data.wolf`, inheriting the original archive's encryption. |
|
||||
| **5 Saves** | *(Optional)* Update existing `.sav` files so old Japanese saves load cleanly in the translated build. Originals are backed up automatically. |
|
||||
|
|
|
|||
|
|
@ -47,6 +47,7 @@ from PyQt5.QtWidgets import (
|
|||
QMessageBox,
|
||||
QPushButton,
|
||||
QScrollArea,
|
||||
QSpinBox,
|
||||
QSplitter,
|
||||
QTabWidget,
|
||||
QTextEdit,
|
||||
|
|
@ -817,6 +818,7 @@ class WolfWorkflowTab(QWidget):
|
|||
|
||||
self._add_tl_mode_selector(layout)
|
||||
self._add_speaker_options(layout)
|
||||
self._add_wrap_options(layout)
|
||||
|
||||
btn = self._register(_make_btn("Translate all files now", "#00a86b"))
|
||||
btn.clicked.connect(lambda: self._navigate_to_translation(auto_start=True))
|
||||
|
|
@ -867,6 +869,91 @@ class WolfWorkflowTab(QWidget):
|
|||
except Exception as exc:
|
||||
self._log(f"❌ Could not save speaker options: {exc}")
|
||||
|
||||
def _add_wrap_options(self, layout: QVBoxLayout):
|
||||
"""Dialogue text-wrap width (like the RPGMaker workflow, via dazedwrap)."""
|
||||
layout.addWidget(_make_hr())
|
||||
layout.addWidget(self._subheading("Text wrap width"))
|
||||
layout.addWidget(self._desc(
|
||||
"Re-wraps the translated English dialogue so it fits WOLF's message box. "
|
||||
"Only the dialogue body is wrapped - a speaker's name line (the line break "
|
||||
"right after a nameplate) is always kept separate. Lower the width if lines "
|
||||
"overflow in-game; raise it if text wraps too early. Applies on the next run."
|
||||
))
|
||||
|
||||
wrap_env = self._read_env_values(["wolfWrap", "wolfWidth"])
|
||||
|
||||
self._wrap_enable_cb = QCheckBox("Wrap translated dialogue")
|
||||
enabled = str(wrap_env.get("wolfWrap", "true")).strip().lower() != "false"
|
||||
self._wrap_enable_cb.setChecked(enabled)
|
||||
self._wrap_enable_cb.stateChanged.connect(self._apply_wrap_config)
|
||||
layout.addWidget(self._wrap_enable_cb)
|
||||
|
||||
row = QHBoxLayout()
|
||||
width_lbl = QLabel("Width (characters):")
|
||||
width_lbl.setStyleSheet("color:#cccccc;font-size:12px;background:transparent;")
|
||||
self._wrap_width_spin = QSpinBox()
|
||||
self._wrap_width_spin.setRange(20, 300)
|
||||
try:
|
||||
self._wrap_width_spin.setValue(int(wrap_env.get("wolfWidth") or 55))
|
||||
except ValueError:
|
||||
self._wrap_width_spin.setValue(55)
|
||||
self._wrap_width_spin.setFixedWidth(70)
|
||||
self._wrap_width_spin.valueChanged.connect(self._apply_wrap_config)
|
||||
row.addWidget(width_lbl)
|
||||
row.addWidget(self._wrap_width_spin)
|
||||
row.addStretch()
|
||||
layout.addLayout(row)
|
||||
|
||||
def _apply_wrap_config(self, *args):
|
||||
"""Persist wolfWrap / wolfWidth to .env so the next translation run uses them."""
|
||||
updates = {
|
||||
"wolfWrap": "true" if self._wrap_enable_cb.isChecked() else "false",
|
||||
"wolfWidth": str(self._wrap_width_spin.value()),
|
||||
}
|
||||
if self._write_env_values(updates):
|
||||
self._log(
|
||||
"Text wrap updated: "
|
||||
+ ", ".join(f"{k}={v}" for k, v in updates.items())
|
||||
)
|
||||
|
||||
def _read_env_values(self, keys) -> dict:
|
||||
"""Read a few `key='value'` pairs from .env (best effort)."""
|
||||
import re as _re
|
||||
out = {}
|
||||
env_path = Path(".env")
|
||||
if not env_path.exists():
|
||||
return out
|
||||
try:
|
||||
text = env_path.read_text(encoding="utf-8")
|
||||
except Exception:
|
||||
return out
|
||||
for key in keys:
|
||||
m = _re.search(rf"^{_re.escape(key)}\s*=\s*'([^']*)'", text, _re.MULTILINE)
|
||||
if m:
|
||||
out[key] = m.group(1)
|
||||
return out
|
||||
|
||||
def _write_env_values(self, updates: dict) -> bool:
|
||||
"""Update/insert `key='value'` pairs in .env. Returns True on success."""
|
||||
import re as _re
|
||||
env_path = Path(".env")
|
||||
try:
|
||||
text = env_path.read_text(encoding="utf-8") if env_path.exists() else ""
|
||||
for key, val in updates.items():
|
||||
text, n = _re.subn(
|
||||
rf"^({_re.escape(key)}\s*=\s*')[^']*(')",
|
||||
rf"\g<1>{val}\g<2>",
|
||||
text,
|
||||
flags=_re.MULTILINE,
|
||||
)
|
||||
if n == 0:
|
||||
text = text.rstrip("\n") + f"\n{key}='{val}'\n"
|
||||
env_path.write_text(text, encoding="utf-8")
|
||||
return True
|
||||
except Exception as exc:
|
||||
self._log(f"❌ Could not update .env: {exc}")
|
||||
return False
|
||||
|
||||
def _add_tl_mode_selector(self, layout: QVBoxLayout):
|
||||
"""Normal vs Batch selector; applies to the name glossary (1b) and full runs."""
|
||||
row = QHBoxLayout()
|
||||
|
|
|
|||
|
|
@ -15,8 +15,14 @@ Supported document ``kind``s (all share the ``{source, text}`` leaf pattern):
|
|||
|
||||
Only entries whose ``source`` contains target-language (Japanese by default)
|
||||
text are sent to the model; everything else keeps ``text == source`` so inject
|
||||
is a no-op for it. Translated text is written back verbatim (no re-wrapping) to
|
||||
keep WolfDawn's inline-code and byte-exact guards happy.
|
||||
is a no-op for it.
|
||||
|
||||
Text wrapping: translated dialogue is re-wrapped to a character width (like the
|
||||
RPGMaker module, via :mod:`util.dazedwrap`) so English fits WOLF's message box.
|
||||
Wrapping is applied to the dialogue *body* only - a speaker's nameplate line (the
|
||||
``\n`` right after a name) is kept on its own line and never folded into the body.
|
||||
Configurable from the workflow (``.env``: ``wolfWrap`` / ``wolfWidth``); a width of
|
||||
0 (or ``wolfWrap=false``) writes the model output back verbatim.
|
||||
|
||||
Speakers: WolfDawn tags each line with ``speaker`` / ``speaker_src``. For the
|
||||
first-line formats (``literal_line1`` / ``literal_line1_lowconf``) the speaker
|
||||
|
|
@ -36,6 +42,7 @@ import traceback
|
|||
from colorama import Fore
|
||||
from tqdm import tqdm
|
||||
|
||||
import util.dazedwrap as dazedwrap
|
||||
from util.paths import PROMPT_PATH, VOCAB_PATH
|
||||
from util.translation import (
|
||||
TranslationConfig,
|
||||
|
|
@ -68,6 +75,17 @@ LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
|
|||
# configurable from the workflow (data/wolf_speakers.json).
|
||||
SPEAKER_CONFIG = wolf_speakers.load_config()
|
||||
|
||||
# Text wrapping: rewrap translated dialogue to a character width the same way the
|
||||
# RPGMaker module does (util.dazedwrap), so English lines fit WOLF's message box.
|
||||
# Only the dialogue *body* is wrapped - the speaker's name line (the "\n" after a
|
||||
# nameplate) is kept on its own line and never merged into the body. Configurable
|
||||
# from the workflow (.env: wolfWrap / wolfWidth). Width <= 0 disables wrapping.
|
||||
WRAP = (os.getenv("wolfWrap", "true").strip().lower() == "true")
|
||||
try:
|
||||
WRAPWIDTH = int(os.getenv("wolfWidth") or 0)
|
||||
except ValueError:
|
||||
WRAPWIDTH = 0
|
||||
|
||||
# Pricing / batching from the configured model
|
||||
PRICING_CONFIG = getPricingConfig(MODEL)
|
||||
INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
|
||||
|
|
@ -167,6 +185,30 @@ def collectEntries(data):
|
|||
return entries
|
||||
|
||||
|
||||
def _wrap_body(body):
|
||||
"""Word-wrap a dialogue body to WRAPWIDTH (no-op when wrapping is disabled)."""
|
||||
if not WRAP or WRAPWIDTH <= 0 or not isinstance(body, str) or not body:
|
||||
return body
|
||||
return dazedwrap.wrapText(body, WRAPWIDTH)
|
||||
|
||||
|
||||
def _wrap_plain(text, is_firstline):
|
||||
"""Wrap a non-reshaped entry, protecting a nameplate first line if present.
|
||||
|
||||
``is_firstline`` is True for first-line-speaker formats that are turned off in
|
||||
the config: their model output is still ``Speaker\\nbody``, so line 1 (the
|
||||
name) is kept intact and only the body is wrapped. Any leading ``@<option>``
|
||||
window prefix is also preserved.
|
||||
"""
|
||||
if not WRAP or WRAPWIDTH <= 0 or not isinstance(text, str) or not text:
|
||||
return text
|
||||
prefix, rest = wolf_speakers.split_window_prefix(text)
|
||||
if is_firstline and "\n" in rest:
|
||||
name, body = rest.split("\n", 1)
|
||||
return prefix + name + "\n" + _wrap_body(body)
|
||||
return prefix + _wrap_body(rest)
|
||||
|
||||
|
||||
def parseDocument(data, filename):
|
||||
"""Translate every translatable leaf entry and return [data, tokens, error]."""
|
||||
global PBAR
|
||||
|
|
@ -190,17 +232,18 @@ def parseDocument(data, filename):
|
|||
# transport format. plans[i] carries what is needed to restore each
|
||||
# entry after translation.
|
||||
sources = []
|
||||
plans = [] # (entry, prefix, has_speaker)
|
||||
plans = [] # (entry, prefix, has_speaker, is_firstline)
|
||||
for entry in translatable:
|
||||
src = entry["source"]
|
||||
is_firstline = entry.get("speaker_src", "") in wolf_speakers.FIRSTLINE_SRCS
|
||||
split = wolf_speakers.split_source(src, entry.get("speaker_src", ""), SPEAKER_CONFIG)
|
||||
if split is not None:
|
||||
prefix, speaker, body = split
|
||||
sources.append(wolf_speakers.to_prefixed(speaker, body))
|
||||
plans.append((entry, prefix, True))
|
||||
plans.append((entry, prefix, True, is_firstline))
|
||||
else:
|
||||
sources.append(src)
|
||||
plans.append((entry, "", False))
|
||||
plans.append((entry, "", False, is_firstline))
|
||||
|
||||
try:
|
||||
response = translateAI(sources, [])
|
||||
|
|
@ -213,18 +256,21 @@ def parseDocument(data, filename):
|
|||
|
||||
# Write translations back (skip in estimate mode: translated == sources).
|
||||
if not ESTIMATE and isinstance(translated, list) and len(translated) == len(plans):
|
||||
for (entry, prefix, has_speaker), text in zip(plans, translated):
|
||||
for (entry, prefix, has_speaker, is_firstline), text in zip(plans, translated):
|
||||
if not isinstance(text, str):
|
||||
continue
|
||||
if has_speaker:
|
||||
speaker_en, body_en = wolf_speakers.parse_prefixed(text)
|
||||
if speaker_en is not None:
|
||||
entry["text"] = wolf_speakers.restore_source(prefix, speaker_en, body_en)
|
||||
# Wrap only the body; the name stays on its own line.
|
||||
entry["text"] = wolf_speakers.restore_source(
|
||||
prefix, speaker_en, _wrap_body(body_en)
|
||||
)
|
||||
else:
|
||||
# Model dropped the [Speaker]: prefix; keep its output as-is.
|
||||
entry["text"] = prefix + text
|
||||
entry["text"] = prefix + _wrap_body(text)
|
||||
else:
|
||||
entry["text"] = text
|
||||
entry["text"] = _wrap_plain(text, is_firstline)
|
||||
|
||||
return [data, totalTokens, None]
|
||||
|
||||
|
|
|
|||
|
|
@ -46,8 +46,10 @@ class _WolfTranslateHarness:
|
|||
|
||||
orig_t = wd.translateAI
|
||||
orig_estimate = wd.ESTIMATE
|
||||
orig_wrap = wd.WRAP
|
||||
wd.translateAI = translate
|
||||
wd.ESTIMATE = estimate
|
||||
wd.WRAP = False # keep write-back byte-faithful; wrapping tested separately
|
||||
try:
|
||||
data_copy = copy.deepcopy(data)
|
||||
result = wd.parseDocument(data_copy, filename)
|
||||
|
|
@ -55,6 +57,7 @@ class _WolfTranslateHarness:
|
|||
finally:
|
||||
wd.translateAI = orig_t
|
||||
wd.ESTIMATE = orig_estimate
|
||||
wd.WRAP = orig_wrap
|
||||
|
||||
|
||||
MAP_DOC = {
|
||||
|
|
@ -192,8 +195,10 @@ SPEAKER_MAP_DOC = {
|
|||
class _SpeakerHarness:
|
||||
"""Run parseDocument with the speaker-aware mock and a chosen speaker config."""
|
||||
|
||||
def __init__(self, config):
|
||||
def __init__(self, config, wrap=False, width=0):
|
||||
self.config = config
|
||||
self.wrap = wrap
|
||||
self.width = width
|
||||
self.captured = []
|
||||
|
||||
def run(self, data, filename="OP.mps.json"):
|
||||
|
|
@ -201,15 +206,17 @@ class _SpeakerHarness:
|
|||
self.captured.append(copy.deepcopy(text))
|
||||
return _mock_translate_speaker(text, history, history_ctx)
|
||||
|
||||
orig_t, orig_est, orig_cfg = wd.translateAI, wd.ESTIMATE, wd.SPEAKER_CONFIG
|
||||
orig = (wd.translateAI, wd.ESTIMATE, wd.SPEAKER_CONFIG, wd.WRAP, wd.WRAPWIDTH)
|
||||
wd.translateAI = translate
|
||||
wd.ESTIMATE = False
|
||||
wd.SPEAKER_CONFIG = self.config
|
||||
wd.WRAP = self.wrap
|
||||
wd.WRAPWIDTH = self.width
|
||||
try:
|
||||
result = wd.parseDocument(copy.deepcopy(data), filename)
|
||||
return result, self.captured
|
||||
finally:
|
||||
wd.translateAI, wd.ESTIMATE, wd.SPEAKER_CONFIG = orig_t, orig_est, orig_cfg
|
||||
(wd.translateAI, wd.ESTIMATE, wd.SPEAKER_CONFIG, wd.WRAP, wd.WRAPWIDTH) = orig
|
||||
|
||||
|
||||
class TestSpeakerReshaping(unittest.TestCase):
|
||||
|
|
@ -244,6 +251,55 @@ class TestSpeakerReshaping(unittest.TestCase):
|
|||
self.assertEqual(lines[0]["text"], "EN_市民\nおはよう\n元気？")
|
||||
|
||||
|
||||
class TestWrapping(unittest.TestCase):
|
||||
"""dazedwrap-style re-wrapping, with the speaker name kept on its own line."""
|
||||
|
||||
def setUp(self):
|
||||
self._orig = (wd.WRAP, wd.WRAPWIDTH)
|
||||
|
||||
def tearDown(self):
|
||||
wd.WRAP, wd.WRAPWIDTH = self._orig
|
||||
|
||||
def _set(self, enabled, width):
|
||||
wd.WRAP, wd.WRAPWIDTH = enabled, width
|
||||
|
||||
def test_wrap_body_respects_width_and_keeps_words(self):
|
||||
self._set(True, 10)
|
||||
out = wd._wrap_body("alpha beta gamma delta")
|
||||
self.assertGreater(out.count("\n"), 0) # actually wrapped
|
||||
for line in out.split("\n"):
|
||||
self.assertLessEqual(len(line), 10)
|
||||
self.assertEqual(out.replace("\n", " "), "alpha beta gamma delta")
|
||||
|
||||
def test_wrap_disabled_is_noop(self):
|
||||
self._set(False, 10)
|
||||
self.assertEqual(wd._wrap_body("alpha beta gamma delta"), "alpha beta gamma delta")
|
||||
|
||||
def test_zero_width_is_noop(self):
|
||||
self._set(True, 0)
|
||||
self.assertEqual(wd._wrap_body("alpha beta gamma delta"), "alpha beta gamma delta")
|
||||
|
||||
def test_wrap_plain_preserves_nameplate_line(self):
|
||||
self._set(True, 12)
|
||||
out = wd._wrap_plain("Name\nalpha beta gamma delta epsilon", is_firstline=True)
|
||||
self.assertEqual(out.split("\n", 1)[0], "Name") # name never merged into body
|
||||
for line in out.split("\n")[1:]:
|
||||
self.assertLessEqual(len(line), 12)
|
||||
|
||||
def test_wrap_plain_preserves_window_prefix(self):
|
||||
self._set(True, 12)
|
||||
out = wd._wrap_plain("@1\nName\nalpha beta gamma delta", is_firstline=True)
|
||||
self.assertTrue(out.startswith("@1\nName\n"))
|
||||
|
||||
def test_speaker_body_wrapped_but_name_kept(self):
|
||||
# Integration: even at a tiny width the (translated) name stays on line 1.
|
||||
cfg = {"literal_line1": True, "literal_line1_lowconf": True}
|
||||
(data, _t, err), _c = _SpeakerHarness(cfg, wrap=True, width=6).run(SPEAKER_MAP_DOC)
|
||||
self.assertIsNone(err)
|
||||
line0 = data["scenes"][0]["lines"][0]["text"]
|
||||
self.assertEqual(line0.split("\n", 1)[0], "EN_市民")
|
||||
|
||||
|
||||
class TestOpenFiles(unittest.TestCase):
|
||||
def test_rejects_unknown_kind(self):
|
||||
with tempfile.TemporaryDirectory() as td:
|
||||
|
|
|
|||
|
|
@ -401,6 +401,21 @@ def is_firstline_enabled(speaker_src: str, config: dict | None = None) -> bool:
|
|||
return bool(cfg.get(speaker_src, DEFAULT_CONFIG.get(speaker_src, False)))
|
||||
|
||||
|
||||
def split_window_prefix(text: str) -> tuple[str, str]:
|
||||
"""Split a leading ``@<option>\\n`` window prefix off *text*.
|
||||
|
||||
Returns ``(prefix, rest)`` where ``prefix`` is ``""`` when there is no such
|
||||
prefix. The prefix is preserved verbatim so wrapping/reshaping stays
|
||||
byte-faithful for WolfDawn's inject drift-guard.
|
||||
"""
|
||||
if not isinstance(text, str):
|
||||
return "", text
|
||||
m = _WINDOW_PREFIX_RE.match(text)
|
||||
if not m:
|
||||
return "", text
|
||||
return m.group(0), text[m.end():]
|
||||
|
||||
|
||||
def split_source(source: str, speaker_src: str, config: dict | None = None):
|
||||
"""Split a first-line-speaker source into (prefix, speaker, body).
|
||||
|
||||
|
|
@ -409,12 +424,7 @@ def split_source(source: str, speaker_src: str, config: dict | None = None):
|
|||
"""
|
||||
if not isinstance(source, str) or not is_firstline_enabled(speaker_src, config):
|
||||
return None
|
||||
prefix = ""
|
||||
rest = source
|
||||
m = _WINDOW_PREFIX_RE.match(source)
|
||||
if m:
|
||||
prefix = m.group(0)
|
||||
rest = source[m.end():]
|
||||
prefix, rest = split_window_prefix(source)
|
||||
if "\n" not in rest:
|
||||
return None
|
||||
line1, body = rest.split("\n", 1)
|
||||
|
|
|
|||
Loading…
Reference in a new issue