dazed/DazedTL
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Surely cleaning up all this mess wont break anything. surely

Browse source
This commit is contained in:
DazedAnon 2026-06-15 17:34:32 -05:00
parent 563299ba21
commit 4dd9d2bbd4
38 changed files with 16162 additions and 15976 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

64
.gitignore vendored
View file

@ -1,32 +1,32 @@
.env .env
*.tmp *.tmp
*.json *.json
*.js *.js
*.yaml *.yaml
*.yml *.yml
*.txt *.txt
*.log *.log
logs/ logs/
*.rpy *.rpy
*.csv *.csv
*.ks *.ks
*.cid *.cid
*.png *.png
*.script *.script
__pycache__ __pycache__
!assets/*.png !assets/*.png
!.pre-commit-config.yaml !.pre-commit-config.yaml
!prompt.txt !data/prompt.txt
!vocab_base.txt !data/vocab_base.txt
!requirements.txt !requirements.txt
!util/tl_inspector/TLInspector.js !util/tl_inspector/TLInspector.js
!util/forge/Forge_MZ.js !util/forge/Forge_MZ.js
!util/forge/Forge_MV.js !util/forge/Forge_MV.js
!util/forge/upstream/Forge_MZ.js !util/forge/upstream/Forge_MZ.js
!util/forge/upstream/Forge_MV.js !util/forge/upstream/Forge_MV.js
util/forge/.forge_version.json util/forge/.forge_version.json
util/ace/*.exe util/ace/*.exe
util/ace/.tools_version.json util/ace/.tools_version.json
!util/ace/offline/*.exe !util/ace/offline/*.exe

2
.pre-commit-config.yaml
View file

@ -3,6 +3,6 @@
'hooks': 'hooks':
- 'id': 'set-defaults' - 'id': 'set-defaults'
'name': 'Set Defaults in rpgmakermvmz.py' 'name': 'Set Defaults in rpgmakermvmz.py'
'entry': 'python set_defaults.py' 'entry': 'python -m util.defaults'
'language': 'python' 'language': 'python'
'files': '^modules/rpgmakermvmz.py$' 'files': '^modules/rpgmakermvmz.py$'

10
DazedMTLTool.desktop Normal file
View file

@ -0,0 +1,10 @@
[Desktop Entry]
Type=Application
Name=DazedMTLTool
GenericName=Translation Tool
Comment=AI translation tool for visual novels and RPG games
Exec=bash -c 'exec "$(cd "$(dirname "%k")" && pwd)/scripts/launch.sh"'
Icon=%k/assets/icon.png
Terminal=false
Categories=Development;Utility;
StartupWMClass=DazedMTLTool

23
README.md
View file

@ -124,12 +124,11 @@ This means Python wasn't added to your PATH. You have two options:
### 3. Launch the GUI ### 3. Launch the GUI
**Double-click `START.bat`**. It will: **Windows:** Double-click `START.bat`. It will create a virtual environment, install dependencies, and open the GUI.
- Create a virtual environment automatically.
- Install all dependencies.
- Launch the GUI.
That's it! From now on, just double-click `START.bat` to open the tool. **Linux/macOS:** Run `./START.sh`, or double-click `DazedMTLTool.desktop` (choose **Allow Launching** when your file manager asks). From then on, either method works.
That's it! Use the same launcher each time you want to open the tool.
--- ---
@ -160,10 +159,10 @@ Specialized tabs with extra options for those specific engines.
## Vocab & Prompt ## Vocab & Prompt
### vocab.txt ### data/vocab.txt
This file gives the AI context about your game — character names, genders, recurring terms, etc. The better your vocab file, the more consistent the translation. This file gives the AI context about your game — character names, genders, recurring terms, etc. The better your vocab file, the more consistent the translation.
Open `vocab.txt` (or copy `vocab.txt.example` to `vocab.txt` if it doesn't exist) and add entries like: On first run, `data/vocab.txt` is created automatically from `data/vocab_base.txt` if it does not exist yet. Add entries like:
```plaintext ```plaintext
# Game Characters # Game Characters
@ -176,14 +175,14 @@ Format: Japanese name, English name in parentheses, then gender.
> **Note:** A very large vocab file can increase API costs and potentially reduce quality. Focus on the most important characters and terms. > **Note:** A very large vocab file can increase API costs and potentially reduce quality. Focus on the most important characters and terms.
### prompt.txt ### data/prompt.txt
This is the system prompt sent to the AI. A default `prompt.txt` is included and works well for most games. You generally don't need to edit it unless you want to customize the translation style. This is the system prompt sent to the AI. A default `data/prompt.txt` is included and works well for most games. You generally don't need to edit it unless you want to customize the translation style.
--- ---
## Tips ## Tips
- **Check `log/translations.txt`** after a run to see what was translated. You can copy useful terms from it into `vocab.txt` for consistency in future runs. - **Check `log/translations.txt`** after a run to see what was translated. You can copy useful terms from it into `data/vocab.txt` for consistency in future runs.
- **Start small** — Translate a few files first to make sure the output looks good before doing the whole game. - **Start small** — Translate a few files first to make sure the output looks good before doing the whole game.
- **Wordwrap** — If text overflows or looks awkward in-game, adjust the `width` setting in `.env` or the Config tab. `60` is a good default for most RPG Maker games. - **Wordwrap** — If text overflows or looks awkward in-game, adjust the `width` setting in `.env` or the Config tab. `60` is a good default for most RPG Maker games.
- **Version control** — Using [Git](https://git-scm.com/) with the game folder is highly recommended. It lets you track every change the translation makes, compare with original files, and roll back if needed. - **Version control** — Using [Git](https://git-scm.com/) with the game folder is highly recommended. It lets you track every change the translation makes, compare with original files, and roll back if needed.
@ -300,8 +299,8 @@ Here's the recommended step-by-step process for translating an RPG Maker MV/MZ g
| Step | Action | | Step | Action |
|------|--------| |------|--------|
| **1** | **Parse speakers → vocab.txt** — Use the Parse Speakers feature to pull character names from the game files into `vocab.txt`. | | **1** | **Parse speakers → vocab** — Use the Parse Speakers feature to pull character names from the game files into `data/vocab.txt`. |
| **2** | **Identify speaker genders** — Figure out which characters are male/female and update `vocab.txt` accordingly. This helps the AI use correct pronouns. | | **2** | **Identify speaker genders** — Figure out which characters are male/female and update `data/vocab.txt` accordingly. This helps the AI use correct pronouns. |
| **3** | **Translate Actors.json, MapInfos.json** — These are small files with character and map names. Good to do first. | | **3** | **Translate Actors.json, MapInfos.json** — These are small files with character and map names. Good to do first. |
| **4** | **Translate Items, System, Weapons, etc.** — All the data files that aren't maps or events. Place them in `files/`, translate, then copy results back. | | **4** | **Translate Items, System, Weapons, etc.** — All the data files that aren't maps or events. Place them in `files/`, translate, then copy results back. |
| **5** | **Find speaker names** — Enable CODE 101 (Speakers), check for bracketed names, or use the "First Line = Speaker" option to capture speaker names properly. | | **5** | **Find speaker names** — Enable CODE 101 (Speakers), check for bracketed names, or use the "First Line = Speaker" option to capture speaker names properly. |

25
START.bat
View file

@ -184,28 +184,29 @@ echo Launching DazedMTLTool GUI...
echo ========================================== echo ==========================================
echo. echo.
:: Ensure vocab.txt exists (create from example if available) :: Ensure data/vocab.txt exists (create from vocab_base.txt if available)
if not exist "vocab.txt" ( if not exist "data\\vocab.txt" (
if exist "vocab.txt.example" ( if not exist "data" mkdir data
echo vocab.txt not found - creating from vocab.txt.example... if exist "data\\vocab_base.txt" (
copy /Y "vocab.txt.example" "vocab.txt" >nul 2>&1 echo data\vocab.txt not found - creating from data\vocab_base.txt...
copy /Y "data\vocab_base.txt" "data\vocab.txt" >nul 2>&1
if errorlevel 1 ( if errorlevel 1 (
echo ERROR: Failed to copy vocab.txt.example to vocab.txt. echo ERROR: Failed to copy data\vocab_base.txt to data\vocab.txt.
) else ( ) else (
echo Created vocab.txt from vocab.txt.example echo Created data\vocab.txt from data\vocab_base.txt
) )
) else ( ) else (
echo vocab.txt and vocab.txt.example not found - creating empty vocab.txt to avoid import errors... echo data\vocab.txt not found - creating empty file to avoid import errors...
type NUL > "vocab.txt" type NUL > "data\vocab.txt"
if errorlevel 1 ( if errorlevel 1 (
echo ERROR: Failed to create empty vocab.txt. echo ERROR: Failed to create empty data\vocab.txt.
) else ( ) else (
echo Created empty vocab.txt echo Created empty data\vocab.txt
) )
) )
) )
python start_gui.py python scripts/start_gui.py
:: Check if GUI launched successfully :: Check if GUI launched successfully
if errorlevel 1 ( if errorlevel 1 (

23
START.sh
View file

@ -127,23 +127,24 @@ activate_venv() {
} }
ensure_vocab_file() { ensure_vocab_file() {
if [[ -f "vocab.txt" ]]; then mkdir -p data
if [[ -f "data/vocab.txt" ]]; then
return 0 return 0
fi fi
if [[ -f "vocab.txt.example" ]]; then if [[ -f "data/vocab_base.txt" ]]; then
echo "vocab.txt not found - creating from vocab.txt.example..." echo "data/vocab.txt not found - creating from data/vocab_base.txt..."
if cp "vocab.txt.example" "vocab.txt"; then if cp "data/vocab_base.txt" "data/vocab.txt"; then
echo "Created vocab.txt from vocab.txt.example" echo "Created data/vocab.txt from data/vocab_base.txt"
else else
echo "ERROR: Failed to copy vocab.txt.example to vocab.txt." echo "ERROR: Failed to copy data/vocab_base.txt to data/vocab.txt."
fi fi
else else
echo "vocab.txt and vocab.txt.example not found - creating empty vocab.txt to avoid import errors..." echo "data/vocab.txt not found - creating empty file to avoid import errors..."
if : > "vocab.txt"; then if : > "data/vocab.txt"; then
echo "Created empty vocab.txt" echo "Created empty data/vocab.txt"
else else
echo "ERROR: Failed to create empty vocab.txt." echo "ERROR: Failed to create empty data/vocab.txt."
fi fi
fi fi
} }
@ -227,7 +228,7 @@ echo
ensure_vocab_file ensure_vocab_file
if ! python start_gui.py; then if ! exec python "$SCRIPT_DIR/scripts/start_gui.py"; then
echo echo
echo "ERROR: Failed to launch GUI." echo "ERROR: Failed to launch GUI."
echo "Check the error messages above." echo "Check the error messages above."

0
prompt.txt → data/prompt.txt
View file

0
vocab_base.txt → data/vocab_base.txt
View file

29
gui/main.py
View file

@ -19,20 +19,16 @@ from PyQt5.QtWidgets import (
QTextEdit, QSplitter, QGroupBox, QStatusBar, QStackedWidget, QToolButton, QTextEdit, QSplitter, QGroupBox, QStatusBar, QStackedWidget, QToolButton,
QDialog QDialog
) )
from PyQt5.QtCore import Qt, QThread, pyqtSignal, QTimer, QSettings from PyQt5.QtCore import Qt, QThread, pyqtSignal, QTimer, QSettings, QCoreApplication
from PyQt5.QtGui import QIcon, QFont, QPixmap, QScreen from PyQt5.QtGui import QIcon, QFont, QPixmap, QScreen, QGuiApplication
def _project_root() -> Path: from util.paths import PROJECT_ROOT, ICON_PATH, LAST_UPDATE_SHA_PATH
"""Directory containing start_gui.py / requirements.txt (parent of gui/)."""
return Path(__file__).resolve().parent.parent
def load_application_icon() -> QIcon: def load_application_icon() -> QIcon:
"""Prefer ICO on Windows; PNG is fine for all platforms. Empty if missing.""" """Prefer ICO on Windows; PNG is fine for all platforms. Empty if missing."""
root = _project_root() for path in (PROJECT_ROOT / "assets/icon.ico", ICON_PATH):
for rel in ("assets/icon.ico", "assets/icon.png"):
path = root / rel
if path.is_file(): if path.is_file():
icon = QIcon(str(path)) icon = QIcon(str(path))
if not icon.isNull(): if not icon.isNull():
@ -126,14 +122,13 @@ class UpdateThread(QThread):
REPO_USER = "DazedAnon" REPO_USER = "DazedAnon"
REPO_NAME = "DazedMTLTool" REPO_NAME = "DazedMTLTool"
REPO_BRANCH = "main" REPO_BRANCH = "main"
SHA_FILE = "last_update_sha.txt" SHA_FILE = str(LAST_UPDATE_SHA_PATH)
# Paths (relative, top-level) that should never be touched during update # Paths (relative, top-level) that should never be touched during update
PROTECTED = {".env", "venv", "log", "files", "translated", PROTECTED = {".env", "venv", "log", "files", "translated", "data"}
"vocab.txt", "last_update_sha.txt"}
# GitLab zip archives do not preserve Unix execute bits; restore after apply. # GitLab zip archives do not preserve Unix execute bits; restore after apply.
EXECUTABLE_SUFFIXES = {".sh"} EXECUTABLE_SUFFIXES = {".sh", ".desktop"}
progress = pyqtSignal(str) # status message progress = pyqtSignal(str) # status message
finished = pyqtSignal(bool, str) # (success, message) finished = pyqtSignal(bool, str) # (success, message)
@ -902,6 +897,14 @@ def main():
# Set high DPI scale factor policy # Set high DPI scale factor policy
if hasattr(QApplication, 'setHighDpiScaleFactorRoundingPolicy'): if hasattr(QApplication, 'setHighDpiScaleFactorRoundingPolicy'):
QApplication.setHighDpiScaleFactorRoundingPolicy(Qt.HighDpiScaleFactorRoundingPolicy.PassThrough) QApplication.setHighDpiScaleFactorRoundingPolicy(Qt.HighDpiScaleFactorRoundingPolicy.PassThrough)
if sys.platform.startswith("linux"):
from util.linux_desktop import ensure_linux_desktop_entry
ensure_linux_desktop_entry()
QGuiApplication.setDesktopFileName("DazedMTLTool")
QCoreApplication.setOrganizationName("DazedTranslations")
QCoreApplication.setApplicationName("DazedMTLTool")
app = QApplication(sys.argv) app = QApplication(sys.argv)
@ -934,9 +937,7 @@ def main():
return 1 return 1
# Set application properties # Set application properties
app.setApplicationName("DazedMTLTool")
app.setApplicationVersion("1.0") app.setApplicationVersion("1.0")
app.setOrganizationName("DazedTranslations")
# Apply dark theme with cleaner, more compact styling # Apply dark theme with cleaner, more compact styling
app.setStyleSheet(""" app.setStyleSheet("""

3
gui/rpgmaker_tab.py
View file

@ -19,8 +19,7 @@ except ImportError:
ConfigIntegration = None ConfigIntegration = None
try: try:
# Prefer importing the project's canonical defaults if available # Prefer importing the project's canonical defaults if available
import set_defaults from util.defaults import DEFAULTS as CANONICAL_DEFAULTS
CANONICAL_DEFAULTS = getattr(set_defaults, 'DEFAULTS', None)
except Exception: except Exception:
CANONICAL_DEFAULTS = None CANONICAL_DEFAULTS = None

3
gui/srpg_tab.py
View file

@ -12,8 +12,7 @@ import re
from gui.platform_glyph import platform_glyph from gui.platform_glyph import platform_glyph
try: try:
import set_defaults from util.defaults import DEFAULTS as CANONICAL_DEFAULTS
CANONICAL_DEFAULTS = getattr(set_defaults, 'DEFAULTS', None)
except Exception: except Exception:
CANONICAL_DEFAULTS = None CANONICAL_DEFAULTS = None

3
gui/wolf_tab.py
View file

@ -11,8 +11,7 @@ from PyQt5.QtCore import pyqtSignal
import re import re
from gui.platform_glyph import platform_glyph from gui.platform_glyph import platform_glyph
try: try:
import set_defaults from util.defaults import DEFAULTS as CANONICAL_DEFAULTS
CANONICAL_DEFAULTS = getattr(set_defaults, 'DEFAULTS', None)
except Exception: except Exception:
CANONICAL_DEFAULTS = None CANONICAL_DEFAULTS = None

12
gui/workflow_tab.py
View file

@ -22,6 +22,8 @@ import sys
import threading import threading
from pathlib import Path from pathlib import Path
from util.paths import VOCAB_BASE_PATH, VOCAB_PATH
import jsbeautifier import jsbeautifier
from PyQt5.QtCore import Qt, QEvent, QSettings, QThread, QTimer, pyqtSignal from PyQt5.QtCore import Qt, QEvent, QSettings, QThread, QTimer, pyqtSignal
@ -3576,7 +3578,7 @@ class WorkflowTab(QWidget):
def _read_vocab_speakers(self) -> list[tuple[str, str]]: def _read_vocab_speakers(self) -> list[tuple[str, str]]:
"""Parse the '# Speakers' section from vocab.txt and return (orig, tl) pairs.""" """Parse the '# Speakers' section from vocab.txt and return (orig, tl) pairs."""
vocab_path = Path("vocab.txt") vocab_path = VOCAB_PATH
if not vocab_path.exists(): if not vocab_path.exists():
return [] return []
try: try:
@ -3606,7 +3608,7 @@ class WorkflowTab(QWidget):
return results return results
def _reload_vocab(self): def _reload_vocab(self):
vocab_path = Path("vocab.txt") vocab_path = VOCAB_PATH
try: try:
if vocab_path.exists(): if vocab_path.exists():
text = vocab_path.read_text(encoding="utf-8") text = vocab_path.read_text(encoding="utf-8")
@ -3623,10 +3625,10 @@ class WorkflowTab(QWidget):
def _save_vocab(self): def _save_vocab(self):
try: try:
game_text = self.vocab_editor.toPlainText().rstrip("\n") game_text = self.vocab_editor.toPlainText().rstrip("\n")
base_path = Path("vocab_base.txt") base_path = VOCAB_BASE_PATH
base_text = base_path.read_text(encoding="utf-8") if base_path.exists() else "" base_text = base_path.read_text(encoding="utf-8") if base_path.exists() else ""
combined = game_text + "\n\n" + self._BASE_SEPARATOR + base_text combined = game_text + "\n\n" + self._BASE_SEPARATOR + base_text
Path("vocab.txt").write_text(combined, encoding="utf-8") VOCAB_PATH.write_text(combined, encoding="utf-8")
self._log("✅ vocab.txt saved (base terms from vocab_base.txt appended).") self._log("✅ vocab.txt saved (base terms from vocab_base.txt appended).")
except Exception as exc: except Exception as exc:
self._log(f"❌ Could not save vocab.txt: {exc}") self._log(f"❌ Could not save vocab.txt: {exc}")
@ -3795,7 +3797,7 @@ class WorkflowTab(QWidget):
self._log("⚠ No game folder set. Complete Step 0 first.") self._log("⚠ No game folder set. Complete Step 0 first.")
return return
src = Path("vocab.txt") src = VOCAB_PATH
if not src.exists(): if not src.exists():
self._log("⚠ vocab.txt not found — save it in Step 3 first.") self._log("⚠ vocab.txt not found — save it in Step 3 first.")
return return

430
modules/aquedi4.py
View file

@ -1,215 +1,217 @@
# modules/aquedi4.py # modules/aquedi4.py
# Libraries # Libraries
import json import json
import os import os
import re import re
import util.dazedwrap as dazedwrap import util.dazedwrap as dazedwrap
import threading import threading
import time import time
import traceback import traceback
import tiktoken import tiktoken
from pathlib import Path from pathlib import Path
from colorama import Fore from colorama import Fore
from dotenv import load_dotenv from dotenv import load_dotenv
from retry import retry from retry import retry
from tqdm import tqdm from tqdm import tqdm
from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
# from modules.json import translateSimpleKeyValueJSON # from modules.json import translateSimpleKeyValueJSON
import tempfile import tempfile
# Boilerplate Globals # Boilerplate Globals
MODEL = os.getenv("model") MODEL = os.getenv("model")
TIMEOUT = int(os.getenv("timeout")) TIMEOUT = int(os.getenv("timeout"))
LANGUAGE = os.getenv("language").capitalize() LANGUAGE = os.getenv("language").capitalize()
PROMPT = Path("prompt.txt").read_text(encoding="utf-8") from util.paths import PROMPT_PATH, VOCAB_PATH
VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
LOCK = threading.Lock() PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
ESTIMATE = "" VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
TOKENS = [0, 0] LOCK = threading.Lock()
MISMATCH = [] ESTIMATE = ""
FILENAME = None TOKENS = [0, 0]
PBAR = None MISMATCH = []
BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}" FILENAME = None
PBAR = None
# Flag to control line break replacement BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
REPLACE_LINEBREAKS_WITH_PIPES = True
# Flag to control line break replacement
LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+" REPLACE_LINEBREAKS_WITH_PIPES = True
TRANSLATION_CONFIG = TranslationConfig(
model=MODEL, LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
language=LANGUAGE, TRANSLATION_CONFIG = TranslationConfig(
prompt=PROMPT, model=MODEL,
vocab=VOCAB, language=LANGUAGE,
langRegex=LANGREGEX, prompt=PROMPT,
batchSize=int(os.getenv("batchsize", 10)), vocab=VOCAB,
maxHistory=10, langRegex=LANGREGEX,
estimateMode=False batchSize=int(os.getenv("batchsize", 10)),
) maxHistory=10,
# End Boilerplate estimateMode=False
)
def handleAquedi4(filename, estimate): # End Boilerplate
"""Main handler for Aquedi4 JSON files."""
global ESTIMATE, TOKENS, FILENAME def handleAquedi4(filename, estimate):
ESTIMATE = estimate """Main handler for Aquedi4 JSON files."""
FILENAME = filename global ESTIMATE, TOKENS, FILENAME
ESTIMATE = estimate
try: FILENAME = filename
start = time.time()
translatedData = openFiles(filename) try:
start = time.time()
if not estimate: translatedData = openFiles(filename)
# Write final result after translation is complete
with open("translated/" + filename, "w", encoding="utf-8", newline="\n") as outFile: if not estimate:
json.dump(translatedData[0], outFile, ensure_ascii=False, indent=4) # Write final result after translation is complete
with open("translated/" + filename, "w", encoding="utf-8", newline="\n") as outFile:
end = time.time() json.dump(translatedData[0], outFile, ensure_ascii=False, indent=4)
tqdm.write(getResultString(translatedData, end - start, filename))
with LOCK: end = time.time()
TOKENS[0] += translatedData[1][0] tqdm.write(getResultString(translatedData, end - start, filename))
TOKENS[1] += translatedData[1][1] with LOCK:
TOKENS[0] += translatedData[1][0]
except Exception as e: TOKENS[1] += translatedData[1][1]
traceback.print_exc()
return "Fail" except Exception as e:
traceback.print_exc()
return getResultString(["", TOKENS, None], end - start, "TOTAL") return "Fail"
return getResultString(["", TOKENS, None], end - start, "TOTAL")
def openFiles(filename):
"""Opens and parses the JSON file."""
with open("files/" + filename, "r", encoding="UTF-8-sig") as f: def openFiles(filename):
data = json.load(f) """Opens and parses the JSON file."""
translatedData = parseAquedi4JSON(data, filename) with open("files/" + filename, "r", encoding="UTF-8-sig") as f:
return translatedData data = json.load(f)
translatedData = parseAquedi4JSON(data, filename)
return translatedData
def parseAquedi4JSON(data, filename):
"""Parses and translates the simple key-value JSON."""
totalTokens = [0, 0] def parseAquedi4JSON(data, filename):
totalLines = 0 """Parses and translates the simple key-value JSON."""
totalLines = len(data) totalTokens = [0, 0]
global LOCK, PBAR totalLines = 0
totalLines = len(data)
stringList = [] global LOCK, PBAR
keyList = []
if isinstance(data, dict) and all(isinstance(v, str) for v in data.values()): stringList = []
for key, value in data.items(): keyList = []
if re.search(LANGREGEX, key): if isinstance(data, dict) and all(isinstance(v, str) for v in data.values()):
stringList.append(key) for key, value in data.items():
keyList.append(key) if re.search(LANGREGEX, key):
stringList.append(key)
with tqdm(total=len(stringList), bar_format=BAR_FORMAT, position=0, leave=False) as pbar: keyList.append(key)
pbar.desc = filename
PBAR = pbar with tqdm(total=len(stringList), bar_format=BAR_FORMAT, position=0, leave=False) as pbar:
pbar.desc = filename
# from modules.json import translateSimpleKeyValueJSON PBAR = pbar
try: # from modules.json import translateSimpleKeyValueJSON
if stringList:
result = translateSimpleKeyValueJSON(data, filename, stringList, keyList) try:
totalTokens[0] += result[0] if stringList:
totalTokens[1] += result[1] result = translateSimpleKeyValueJSON(data, filename, stringList, keyList)
else: totalTokens[0] += result[0]
pbar.write(f"Warning: {filename} is not in the expected simple key-value format. Skipping.") totalTokens[1] += result[1]
except Exception as e: else:
traceback.print_exc() pbar.write(f"Warning: {filename} is not in the expected simple key-value format. Skipping.")
return [data, totalTokens, e] except Exception as e:
traceback.print_exc()
return [data, totalTokens, None] return [data, totalTokens, e]
return [data, totalTokens, None]
def translateSimpleKeyValueJSON(data, filename, stringList, keyList):
"""
Translate simple key-value JSON format where keys contain Japanese text def translateSimpleKeyValueJSON(data, filename, stringList, keyList):
and values are placeholder strings like "TODO". """
Translate simple key-value JSON format where keys contain Japanese text
Format example: and values are placeholder strings like "TODO".
{
"\\r\\nスタビライザー": "TODO", Format example:
"\\r\\nプラグインなし": "TODO" {
} "\\r\\nスタビライザー": "TODO",
""" "\\r\\nプラグインなし": "TODO"
global LOCK, ESTIMATE, FILENAME, PBAR, MISMATCH }
tokens = [0, 0] """
global LOCK, ESTIMATE, FILENAME, PBAR, MISMATCH
preparedStringList = [] tokens = [0, 0]
if REPLACE_LINEBREAKS_WITH_PIPES:
for s in stringList: preparedStringList = []
preparedStringList.append(s.replace("\r\n", "|").replace("\n", "|")) if REPLACE_LINEBREAKS_WITH_PIPES:
else: for s in stringList:
preparedStringList = stringList preparedStringList.append(s.replace("\r\n", "|").replace("\n", "|"))
else:
# Translate all keys if any were found preparedStringList = stringList
if preparedStringList:
PBAR.total = len(preparedStringList) # Translate all keys if any were found
PBAR.refresh() if preparedStringList:
PBAR.total = len(preparedStringList)
response = translateAI(preparedStringList, "Reply with the English Translation") PBAR.refresh()
tokens[0] += response[1][0]
tokens[1] += response[1][1] response = translateAI(preparedStringList, "Reply with the English Translation")
translatedList = response[0] tokens[0] += response[1][0]
tokens[1] += response[1][1]
# Check for mismatch translatedList = response[0]
if len(preparedStringList) != len(translatedList):
with LOCK: # Check for mismatch
if FILENAME not in MISMATCH: if len(preparedStringList) != len(translatedList):
MISMATCH.append(FILENAME) with LOCK:
if FILENAME not in MISMATCH:
# Pass 2: Update the values with translations MISMATCH.append(FILENAME)
for i, original_key in enumerate(keyList):
if i < len(translatedList): # Pass 2: Update the values with translations
translatedText = translatedList[i] for i, original_key in enumerate(keyList):
if i < len(translatedList):
if REPLACE_LINEBREAKS_WITH_PIPES: translatedText = translatedList[i]
translatedText = translatedText.replace("|", "\r\n")
# Set the translated text as the value if REPLACE_LINEBREAKS_WITH_PIPES:
data[original_key] = translatedText translatedText = translatedText.replace("|", "\r\n")
# Set the translated text as the value
# Save progress after each translation data[original_key] = translatedText
save_progress_json(data, filename)
# Save progress after each translation
return tokens save_progress_json(data, filename)
def save_progress_json(data, filename): return tokens
"""Save current JSON translation progress."""
try: def save_progress_json(data, filename):
if ESTIMATE: return """Save current JSON translation progress."""
os.makedirs("translated", exist_ok=True) try:
tmp_path = os.path.join("translated", f"{filename}.tmp") if ESTIMATE: return
final_path = os.path.join("translated", filename) os.makedirs("translated", exist_ok=True)
with open(tmp_path, "w", encoding="utf-8", newline="\n") as outFile: tmp_path = os.path.join("translated", f"{filename}.tmp")
json.dump(data, outFile, ensure_ascii=False, indent=4) final_path = os.path.join("translated", filename)
os.replace(tmp_path, final_path) with open(tmp_path, "w", encoding="utf-8", newline="\n") as outFile:
except Exception: json.dump(data, outFile, ensure_ascii=False, indent=4)
traceback.print_exc() os.replace(tmp_path, final_path)
except Exception:
traceback.print_exc()
def getResultString(translatedData, translationTime, filename):
cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
totalTokenstring = ( def getResultString(translatedData, translationTime, filename):
Fore.YELLOW + f"[Input: {translatedData[1][0]}]" cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
f"[Output: {translatedData[1][1]}]" totalTokenstring = (
f"[Cost: ${cost:,.4f}]" Fore.YELLOW + f"[Input: {translatedData[1][0]}]"
) f"[Output: {translatedData[1][1]}]"
timeString = Fore.BLUE + f"[{round(translationTime, 1)}s]" f"[Cost: ${cost:,.4f}]"
)
if translatedData[2] is None: timeString = Fore.BLUE + f"[{round(translationTime, 1)}s]"
return f"{filename}: {totalTokenstring}{timeString}" + Fore.GREEN + " \u2713 " + Fore.RESET
else: if translatedData[2] is None:
errorString = str(translatedData[2]) + Fore.RED return f"{filename}: {totalTokenstring}{timeString}" + Fore.GREEN + " \u2713 " + Fore.RESET
return f"{filename}: {totalTokenstring}{timeString}" + Fore.RED + " \u2717 " + errorString + Fore.RESET else:
errorString = str(translatedData[2]) + Fore.RED
def translateAI(text, history, history_ctx=None): return f"{filename}: {totalTokenstring}{timeString}" + Fore.RED + " \u2717 " + errorString + Fore.RESET
"""Legacy wrapper for the shared translation utility."""
global PBAR, MISMATCH, FILENAME def translateAI(text, history, history_ctx=None):
TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE) """Legacy wrapper for the shared translation utility."""
return sharedtranslateAI( global PBAR, MISMATCH, FILENAME
text=text, TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
history=history, return sharedtranslateAI(
config=TRANSLATION_CONFIG, text=text,
filename=FILENAME, history=history,
pbar=PBAR, config=TRANSLATION_CONFIG,
lock=LOCK, filename=FILENAME,
mismatchList=MISMATCH pbar=PBAR,
lock=LOCK,
mismatchList=MISMATCH
) )

918
modules/csv.py
View file

@ -1,458 +1,460 @@
# Libraries # Libraries
import json import json
import os import os
import re import re
import util.dazedwrap as dazedwrap import util.dazedwrap as dazedwrap
import threading import threading
import time import time
import traceback import traceback
import tiktoken import tiktoken
import csv import csv
from pathlib import Path from pathlib import Path
from colorama import Fore from colorama import Fore
from dotenv import load_dotenv from dotenv import load_dotenv
from retry import retry from retry import retry
from tqdm import tqdm from tqdm import tqdm
from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
import tempfile import tempfile
# Globals # Globals
MODEL = os.getenv("model") MODEL = os.getenv("model")
TIMEOUT = int(os.getenv("timeout")) TIMEOUT = int(os.getenv("timeout"))
LANGUAGE = os.getenv("language").capitalize() LANGUAGE = os.getenv("language").capitalize()
PROMPT = Path("prompt.txt").read_text(encoding="utf-8") from util.paths import PROMPT_PATH, VOCAB_PATH
VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
LOCK = threading.Lock() PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
WIDTH = int(os.getenv("width")) VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
LISTWIDTH = int(os.getenv("listWidth")) LOCK = threading.Lock()
NOTEWIDTH = int(os.getenv("noteWidth")) WIDTH = int(os.getenv("width"))
MAXHISTORY = 10 LISTWIDTH = int(os.getenv("listWidth"))
ESTIMATE = "" NOTEWIDTH = int(os.getenv("noteWidth"))
TOKENS = [0, 0] MAXHISTORY = 10
NAMESLIST = [] ESTIMATE = ""
NAMES = False # Output a list of all the character names found TOKENS = [0, 0]
BRFLAG = False # If the game uses <br> instead NAMESLIST = []
FIXTEXTWRAP = True # Overwrites textwrap NAMES = False # Output a list of all the character names found
IGNORETLTEXT = True # Ignores all translated text. BRFLAG = False # If the game uses <br> instead
MISMATCH = [] # Lists files that thdata a mismatch error (Length of GPT list response is wrong) FIXTEXTWRAP = True # Overwrites textwrap
FILENAME = None IGNORETLTEXT = True # Ignores all translated text.
BRACKETNAMES = False MISMATCH = [] # Lists files that thdata a mismatch error (Length of GPT list response is wrong)
FILENAME = None
# CSV Configuration Settings (configurable via GUI) BRACKETNAMES = False
CSV_DELIMITER = " " # CSV delimiter character (comma, semicolon, tab)
SOURCE_COLUMN = 2 # Which column has the source text to translate (0-indexed) # CSV Configuration Settings (configurable via GUI)
TARGET_COLUMN = 3 # Which column to write translations to CSV_DELIMITER = " " # CSV delimiter character (comma, semicolon, tab)
SPEAKER_COLUMN = 1 # Which column has speaker names (-1 = none) SOURCE_COLUMN = 2 # Which column has the source text to translate (0-indexed)
SKIP_HEADER_ROW = False # Skip the first row (header) TARGET_COLUMN = 3 # Which column to write translations to
USE_TARGET_IF_NOT_EMPTY = False # Use target column text if not empty (T++ style) SPEAKER_COLUMN = 1 # Which column has speaker names (-1 = none)
WRITE_TO_NEXT_COLUMN = False # Write to column after target instead of overwriting SKIP_HEADER_ROW = False # Skip the first row (header)
PARSE_NAME_TAGS = False # Parse :name[] tags in text USE_TARGET_IF_NOT_EMPTY = False # Use target column text if not empty (T++ style)
PARSE_M_MARKERS = False # Parse \M markers in text WRITE_TO_NEXT_COLUMN = False # Write to column after target instead of overwriting
REMOVE_FURIGANA = True # Remove furigana annotations ＜＝＞ PARSE_NAME_TAGS = False # Parse :name[] tags in text
SKIP_COMMENT_ROWS = False # Skip rows starting with 'comment' PARSE_M_MARKERS = False # Parse \M markers in text
REMOVE_FURIGANA = True # Remove furigana annotations ＜＝＞
# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex SKIP_COMMENT_ROWS = False # Skip rows starting with 'comment'
LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
# Get pricing configuration based on the model LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
PRICING_CONFIG = getPricingConfig(MODEL)
INPUTAPICOST = PRICING_CONFIG["inputAPICost"] # Get pricing configuration based on the model
OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"] PRICING_CONFIG = getPricingConfig(MODEL)
BATCHSIZE = PRICING_CONFIG["batchSize"] INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"] OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
BATCHSIZE = PRICING_CONFIG["batchSize"]
# tqdm Globals FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
POSITION = 0 # tqdm Globals
LEAVE = False BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
PBAR = None POSITION = 0
ENCODING = "utf8" LEAVE = False
PBAR = None
# Initialize Translation Config ENCODING = "utf8"
TRANSLATION_CONFIG = TranslationConfig(
model=MODEL, # Initialize Translation Config
language=LANGUAGE, TRANSLATION_CONFIG = TranslationConfig(
prompt=PROMPT, model=MODEL,
vocab=VOCAB, language=LANGUAGE,
langRegex=LANGREGEX, prompt=PROMPT,
batchSize=BATCHSIZE, vocab=VOCAB,
maxHistory=MAXHISTORY, langRegex=LANGREGEX,
estimateMode=False # Will be set dynamically based on ESTIMATE batchSize=BATCHSIZE,
) maxHistory=MAXHISTORY,
LEAVE = False estimateMode=False # Will be set dynamically based on ESTIMATE
)
def handleCSV(filename, estimate): LEAVE = False
global ESTIMATE, TOKENS, FILENAME
ESTIMATE = estimate def handleCSV(filename, estimate):
FILENAME = filename global ESTIMATE, TOKENS, FILENAME
ESTIMATE = estimate
if not ESTIMATE: FILENAME = filename
with open("translated/" + filename, "w+t", newline="", encoding=ENCODING, errors="xmlcharrefreplace") as writeFile:
# Translate if not ESTIMATE:
start = time.time() with open("translated/" + filename, "w+t", newline="", encoding=ENCODING, errors="xmlcharrefreplace") as writeFile:
translatedData = openFiles(filename, writeFile) # Translate
start = time.time()
# Print Result translatedData = openFiles(filename, writeFile)
end = time.time()
tqdm.write(getResultString(translatedData, end - start, filename)) # Print Result
with LOCK: end = time.time()
TOKENS[0] += translatedData[1][0] tqdm.write(getResultString(translatedData, end - start, filename))
TOKENS[1] += translatedData[1][1] with LOCK:
else: TOKENS[0] += translatedData[1][0]
# Translate TOKENS[1] += translatedData[1][1]
start = time.time() else:
translatedData = openFilesEstimate(filename) # Translate
start = time.time()
# Print Result translatedData = openFilesEstimate(filename)
end = time.time()
tqdm.write(getResultString(translatedData, end - start, filename)) # Print Result
with LOCK: end = time.time()
TOKENS[0] += translatedData[1][0] tqdm.write(getResultString(translatedData, end - start, filename))
TOKENS[1] += translatedData[1][1] with LOCK:
TOKENS[0] += translatedData[1][0]
# Print Total TOKENS[1] += translatedData[1][1]
totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
# Print Total
# Print any errors on maps totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
if len(MISMATCH) > 0:
return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET # Print any errors on maps
else: if len(MISMATCH) > 0:
return totalString return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
else:
return totalString
def openFiles(filename, writeFile):
with open("files/" + filename, "r", encoding=ENCODING) as readFile, writeFile:
translatedData = parseCSV(readFile, writeFile, filename) def openFiles(filename, writeFile):
with open("files/" + filename, "r", encoding=ENCODING) as readFile, writeFile:
return translatedData translatedData = parseCSV(readFile, writeFile, filename)
return translatedData
def openFilesEstimate(filename):
with open("files/" + filename, "r", encoding="utf8") as readFile:
translatedData = parseCSV(readFile, "", filename) def openFilesEstimate(filename):
with open("files/" + filename, "r", encoding="utf8") as readFile:
return translatedData translatedData = parseCSV(readFile, "", filename)
return translatedData
def getResultString(translatedData, translationTime, filename):
# File Print String
cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL) def getResultString(translatedData, translationTime, filename):
totalTokenstring = ( # File Print String
Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]" cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
"[Output: " totalTokenstring = (
+ str(translatedData[1][1]) Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
+ "]" "[Cost: ${:,.4f}".format(cost) "[Output: "
+ "]" + str(translatedData[1][1])
) + "]" "[Cost: ${:,.4f}".format(cost)
timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]" + "]"
)
if translatedData[2] is None: timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
# Success
return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET if translatedData[2] is None:
else: # Success
# Fail return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
try: else:
raise translatedData[2] # Fail
except Exception as e: try:
traceback.print_exc() raise translatedData[2]
errorString = str(e) + Fore.RED except Exception as e:
return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET traceback.print_exc()
errorString = str(e) + Fore.RED
return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
def parseCSV(readFile, writeFile, filename):
totalTokens = [0, 0]
totalLines = 0 def parseCSV(readFile, writeFile, filename):
global LOCK totalTokens = [0, 0]
totalLines = 0
# Get total for progress bar global LOCK
totalLines = len(readFile.readlines())
readFile.seek(0) # Get total for progress bar
totalLines = len(readFile.readlines())
reader = csv.reader(readFile, delimiter=CSV_DELIMITER) readFile.seek(0)
if not ESTIMATE:
writer = csv.writer( reader = csv.reader(readFile, delimiter=CSV_DELIMITER)
writeFile, if not ESTIMATE:
delimiter=CSV_DELIMITER, writer = csv.writer(
) writeFile,
else: delimiter=CSV_DELIMITER,
writer = "" )
else:
with tqdm(bar_format=BAR_FORMAT, position=POSITION, total=totalLines, leave=LEAVE) as pbar: writer = ""
pbar.desc = filename
pbar.total = totalLines with tqdm(bar_format=BAR_FORMAT, position=POSITION, total=totalLines, leave=LEAVE) as pbar:
pbar.desc = filename
# Grab All Rows pbar.total = totalLines
data = []
for row in reader: # Grab All Rows
data.append(row) data = []
for row in reader:
try: data.append(row)
response = translateCSV(data, pbar, writeFile, writer, filename, None)
totalTokens[0] = response[0] try:
totalTokens[1] = response[1] response = translateCSV(data, pbar, writeFile, writer, filename, None)
except Exception: totalTokens[0] = response[0]
traceback.print_exc() totalTokens[1] = response[1]
return [data, totalTokens, None] except Exception:
traceback.print_exc()
return [data, totalTokens, None]
def flush_progress_csv(writeFile, writer, rows):
"""Flush current CSV progress to the already-open output file (Windows-safe)."""
try: def flush_progress_csv(writeFile, writer, rows):
if ESTIMATE or writeFile is None: """Flush current CSV progress to the already-open output file (Windows-safe)."""
return try:
with LOCK: if ESTIMATE or writeFile is None:
writeFile.seek(0) return
# Recreate writer at current position to avoid state issues with LOCK:
tmp_writer = csv.writer(writeFile, delimiter=CSV_DELIMITER) writeFile.seek(0)
tmp_writer.writerows(rows) # Recreate writer at current position to avoid state issues
writeFile.truncate() tmp_writer = csv.writer(writeFile, delimiter=CSV_DELIMITER)
writeFile.flush() tmp_writer.writerows(rows)
os.fsync(writeFile.fileno()) writeFile.truncate()
except Exception: writeFile.flush()
traceback.print_exc() os.fsync(writeFile.fileno())
except Exception:
traceback.print_exc()
def translateCSV(data, pbar, writeFile, writer, filename, translatedList):
"""
Unified CSV translation function using configurable settings. def translateCSV(data, pbar, writeFile, writer, filename, translatedList):
"""
Uses global settings: Unified CSV translation function using configurable settings.
- SOURCE_COLUMN: column index for source text
- TARGET_COLUMN: column index to write translations Uses global settings:
- SPEAKER_COLUMN: column index for speaker names (-1 = none) - SOURCE_COLUMN: column index for source text
- SKIP_HEADER_ROW: whether to skip first row - TARGET_COLUMN: column index to write translations
- USE_TARGET_IF_NOT_EMPTY: use existing target text if present (T++ style) - SPEAKER_COLUMN: column index for speaker names (-1 = none)
- WRITE_TO_NEXT_COLUMN: write to column after target - SKIP_HEADER_ROW: whether to skip first row
- PARSE_NAME_TAGS: parse :name[] tags - USE_TARGET_IF_NOT_EMPTY: use existing target text if present (T++ style)
- PARSE_M_MARKERS: parse \\M markers - WRITE_TO_NEXT_COLUMN: write to column after target
- REMOVE_FURIGANA: remove furigana - PARSE_NAME_TAGS: parse :name[] tags
- SKIP_COMMENT_ROWS: skip rows with 'comment' in first column - PARSE_M_MARKERS: parse \\M markers
""" - REMOVE_FURIGANA: remove furigana
global LOCK, ESTIMATE, PBAR - SKIP_COMMENT_ROWS: skip rows with 'comment' in first column
PBAR = pbar """
translatedText = "" global LOCK, ESTIMATE, PBAR
totalTokens = [0, 0] PBAR = pbar
i = 0 translatedText = ""
stringList = [] totalTokens = [0, 0]
i = 0
try: stringList = []
# Translate
while i < len(data): try:
# Skip header row if configured # Translate
if SKIP_HEADER_ROW and i == 0: while i < len(data):
i += 1 # Skip header row if configured
continue if SKIP_HEADER_ROW and i == 0:
i += 1
# Skip comment rows if configured continue
if SKIP_COMMENT_ROWS and len(data[i]) > 0 and 'comment' in str(data[i][0]).lower():
i += 1 # Skip comment rows if configured
continue if SKIP_COMMENT_ROWS and len(data[i]) > 0 and 'comment' in str(data[i][0]).lower():
i += 1
# Check if row has enough columns continue
if len(data[i]) <= SOURCE_COLUMN:
i += 1 # Check if row has enough columns
continue if len(data[i]) <= SOURCE_COLUMN:
i += 1
# Get source text continue
jaString = ""
speaker = "" # Get source text
jaString = ""
# If USE_TARGET_IF_NOT_EMPTY is enabled (T++ style), check target column first speaker = ""
if USE_TARGET_IF_NOT_EMPTY and len(data[i]) > TARGET_COLUMN and data[i][TARGET_COLUMN]:
jaString = data[i][TARGET_COLUMN] # If USE_TARGET_IF_NOT_EMPTY is enabled (T++ style), check target column first
else: if USE_TARGET_IF_NOT_EMPTY and len(data[i]) > TARGET_COLUMN and data[i][TARGET_COLUMN]:
jaString = data[i][SOURCE_COLUMN] if data[i][SOURCE_COLUMN] else "" jaString = data[i][TARGET_COLUMN]
else:
# Skip empty strings jaString = data[i][SOURCE_COLUMN] if data[i][SOURCE_COLUMN] else ""
if not jaString:
i += 1 # Skip empty strings
continue if not jaString:
i += 1
# Handle speaker column if configured continue
if SPEAKER_COLUMN >= 0 and len(data[i]) > SPEAKER_COLUMN and data[i][SPEAKER_COLUMN]:
speakerResponse = getSpeaker(data[i][SPEAKER_COLUMN]) # Handle speaker column if configured
totalTokens[0] += speakerResponse[1][0] if SPEAKER_COLUMN >= 0 and len(data[i]) > SPEAKER_COLUMN and data[i][SPEAKER_COLUMN]:
totalTokens[1] += speakerResponse[1][1] speakerResponse = getSpeaker(data[i][SPEAKER_COLUMN])
speaker = speakerResponse[0] totalTokens[0] += speakerResponse[1][0]
data[i][SPEAKER_COLUMN] = speaker totalTokens[1] += speakerResponse[1][1]
speaker = speakerResponse[0]
# Parse :name[] tags if configured data[i][SPEAKER_COLUMN] = speaker
if PARSE_NAME_TAGS and ':name' in jaString:
match = re.search(r":name\[([^\]]+?)\]\n([\w\W]*)", jaString) # Parse :name[] tags if configured
if match: if PARSE_NAME_TAGS and ':name' in jaString:
# Translate speaker name match = re.search(r":name\[([^\]]+?)\]\n([\w\W]*)", jaString)
response = getSpeaker(match.group(1)) if match:
speaker = response[0] # Translate speaker name
totalTokens[0] += response[1][0] response = getSpeaker(match.group(1))
totalTokens[1] += response[1][1] speaker = response[0]
data[i][SOURCE_COLUMN] = data[i][SOURCE_COLUMN].replace(match.group(1), speaker) totalTokens[0] += response[1][0]
totalTokens[1] += response[1][1]
# Extract text portion data[i][SOURCE_COLUMN] = data[i][SOURCE_COLUMN].replace(match.group(1), speaker)
jaString = match.group(2)
# Remove voice markers # Extract text portion
voMatch = re.search(r"\\[vfF]+\[.+]", jaString) jaString = match.group(2)
if voMatch: # Remove voice markers
jaString = jaString.replace(voMatch.group(0), "") voMatch = re.search(r"\\[vfF]+\[.+]", jaString)
if voMatch:
# Parse \M markers if configured jaString = jaString.replace(voMatch.group(0), "")
if PARSE_M_MARKERS and '\\M' in jaString:
match = re.search(r"\\M.+\n([\w\W]*)", jaString) # Parse \M markers if configured
if match: if PARSE_M_MARKERS and '\\M' in jaString:
jaString = match.group(1) match = re.search(r"\\M.+\n([\w\W]*)", jaString)
voMatch = re.search(r"\\[vfF]+\[.+]", jaString) if match:
if voMatch: jaString = match.group(1)
jaString = jaString.replace(voMatch.group(0), "") voMatch = re.search(r"\\[vfF]+\[.+]", jaString)
if voMatch:
# Remove furigana if configured jaString = jaString.replace(voMatch.group(0), "")
if REMOVE_FURIGANA:
jaString = re.sub(r"＜(.*)＝.*＞", r"\1", jaString) # Remove furigana if configured
if REMOVE_FURIGANA:
# Store original for replacement jaString = re.sub(r"＜(.*)＝.*＞", r"\1", jaString)
ojaString = jaString
# Store original for replacement
# Remove textwrap ojaString = jaString
jaString = jaString.replace("\\n", " ")
jaString = jaString.replace("\n", " ") # Remove textwrap
jaString = jaString.replace("\\n", " ")
# Pass 1: Collect strings jaString = jaString.replace("\n", " ")
if not translatedList:
if speaker: # Pass 1: Collect strings
stringList.append(f"[{speaker}]: {jaString}") if not translatedList:
else: if speaker:
stringList.append(jaString) stringList.append(f"[{speaker}]: {jaString}")
else:
# Pass 2: Apply translations stringList.append(jaString)
else:
# Grab and pop translation # Pass 2: Apply translations
translatedText = translatedList[0] else:
translatedList.pop(0) # Grab and pop translation
translatedText = translatedList[0]
# Remove speaker prefix from translation if present translatedList.pop(0)
if speaker:
translatedText = re.sub(r"^\[?(.+?)\]?\s?[|:]\s?", "", translatedText) # Remove speaker prefix from translation if present
if speaker:
# Add wordwrap translatedText = re.sub(r"^\[?(.+?)\]?\s?[|:]\s?", "", translatedText)
translatedText = dazedwrap.wrapText(translatedText, WIDTH)
translatedText = translatedText.replace("\n", "\\n") # Add wordwrap
translatedText = dazedwrap.wrapText(translatedText, WIDTH)
# Determine target column translatedText = translatedText.replace("\n", "\\n")
actual_target = TARGET_COLUMN + 1 if WRITE_TO_NEXT_COLUMN else TARGET_COLUMN
# Determine target column
# Ensure row has enough columns actual_target = TARGET_COLUMN + 1 if WRITE_TO_NEXT_COLUMN else TARGET_COLUMN
while len(data[i]) <= actual_target:
data[i].append("") # Ensure row has enough columns
while len(data[i]) <= actual_target:
# Set data data[i].append("")
if PARSE_NAME_TAGS or PARSE_M_MARKERS:
# Replace original text portion # Set data
data[i][actual_target] = data[i][actual_target].replace(ojaString, translatedText) if data[i][actual_target] else translatedText if PARSE_NAME_TAGS or PARSE_M_MARKERS:
else: # Replace original text portion
data[i][actual_target] = translatedText data[i][actual_target] = data[i][actual_target].replace(ojaString, translatedText) if data[i][actual_target] else translatedText
else:
flush_progress_csv(writeFile, writer, data) data[i][actual_target] = translatedText
# Iterate flush_progress_csv(writeFile, writer, data)
i += 1
# Iterate
# EOF - Process collected strings i += 1
if len(stringList) > 0:
# Set Progress # EOF - Process collected strings
pbar.total = len(stringList) if len(stringList) > 0:
pbar.refresh() # Set Progress
pbar.total = len(stringList)
# Translate pbar.refresh()
response = translateAI(stringList, "")
totalTokens[0] += response[1][0] # Translate
totalTokens[1] += response[1][1] response = translateAI(stringList, "")
translatedList = response[0] totalTokens[0] += response[1][0]
totalTokens[1] += response[1][1]
# Set Strings translatedList = response[0]
if len(stringList) == len(translatedList):
translateCSV(data, pbar, writeFile, writer, filename, translatedList) # Set Strings
# Mismatch if len(stringList) == len(translatedList):
else: translateCSV(data, pbar, writeFile, writer, filename, translatedList)
with LOCK: # Mismatch
if filename not in MISMATCH: else:
MISMATCH.append(filename) with LOCK:
if filename not in MISMATCH:
# Write all Data MISMATCH.append(filename)
with LOCK:
if not ESTIMATE: # Write all Data
for row in data: with LOCK:
writer.writerow(row) if not ESTIMATE:
flush_progress_csv(writeFile, writer, data) for row in data:
writer.writerow(row)
except Exception: flush_progress_csv(writeFile, writer, data)
traceback.print_exc()
except Exception:
# Write all Data traceback.print_exc()
with LOCK:
if not ESTIMATE: # Write all Data
for row in data: with LOCK:
writer.writerow(row) if not ESTIMATE:
flush_progress_csv(writeFile, writer, data) for row in data:
return totalTokens writer.writerow(row)
flush_progress_csv(writeFile, writer, data)
return totalTokens return totalTokens
# Save some money and enter the character before translation return totalTokens
def getSpeaker(speaker):
match speaker: # Save some money and enter the character before translation
case "ファイン": def getSpeaker(speaker):
return ["Fine", [0, 0]] match speaker:
case "": case "ファイン":
return ["", [0, 0]] return ["Fine", [0, 0]]
case _: case "":
# Find Speaker return ["", [0, 0]]
for i in range(len(NAMESLIST)): case _:
if speaker == NAMESLIST[i][0]: # Find Speaker
return [NAMESLIST[i][1], [0, 0]] for i in range(len(NAMESLIST)):
if speaker == NAMESLIST[i][0]:
if speaker == "？？？": return [NAMESLIST[i][1], [0, 0]]
return ["???", [0, 0]]
if speaker == "？？？":
# Translate and Store Speaker return ["???", [0, 0]]
response = translateAI(
f"{speaker}", # Translate and Store Speaker
"Reply with the " + LANGUAGE + " translation of the NPC name.", response = translateAI(
False, f"{speaker}",
) "Reply with the " + LANGUAGE + " translation of the NPC name.",
response[0] = response[0].title() False,
response[0] = response[0].replace("'S", "'s") )
response[0] = response[0].replace("Speaker: ", "") response[0] = response[0].title()
response[0] = response[0].replace("'S", "'s")
# Retry if name doesn't translate for some reason response[0] = response[0].replace("Speaker: ", "")
if re.search(r"([a-zA-Z？?])", response[0]) == None:
response = translateAI( # Retry if name doesn't translate for some reason
f"{speaker}", if re.search(r"([a-zA-Z？?])", response[0]) == None:
"Reply with the " + LANGUAGE + " translation of the NPC name.", response = translateAI(
False, f"{speaker}",
) "Reply with the " + LANGUAGE + " translation of the NPC name.",
response[0] = response[0].title() False,
response[0] = response[0].replace("'S", "'s") )
response[0] = response[0].title()
speakerList = [speaker, response[0]] response[0] = response[0].replace("'S", "'s")
NAMESLIST.append(speakerList)
return response speakerList = [speaker, response[0]]
return [speaker, [0, 0]] NAMESLIST.append(speakerList)
return response
def translateAI(text, history, history_ctx=None): return [speaker, [0, 0]]
"""
Legacy wrapper function for the new shared translation utility. def translateAI(text, history, history_ctx=None):
This maintains compatibility with existing code while using the new shared implementation. """
""" Legacy wrapper function for the new shared translation utility.
global PBAR, MISMATCH, FILENAME This maintains compatibility with existing code while using the new shared implementation.
"""
# Update config estimate mode based on global ESTIMATE global PBAR, MISMATCH, FILENAME
TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
# Update config estimate mode based on global ESTIMATE
# Call the new shared translation function TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
return sharedtranslateAI(
text=text, # Call the new shared translation function
history=history, return sharedtranslateAI(
config=TRANSLATION_CONFIG, text=text,
filename=FILENAME, history=history,
pbar=PBAR, config=TRANSLATION_CONFIG,
lock=LOCK, filename=FILENAME,
mismatchList=MISMATCH pbar=PBAR,
) lock=LOCK,
mismatchList=MISMATCH
)

880
modules/images.py
View file

@ -1,439 +1,441 @@
# Libraries # Libraries
from PIL import Image, ImageDraw, ImageFont from PIL import Image, ImageDraw, ImageFont
import json import json
import os import os
import re import re
import threading import threading
import time import time
import traceback import traceback
import tiktoken import tiktoken
from pathlib import Path from pathlib import Path
from colorama import Fore from colorama import Fore
from dotenv import load_dotenv from dotenv import load_dotenv
from retry import retry from retry import retry
from tqdm import tqdm from tqdm import tqdm
from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
# Globals # Globals
MODEL = os.getenv("model") MODEL = os.getenv("model")
TIMEOUT = int(os.getenv("timeout")) TIMEOUT = int(os.getenv("timeout"))
LANGUAGE = os.getenv("language").capitalize() LANGUAGE = os.getenv("language").capitalize()
PROMPT = Path("prompt.txt").read_text(encoding="utf-8") from util.paths import PROMPT_PATH, VOCAB_PATH
VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
LOCK = threading.Lock() PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
PBAR = None VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
WIDTH = int(os.getenv("width")) LOCK = threading.Lock()
LISTWIDTH = int(os.getenv("listWidth")) PBAR = None
NOTEWIDTH = int(os.getenv("noteWidth")) WIDTH = int(os.getenv("width"))
MAXHISTORY = 10 LISTWIDTH = int(os.getenv("listWidth"))
ESTIMATE = "" NOTEWIDTH = int(os.getenv("noteWidth"))
TOKENS = [0, 0] MAXHISTORY = 10
NAMESLIST = [] ESTIMATE = ""
MISMATCH = [] # Lists files that throw a mismatch error (Length of GPT list response is wrong) TOKENS = [0, 0]
FILENAME = None NAMESLIST = []
MISMATCH = [] # Lists files that throw a mismatch error (Length of GPT list response is wrong)
# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex FILENAME = None
LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
# Get pricing configuration based on the model LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
PRICING_CONFIG = getPricingConfig(MODEL)
INPUTAPICOST = PRICING_CONFIG["inputAPICost"] # Get pricing configuration based on the model
OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"] PRICING_CONFIG = getPricingConfig(MODEL)
BATCHSIZE = PRICING_CONFIG["batchSize"] INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"] OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
BATCHSIZE = PRICING_CONFIG["batchSize"]
# Initialize Translation Config FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
TRANSLATION_CONFIG = TranslationConfig(
model=MODEL, # Initialize Translation Config
language=LANGUAGE, TRANSLATION_CONFIG = TranslationConfig(
prompt=PROMPT, model=MODEL,
vocab=VOCAB, language=LANGUAGE,
langRegex=LANGREGEX, prompt=PROMPT,
batchSize=BATCHSIZE, vocab=VOCAB,
maxHistory=MAXHISTORY, langRegex=LANGREGEX,
estimateMode=False # Will be set dynamically based on ESTIMATE batchSize=BATCHSIZE,
) maxHistory=MAXHISTORY,
LEAVE = False estimateMode=False # Will be set dynamically based on ESTIMATE
)
# tqdm Globals LEAVE = False
BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
POSITION = 0 # tqdm Globals
LEAVE = False BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
POSITION = 0
LEAVE = False
def handleImages(folderName, estimate):
global ESTIMATE, TOKENS, FILENAME
ESTIMATE = estimate def handleImages(folderName, estimate):
FILENAME = folderName global ESTIMATE, TOKENS, FILENAME
start = time.time() ESTIMATE = estimate
FILENAME = folderName
# Translate Strings start = time.time()
translatedData = openFiles(f"files/{folderName}")
# Translate Strings
# Custom Names translatedData = openFiles(f"files/{folderName}")
# customList = [[], []]
# customList = processImagesDir("Custom", customList) # Custom Names
# customList = [[], []]
# Write TL To Images # customList = processImagesDir("Custom", customList)
try:
translatedList, originalList, dimensionsList = translatedData[0] # Write TL To Images
for i in range(len(translatedList)): try:
try: translatedList, originalList, dimensionsList = translatedData[0]
# Create image from string for i in range(len(translatedList)):
image = stringToImageOutline(translatedList[i], dimensionsList[i][0], dimensionsList[i][1]) try:
# Save image using the corresponding original filename # Create image from string
image.save(rf"translated/{folderName}/{originalList[i]}.png", quality=100) image = stringToImageOutline(translatedList[i], dimensionsList[i][0], dimensionsList[i][1])
except Exception as e: # Save image using the corresponding original filename
# Log error if image saving fails image.save(rf"translated/{folderName}/{originalList[i]}.png", quality=100)
PBAR.write(f"Error processing {translatedList[i]}: {str(e)}") except Exception as e:
except IndexError: # Log error if image saving fails
PBAR.write("Translated data is incomplete. Please check your input.") PBAR.write(f"Error processing {translatedList[i]}: {str(e)}")
except IndexError:
# Print File PBAR.write("Translated data is incomplete. Please check your input.")
end = time.time()
tqdm.write(getResultString(translatedData, end - start, folderName)) # Print File
with LOCK: end = time.time()
TOKENS[0] += translatedData[1][0] tqdm.write(getResultString(translatedData, end - start, folderName))
TOKENS[1] += translatedData[1][1] with LOCK:
TOKENS[0] += translatedData[1][0]
# Print Total TOKENS[1] += translatedData[1][1]
totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
# Print Total
# Print any errors on maps totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
if len(MISMATCH) > 0:
return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET # Print any errors on maps
else: if len(MISMATCH) > 0:
return totalString return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
else:
return totalString
def openFiles(folderName):
global PBAR
def openFiles(folderName):
if os.path.isdir(folderName): global PBAR
imageList = [[], [], []]
imageList = processImagesDir(folderName, imageList) if os.path.isdir(folderName):
imageList = [[], [], []]
# Start Translation imageList = processImagesDir(folderName, imageList)
with tqdm(
bar_format=BAR_FORMAT, # Start Translation
position=POSITION, with tqdm(
leave=LEAVE, bar_format=BAR_FORMAT,
desc=folderName, position=POSITION,
total=len(imageList[0]), leave=LEAVE,
) as PBAR: desc=folderName,
translatedData = translateImages(imageList) total=len(imageList[0]),
translatedData = [ ) as PBAR:
[translatedData[0], imageList[2], imageList[1]], translatedData = translateImages(imageList)
translatedData[1], translatedData = [
translatedData[2], [translatedData[0], imageList[2], imageList[1]],
] translatedData[1],
translatedData[2],
return translatedData ]
else:
print("The provided directory path does not exist.") return translatedData
else:
print("The provided directory path does not exist.")
def getResultString(translatedData, translationTime, filename):
# File Print String
cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL) def getResultString(translatedData, translationTime, filename):
totalTokenstring = ( # File Print String
Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]" cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
"[Output: " totalTokenstring = (
+ str(translatedData[1][1]) Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
+ "]" "[Cost: ${:,.4f}".format(cost) "[Output: "
+ "]" + str(translatedData[1][1])
) + "]" "[Cost: ${:,.4f}".format(cost)
timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]" + "]"
)
if translatedData[2] is None: timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
# Success
return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET if translatedData[2] is None:
else: # Success
# Fail return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
try: else:
raise translatedData[2] # Fail
except Exception as e: try:
traceback.print_exc() raise translatedData[2]
errorString = str(e) + Fore.RED except Exception as e:
return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET traceback.print_exc()
errorString = str(e) + Fore.RED
return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
def getFontSize(text, image_width, image_height, font_path):
# Start with a high font size and keep reducing it until the text fits within the image bounds
font_size = min(image_width, image_height) def getFontSize(text, image_width, image_height, font_path):
# Start with a high font size and keep reducing it until the text fits within the image bounds
while font_size > 0: font_size = min(image_width, image_height)
font = ImageFont.truetype(font_path, font_size)
text_bbox = ImageDraw.Draw(Image.new("RGB", (1, 1))).textbbox((0, 0), text, font=font) while font_size > 0:
text_width = text_bbox[2] - text_bbox[0] font = ImageFont.truetype(font_path, font_size)
text_height = text_bbox[3] - text_bbox[1] text_bbox = ImageDraw.Draw(Image.new("RGB", (1, 1))).textbbox((0, 0), text, font=font)
text_width = text_bbox[2] - text_bbox[0]
if text_width <= image_width and text_height <= image_height: text_height = text_bbox[3] - text_bbox[1]
return font_size
font_size -= 1 if text_width <= image_width and text_height <= image_height:
return font_size
return font_size font_size -= 1
return font_size
def stringToImage(text, width, height, font_path="fonts/TsunagiGothic.ttf", scale_factor=4):
# Increase the resolution
scaled_width = int(width * scale_factor) def stringToImage(text, width, height, font_path="fonts/TsunagiGothic.ttf", scale_factor=4):
scaled_height = int(height * scale_factor) # Increase the resolution
scaled_width = int(width * scale_factor)
# Find the appropriate font size for the scaled up image scaled_height = int(height * scale_factor)
font_size = getFontSize(text, scaled_width, scaled_height, font_path)
if font_size == 0: # Find the appropriate font size for the scaled up image
raise ValueError("Text is too long to fit in the supplied dimensions.") font_size = getFontSize(text, scaled_width, scaled_height, font_path)
if font_size == 0:
# Create a new image with the scaled width and height and a transparent background raise ValueError("Text is too long to fit in the supplied dimensions.")
image = Image.new("RGBA", (scaled_width, scaled_height), (255, 255, 255, 0))
# Create a new image with the scaled width and height and a transparent background
# Create a drawing context image = Image.new("RGBA", (scaled_width, scaled_height), (255, 255, 255, 0))
draw = ImageDraw.Draw(image)
# Create a drawing context
# Load the appropriate font draw = ImageDraw.Draw(image)
font = ImageFont.truetype(font_path, font_size)
# Load the appropriate font
# Calculate the size of the text to center it font = ImageFont.truetype(font_path, font_size)
text_bbox = draw.textbbox((0, 0), text, font=font)
text_width = text_bbox[2] - text_bbox[0] # Calculate the size of the text to center it
text_height = text_bbox[3] - text_bbox[1] + 20 text_bbox = draw.textbbox((0, 0), text, font=font)
x = 0 text_width = text_bbox[2] - text_bbox[0]
text_height = text_bbox[3] - text_bbox[1] + 20
x = (scaled_width - text_width) // 2 x = 0
y = (scaled_height - text_height) // 2
x = (scaled_width - text_width) // 2
# Draw the text on the image y = (scaled_height - text_height) // 2
draw.text((x, y), text, font=font, fill=(255, 255, 255, 255))
# Draw the text on the image
# Resize back to the original dimensions to get a clearer text rendering draw.text((x, y), text, font=font, fill=(255, 255, 255, 255))
image = image.resize(
(width, height), # Resize back to the original dimensions to get a clearer text rendering
Image.LANCZOS, image = image.resize(
) (width, height),
Image.LANCZOS,
return image )
return image
from PIL import Image, ImageDraw, ImageFont
from PIL import Image, ImageDraw, ImageFont
def stringToImageOutline(text, width, height, font_path="fonts/TsunagiGothic.ttf", scale_factor=4):
# Outline
outline_color = (255, 255, 255, 255) def stringToImageOutline(text, width, height, font_path="fonts/TsunagiGothic.ttf", scale_factor=4):
text_color = (0, 0, 0, 255) # Outline
outline_thickness = 4 outline_color = (255, 255, 255, 255)
text_color = (0, 0, 0, 255)
# Increase the resolution outline_thickness = 4
scaled_width = int(width * scale_factor)
scaled_height = int(height * scale_factor) # Increase the resolution
scaled_width = int(width * scale_factor)
# Find the appropriate font size for the scaled up image scaled_height = int(height * scale_factor)
font_size = getFontSize(text, scaled_width, scaled_height, font_path)
if font_size == 0: # Find the appropriate font size for the scaled up image
raise ValueError("Text is too long to fit in the supplied dimensions.") font_size = getFontSize(text, scaled_width, scaled_height, font_path)
if font_size == 0:
# Create a new image with the scaled width and height and a transparent background raise ValueError("Text is too long to fit in the supplied dimensions.")
image = Image.new("RGBA", (scaled_width, scaled_height), (255, 255, 255, 0))
# Create a new image with the scaled width and height and a transparent background
# Create a drawing context image = Image.new("RGBA", (scaled_width, scaled_height), (255, 255, 255, 0))
draw = ImageDraw.Draw(image)
# Create a drawing context
# Load the appropriate font draw = ImageDraw.Draw(image)
font = ImageFont.truetype(font_path, font_size)
# Load the appropriate font
# Calculate the size of the text to center it font = ImageFont.truetype(font_path, font_size)
text_bbox = draw.textbbox((0, 0), text, font=font)
text_width = text_bbox[2] - text_bbox[0] # Calculate the size of the text to center it
text_height = text_bbox[3] - text_bbox[1] + 20 text_bbox = draw.textbbox((0, 0), text, font=font)
x = (scaled_width - text_width) // 2 text_width = text_bbox[2] - text_bbox[0]
y = (scaled_height - text_height) // 2 text_height = text_bbox[3] - text_bbox[1] + 20
x = (scaled_width - text_width) // 2
# Draw the text outline by applying the text multiple times with small offsets y = (scaled_height - text_height) // 2
for dx in range(-outline_thickness, outline_thickness + 1):
for dy in range(-outline_thickness, outline_thickness + 1): # Draw the text outline by applying the text multiple times with small offsets
if dx != 0 or dy != 0: for dx in range(-outline_thickness, outline_thickness + 1):
draw.text((x + dx, y + dy), text, font=font, fill=outline_color) for dy in range(-outline_thickness, outline_thickness + 1):
if dx != 0 or dy != 0:
# Draw the main text draw.text((x + dx, y + dy), text, font=font, fill=outline_color)
draw.text((x, y), text, font=font, fill=text_color)
# Draw the main text
# Resize back to the original dimensions to get a clearer text rendering draw.text((x, y), text, font=font, fill=text_color)
image = image.resize((width, height), Image.LANCZOS)
# Resize back to the original dimensions to get a clearer text rendering
return image image = image.resize((width, height), Image.LANCZOS)
return image
def stringToImageBox(text, width, height, font_path="fonts/TsunagiGothic.ttf", scale_factor=4):
# Increase the resolution
scaled_width = int(width * scale_factor) def stringToImageBox(text, width, height, font_path="fonts/TsunagiGothic.ttf", scale_factor=4):
scaled_height = int(height * scale_factor) # Increase the resolution
scaled_width = int(width * scale_factor)
# Padding around the text scaled_height = int(height * scale_factor)
padding = 10
# Padding around the text
# Calculate the dimensions available for text placement padding = 10
available_width = scaled_width - 2 * padding
available_height = scaled_height - 2 * padding # Calculate the dimensions available for text placement
available_width = scaled_width - 2 * padding
# Determine the best font size to fit within the available dimensions available_height = scaled_height - 2 * padding
font_size = getFontSize(text, available_width, available_height, font_path)
if font_size <= 0: # Determine the best font size to fit within the available dimensions
raise ValueError("Text is too long to fit in the supplied dimensions.") font_size = getFontSize(text, available_width, available_height, font_path)
if font_size <= 0:
# Create a new image with increased resolution raise ValueError("Text is too long to fit in the supplied dimensions.")
image = Image.new("RGBA", (scaled_width, scaled_height), (255, 255, 255, 0))
draw = ImageDraw.Draw(image) # Create a new image with increased resolution
image = Image.new("RGBA", (scaled_width, scaled_height), (255, 255, 255, 0))
# Load the calculated font draw = ImageDraw.Draw(image)
font = ImageFont.truetype(font_path, font_size)
# Load the calculated font
# Calculate the size and bounding box of the text font = ImageFont.truetype(font_path, font_size)
text_bbox = draw.textbbox((0, 0), text, font=font)
text_width = text_bbox[2] - text_bbox[0] # Calculate the size and bounding box of the text
text_height = text_bbox[3] - text_bbox[1] + 20 text_bbox = draw.textbbox((0, 0), text, font=font)
text_width = text_bbox[2] - text_bbox[0]
# Determine centered position for the text while considering padding text_height = text_bbox[3] - text_bbox[1] + 20
# Additional adjustment ensures text appears centrally aligned
x = (scaled_width - text_width) // 2 # Determine centered position for the text while considering padding
y = (scaled_height - text_height) // 2 # Additional adjustment ensures text appears centrally aligned
x = (scaled_width - text_width) // 2
# Draw a black box with a white outline that fits the image dimensions precisely y = (scaled_height - text_height) // 2
draw.rectangle([0, 0, scaled_width - 1, scaled_height - 1], outline=(255, 255, 255, 255), width=1)
# Draw a black box with a white outline that fits the image dimensions precisely
# Fill the inside box with black color draw.rectangle([0, 0, scaled_width - 1, scaled_height - 1], outline=(255, 255, 255, 255), width=1)
draw.rectangle([1, 1, scaled_width - 2, scaled_height - 2], fill=(0, 0, 0, 255))
# Fill the inside box with black color
# Render the text within the image draw.rectangle([1, 1, scaled_width - 2, scaled_height - 2], fill=(0, 0, 0, 255))
draw.text((x, y), text, font=font, fill=(255, 255, 255, 255))
# Render the text within the image
# Shrink the image back to original dimensions with high-quality interpolation draw.text((x, y), text, font=font, fill=(255, 255, 255, 255))
image = image.resize(
(width, height), # Shrink the image back to original dimensions with high-quality interpolation
Image.LANCZOS, image = image.resize(
) (width, height),
Image.LANCZOS,
return image )
return image
def getImageDimensions(file_path):
try:
with Image.open(file_path) as img: def getImageDimensions(file_path):
width, height = img.size try:
return width, height with Image.open(file_path) as img:
except Exception as e: width, height = img.size
print(f"Error reading {file_path}: {e}") return width, height
return None, None except Exception as e:
print(f"Error reading {file_path}: {e}")
return None, None
def processImagesDir(directory_path, imageList):
for file_name in os.listdir(directory_path):
# .png and Japanese def processImagesDir(directory_path, imageList):
if ".png" in file_name: for file_name in os.listdir(directory_path):
file_path = os.path.join(directory_path, file_name) # .png and Japanese
if os.path.isfile(file_path): if ".png" in file_name:
# Check if the file is an image file_path = os.path.join(directory_path, file_name)
try: if os.path.isfile(file_path):
width, height = getImageDimensions(file_path) # Check if the file is an image
if width is not None and height is not None: try:
placeholders = { width, height = getImageDimensions(file_path)
".png": "", if width is not None and height is not None:
} placeholders = {
for target, replacement in placeholders.items(): ".png": "",
file_name = file_name.replace(target, replacement) }
match = re.search(r"[\[【].+?[\]】](.*)", file_name) for target, replacement in placeholders.items():
if match: file_name = file_name.replace(target, replacement)
text = match.group(1) match = re.search(r"[\[【].+?[\]】](.*)", file_name)
else: if match:
text = file_name text = match.group(1)
imageList[0].append(text) else:
imageList[1].append([width, height]) text = file_name
imageList[2].append(file_name) imageList[0].append(text)
except Exception as e: imageList[1].append([width, height])
print(f"Error processing {file_name}: {e}") imageList[2].append(file_name)
except Exception as e:
if ".txt" in file_name: print(f"Error processing {file_name}: {e}")
try:
with open(f"{directory_path}/{file_name}", "r", encoding="utf8") as file: if ".txt" in file_name:
for line in file: try:
line = line.strip() with open(f"{directory_path}/{file_name}", "r", encoding="utf8") as file:
line = line.replace(":", "：") for line in file:
line = line.replace("/", "／") line = line.strip()
line = line.replace("?", "？") line = line.replace(":", "：")
imageList[0].append(line) # Using strip() to remove any extra newlines or spaces line = line.replace("/", "／")
imageList[1].append([104, 15]) line = line.replace("?", "？")
except FileNotFoundError: imageList[0].append(line) # Using strip() to remove any extra newlines or spaces
print(f"The file at {file_path} was not found.") imageList[1].append([104, 15])
except IOError: except FileNotFoundError:
print(f"An error occurred while reading the file at {file_path}.") print(f"The file at {file_path} was not found.")
return imageList except IOError:
print(f"An error occurred while reading the file at {file_path}.")
return imageList
def translateImages(imageList):
totalTokens = [0, 0]
def translateImages(imageList):
# Translate GPT totalTokens = [0, 0]
response = translateAI(imageList[0], "Keep the Translation as brief as possible")
translatedList = response[0] # Translate GPT
totalTokens[0] += response[1][0] response = translateAI(imageList[0], "Keep the Translation as brief as possible")
totalTokens[1] += response[1][1] translatedList = response[0]
totalTokens[0] += response[1][0]
return [translatedList, totalTokens, None] totalTokens[1] += response[1][1]
# Save some money and enter the character before translation return [translatedList, totalTokens, None]
def getSpeaker(speaker):
match speaker: # Save some money and enter the character before translation
case "ファイン": def getSpeaker(speaker):
return ["Fine", [0, 0]] match speaker:
case "": case "ファイン":
return ["", [0, 0]] return ["Fine", [0, 0]]
case _: case "":
# Find Speaker return ["", [0, 0]]
for i in range(len(NAMESLIST)): case _:
if speaker == NAMESLIST[i][0]: # Find Speaker
return [NAMESLIST[i][1], [0, 0]] for i in range(len(NAMESLIST)):
if speaker == NAMESLIST[i][0]:
# Translate and Store Speaker return [NAMESLIST[i][1], [0, 0]]
response = translateAI(
f"{speaker}", # Translate and Store Speaker
"Reply with the " + LANGUAGE + " translation of the NPC name.", response = translateAI(
False, f"{speaker}",
) "Reply with the " + LANGUAGE + " translation of the NPC name.",
response[0] = response[0].title() False,
response[0] = response[0].replace("'S", "'s") )
response[0] = response[0].replace("Speaker: ", "") response[0] = response[0].title()
response[0] = response[0].replace("'S", "'s")
# Retry if name doesn't translate for some reason response[0] = response[0].replace("Speaker: ", "")
if re.search(r"([a-zA-Z？?])", response[0]) == None:
response = translateAI( # Retry if name doesn't translate for some reason
f"{speaker}", if re.search(r"([a-zA-Z？?])", response[0]) == None:
"Reply with the " + LANGUAGE + " translation of the NPC name.", response = translateAI(
False, f"{speaker}",
) "Reply with the " + LANGUAGE + " translation of the NPC name.",
response[0] = response[0].title() False,
response[0] = response[0].replace("'S", "'s") )
response[0] = response[0].title()
speakerList = [speaker, response[0]] response[0] = response[0].replace("'S", "'s")
NAMESLIST.append(speakerList)
return response speakerList = [speaker, response[0]]
return [speaker, [0, 0]] NAMESLIST.append(speakerList)
def translateAI(text, history, history_ctx=None): return response
""" return [speaker, [0, 0]]
Legacy wrapper function for the new shared translation utility. def translateAI(text, history, history_ctx=None):
This maintains compatibility with existing code while using the new shared implementation. """
""" Legacy wrapper function for the new shared translation utility.
global PBAR, MISMATCH, FILENAME This maintains compatibility with existing code while using the new shared implementation.
"""
# Update config estimate mode based on global ESTIMATE global PBAR, MISMATCH, FILENAME
TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
# Update config estimate mode based on global ESTIMATE
# Call the new shared translation function TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
return sharedtranslateAI(
text=text, # Call the new shared translation function
history=history, return sharedtranslateAI(
config=TRANSLATION_CONFIG, text=text,
filename=FILENAME, history=history,
pbar=PBAR, config=TRANSLATION_CONFIG,
lock=LOCK, filename=FILENAME,
mismatchList=MISMATCH pbar=PBAR,
) lock=LOCK,
mismatchList=MISMATCH
)

2720
modules/json.py
View file

File diff suppressed because it is too large Load diff

1064
modules/kirikiri.py
View file

File diff suppressed because it is too large Load diff

746
modules/lune.py
View file

@ -1,372 +1,374 @@
# Libraries # Libraries
import json import json
import os import os
import re import re
import util.dazedwrap as dazedwrap import util.dazedwrap as dazedwrap
import threading import threading
import time import time
import traceback import traceback
import tiktoken import tiktoken
from pathlib import Path from pathlib import Path
from colorama import Fore from colorama import Fore
from dotenv import load_dotenv from dotenv import load_dotenv
from retry import retry from retry import retry
from tqdm import tqdm from tqdm import tqdm
from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
import tempfile import tempfile
# Globals # Globals
MODEL = os.getenv("model") MODEL = os.getenv("model")
TIMEOUT = int(os.getenv("timeout")) TIMEOUT = int(os.getenv("timeout"))
LANGUAGE = os.getenv("language").capitalize() LANGUAGE = os.getenv("language").capitalize()
PROMPT = Path("prompt.txt").read_text(encoding="utf-8") from util.paths import PROMPT_PATH, VOCAB_PATH
VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
LOCK = threading.Lock() PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
WIDTH = int(os.getenv("width")) VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
LISTWIDTH = int(os.getenv("listWidth")) LOCK = threading.Lock()
NOTEWIDTH = 70 WIDTH = int(os.getenv("width"))
MAXHISTORY = 10 LISTWIDTH = int(os.getenv("listWidth"))
ESTIMATE = "" NOTEWIDTH = 70
TOKENS = [0, 0] MAXHISTORY = 10
NAMESLIST = [] ESTIMATE = ""
NAMES = False # Output a list of all the character names found TOKENS = [0, 0]
BRFLAG = False # If the game uses <br> instead NAMESLIST = []
FIXTEXTWRAP = True # Overwrites textwrap NAMES = False # Output a list of all the character names found
IGNORETLTEXT = False # Ignores all translated text. BRFLAG = False # If the game uses <br> instead
MISMATCH = [] # Lists files that throw a mismatch error (Length of GPT list response is wrong) FIXTEXTWRAP = True # Overwrites textwrap
PBAR = None IGNORETLTEXT = False # Ignores all translated text.
FILENAME = None MISMATCH = [] # Lists files that throw a mismatch error (Length of GPT list response is wrong)
PBAR = None
# tqdm Globals FILENAME = None
BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
POSITION = 0 # tqdm Globals
LEAVE = False BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
POSITION = 0
# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex LEAVE = False
LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
# Get pricing configuration based on the model LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
PRICING_CONFIG = getPricingConfig(MODEL)
INPUTAPICOST = PRICING_CONFIG["inputAPICost"] # Get pricing configuration based on the model
OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"] PRICING_CONFIG = getPricingConfig(MODEL)
BATCHSIZE = PRICING_CONFIG["batchSize"] INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"] OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
BATCHSIZE = PRICING_CONFIG["batchSize"]
# Initialize Translation Config FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
TRANSLATION_CONFIG = TranslationConfig(
model=MODEL, # Initialize Translation Config
language=LANGUAGE, TRANSLATION_CONFIG = TranslationConfig(
prompt=PROMPT, model=MODEL,
vocab=VOCAB, language=LANGUAGE,
langRegex=LANGREGEX, prompt=PROMPT,
batchSize=BATCHSIZE, vocab=VOCAB,
maxHistory=MAXHISTORY, langRegex=LANGREGEX,
estimateMode=False # Will be set dynamically based on ESTIMATE batchSize=BATCHSIZE,
) maxHistory=MAXHISTORY,
LEAVE = False estimateMode=False # Will be set dynamically based on ESTIMATE
)
def handleLune(filename, estimate): LEAVE = False
global FILENAME, ESTIMATE, totalTokens
ESTIMATE = estimate def handleLune(filename, estimate):
FILENAME = filename global FILENAME, ESTIMATE, totalTokens
ESTIMATE = estimate
if estimate: FILENAME = filename
start = time.time()
translatedData = openFiles(filename) if estimate:
start = time.time()
# Print Result translatedData = openFiles(filename)
end = time.time()
tqdm.write(getResultString(translatedData, end - start, filename)) # Print Result
with LOCK: end = time.time()
TOKENS[0] += translatedData[1][0] tqdm.write(getResultString(translatedData, end - start, filename))
TOKENS[1] += translatedData[1][1] with LOCK:
TOKENS[0] += translatedData[1][0]
return getResultString(["", TOKENS, None], end - start, "TOTAL") TOKENS[1] += translatedData[1][1]
else: return getResultString(["", TOKENS, None], end - start, "TOTAL")
try:
with open("translated/" + filename, "w", encoding="utf-8", newline="\n") as outFile: else:
start = time.time() try:
translatedData = openFiles(filename) with open("translated/" + filename, "w", encoding="utf-8", newline="\n") as outFile:
start = time.time()
# Print Result translatedData = openFiles(filename)
end = time.time()
json.dump(translatedData[0], outFile, ensure_ascii=False, indent=4) # Print Result
tqdm.write(getResultString(translatedData, end - start, filename)) end = time.time()
with LOCK: json.dump(translatedData[0], outFile, ensure_ascii=False, indent=4)
TOKENS[0] += translatedData[1][0] tqdm.write(getResultString(translatedData, end - start, filename))
TOKENS[1] += translatedData[1][1] with LOCK:
except Exception: TOKENS[0] += translatedData[1][0]
return "Fail" TOKENS[1] += translatedData[1][1]
except Exception:
return getResultString(["", TOKENS, None], end - start, "TOTAL") return "Fail"
return getResultString(["", TOKENS, None], end - start, "TOTAL")
def openFiles(filename):
with open("files/" + filename, "r", encoding="UTF-8-sig") as f:
data = json.load(f) def openFiles(filename):
with open("files/" + filename, "r", encoding="UTF-8-sig") as f:
# Map Files data = json.load(f)
if ".json" in filename:
translatedData = parseJSON(data, filename) # Map Files
if ".json" in filename:
else: translatedData = parseJSON(data, filename)
raise NameError(filename + " Not Supported")
else:
return translatedData raise NameError(filename + " Not Supported")
return translatedData
def getResultString(translatedData, translationTime, filename):
# File Print String
cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL) def getResultString(translatedData, translationTime, filename):
totalTokenstring = ( # File Print String
Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]" cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
"[Output: " totalTokenstring = (
+ str(translatedData[1][1]) Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
+ "]" "[Cost: ${:,.4f}".format(cost) "[Output: "
+ "]" + str(translatedData[1][1])
) + "]" "[Cost: ${:,.4f}".format(cost)
timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]" + "]"
)
if translatedData[2] == None: timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
# Success
return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET if translatedData[2] == None:
# Success
else: return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
# Fail
try: else:
raise translatedData[2] # Fail
except Exception as e: try:
traceback.print_exc() raise translatedData[2]
errorString = str(e) + Fore.RED except Exception as e:
return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET traceback.print_exc()
errorString = str(e) + Fore.RED
return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
def parseJSON(data, filename):
totalTokens = [0, 0]
totalLines = 0 def parseJSON(data, filename):
totalLines = len(data) totalTokens = [0, 0]
global LOCK totalLines = 0
totalLines = len(data)
with tqdm(bar_format=BAR_FORMAT, position=POSITION, total=totalLines, leave=LEAVE) as pbar: global LOCK
pbar.desc = filename
pbar.total = totalLines with tqdm(bar_format=BAR_FORMAT, position=POSITION, total=totalLines, leave=LEAVE) as pbar:
try: pbar.desc = filename
result = translateJSON(data, pbar) pbar.total = totalLines
totalTokens[0] += result[0] try:
totalTokens[1] += result[1] result = translateJSON(data, pbar)
except Exception as e: totalTokens[0] += result[0]
return [data, totalTokens, e] totalTokens[1] += result[1]
return [data, totalTokens, None] except Exception as e:
return [data, totalTokens, e]
return [data, totalTokens, None]
def save_progress_json(data, filename):
"""Atomically save current JSON translation progress."""
try: def save_progress_json(data, filename):
if ESTIMATE: """Atomically save current JSON translation progress."""
return try:
os.makedirs("translated", exist_ok=True) if ESTIMATE:
tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated") return
try: os.makedirs("translated", exist_ok=True)
with os.fdopen(tmp_fd, "w", encoding="utf-8", newline="\n") as tmp_file: tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
json.dump(data, tmp_file, ensure_ascii=False, indent=4) try:
os.replace(tmp_path, os.path.join("translated", filename)) with os.fdopen(tmp_fd, "w", encoding="utf-8", newline="\n") as tmp_file:
finally: json.dump(data, tmp_file, ensure_ascii=False, indent=4)
if os.path.exists(tmp_path): os.replace(tmp_path, os.path.join("translated", filename))
try: finally:
os.remove(tmp_path) if os.path.exists(tmp_path):
except OSError: try:
pass os.remove(tmp_path)
except Exception: except OSError:
traceback.print_exc() pass
except Exception:
traceback.print_exc()
def translateJSON(data, pbar):
global PBAR
PBAR = pbar def translateJSON(data, pbar):
textHistory = [] global PBAR
batch = [] PBAR = pbar
maxHistory = MAXHISTORY textHistory = []
tokens = [0, 0] batch = []
speaker = "None" maxHistory = MAXHISTORY
insertBool = False tokens = [0, 0]
i = 0 speaker = "None"
batchStartIndex = 0 insertBool = False
i = 0
while i < len(data): batchStartIndex = 0
item = data[i]
# Speaker while i < len(data):
if "name" in item: item = data[i]
if item["name"] not in [None, "-"]: # Speaker
response = getSpeaker(item["name"]) if "name" in item:
speaker = response[0] if item["name"] not in [None, "-"]:
tokens[0] += response[1][0] response = getSpeaker(item["name"])
tokens[1] += response[1][1] speaker = response[0]
item["name"] = speaker tokens[0] += response[1][0]
save_progress_json(data, FILENAME or "output.json") tokens[1] += response[1][1]
else: item["name"] = speaker
speaker = "None" save_progress_json(data, FILENAME or "output.json")
else:
# Text speaker = "None"
if "message" in item:
for text in [ # Text
"text", if "message" in item:
"text2", for text in [
"help1", "text",
"help2", "text2",
"help3", "help1",
"like", "help2",
"message", "help3",
"me", "like",
]: "message",
if text in item: "me",
if item[text] != None: ]:
jaString = item[text] if text in item:
if item[text] != None:
# Remove any textwrap jaString = item[text]
if FIXTEXTWRAP == True:
finalJAString = jaString.replace("\n", " ") # Remove any textwrap
if FIXTEXTWRAP == True:
# [Passthrough 1] Pulling From File finalJAString = jaString.replace("\n", " ")
if insertBool is False:
# Append to List and Clear Values # [Passthrough 1] Pulling From File
batch.append(finalJAString) if insertBool is False:
speaker = "" # Append to List and Clear Values
batch.append(finalJAString)
# Translate Batch if Full speaker = ""
if len(batch) == BATCHSIZE:
# Translate # Translate Batch if Full
response = translateAI(batch, textHistory) if len(batch) == BATCHSIZE:
tokens[0] += response[1][0] # Translate
tokens[1] += response[1][1] response = translateAI(batch, textHistory)
translatedBatch = response[0] tokens[0] += response[1][0]
textHistory = translatedBatch[-10:] tokens[1] += response[1][1]
translatedBatch = response[0]
# Set Values textHistory = translatedBatch[-10:]
if len(batch) == len(translatedBatch):
i = batchStartIndex # Set Values
insertBool = True if len(batch) == len(translatedBatch):
i = batchStartIndex
# Mismatch insertBool = True
else:
pbar.write(f"Mismatch: {batchStartIndex} - {i}") # Mismatch
MISMATCH.append(batch) else:
batchStartIndex = i pbar.write(f"Mismatch: {batchStartIndex} - {i}")
batch.clear() MISMATCH.append(batch)
batchStartIndex = i
if insertBool is False: batch.clear()
i += 1
if insertBool is False:
currentGroup = [] i += 1
# [Passthrough 2] Setting Data currentGroup = []
else:
# Get Text # [Passthrough 2] Setting Data
translatedText = translatedBatch[0] else:
# Get Text
# Remove added speaker translatedText = translatedBatch[0]
translatedText = re.sub(r"^.+?:\s", "", translatedText)
# Remove added speaker
# Textwrap translatedText = re.sub(r"^.+?:\s", "", translatedText)
translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
# Textwrap
# Set Text translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
item[text] = translatedText
save_progress_json(data, FILENAME or "output.json") # Set Text
translatedBatch.pop(0) item[text] = translatedText
speaker = "" save_progress_json(data, FILENAME or "output.json")
currentGroup = [] translatedBatch.pop(0)
i += 1 speaker = ""
currentGroup = []
# If Batch is empty. Move on. i += 1
if len(translatedBatch) == 0:
insertBool = False # If Batch is empty. Move on.
batchStartIndex = i if len(translatedBatch) == 0:
batch.clear() insertBool = False
else: batchStartIndex = i
i += 1 batch.clear()
else:
# Translate Batch if not empty and EOF i += 1
if len(batch) != 0 and i >= len(data):
# Translate # Translate Batch if not empty and EOF
response = translateAI(batch, textHistory) if len(batch) != 0 and i >= len(data):
tokens[0] += response[1][0] # Translate
tokens[1] += response[1][1] response = translateAI(batch, textHistory)
translatedBatch = response[0] tokens[0] += response[1][0]
textHistory = translatedBatch[-10:] tokens[1] += response[1][1]
translatedBatch = response[0]
# Set Values textHistory = translatedBatch[-10:]
if len(batch) == len(translatedBatch):
i = batchStartIndex # Set Values
insertBool = True if len(batch) == len(translatedBatch):
i = batchStartIndex
# Mismatch insertBool = True
else:
pbar.write(f"Mismatch: {batchStartIndex} - {i}") # Mismatch
MISMATCH.append(batch) else:
batchStartIndex = i pbar.write(f"Mismatch: {batchStartIndex} - {i}")
batch.clear() MISMATCH.append(batch)
batchStartIndex = i
currentGroup = [] batch.clear()
# After applying a batch, persist progress
save_progress_json(data, FILENAME or "output.json") currentGroup = []
return tokens # After applying a batch, persist progress
save_progress_json(data, FILENAME or "output.json")
# Save some money and enter the character before translation return tokens
def getSpeaker(speaker):
match speaker: # Save some money and enter the character before translation
case "ファイン": def getSpeaker(speaker):
return ["Fine", [0, 0]] match speaker:
case "": case "ファイン":
return ["", [0, 0]] return ["Fine", [0, 0]]
case _: case "":
# Find Speaker return ["", [0, 0]]
for i in range(len(NAMESLIST)): case _:
if speaker == NAMESLIST[i][0]: # Find Speaker
return [NAMESLIST[i][1], [0, 0]] for i in range(len(NAMESLIST)):
if speaker == NAMESLIST[i][0]:
# Translate and Store Speaker return [NAMESLIST[i][1], [0, 0]]
response = translateAI(
f"{speaker}", # Translate and Store Speaker
"Reply with the " + LANGUAGE + " translation of the NPC name.", response = translateAI(
False, f"{speaker}",
) "Reply with the " + LANGUAGE + " translation of the NPC name.",
response[0] = response[0].title() False,
response[0] = response[0].replace("'S", "'s") )
response[0] = response[0].replace("Speaker: ", "") response[0] = response[0].title()
response[0] = response[0].replace("'S", "'s")
# Retry if name doesn't translate for some reason response[0] = response[0].replace("Speaker: ", "")
if re.search(r"([a-zA-Z？?])", response[0]) == None:
response = translateAI( # Retry if name doesn't translate for some reason
f"{speaker}", if re.search(r"([a-zA-Z？?])", response[0]) == None:
"Reply with the " + LANGUAGE + " translation of the NPC name.", response = translateAI(
False, f"{speaker}",
) "Reply with the " + LANGUAGE + " translation of the NPC name.",
response[0] = response[0].title() False,
response[0] = response[0].replace("'S", "'s") )
response[0] = response[0].title()
speakerList = [speaker, response[0]] response[0] = response[0].replace("'S", "'s")
NAMESLIST.append(speakerList)
return response speakerList = [speaker, response[0]]
return [speaker, [0, 0]] NAMESLIST.append(speakerList)
return response
def translateAI(text, history, history_ctx=None): return [speaker, [0, 0]]
"""
Legacy wrapper function for the new shared translation utility. def translateAI(text, history, history_ctx=None):
This maintains compatibility with existing code while using the new shared implementation. """
""" Legacy wrapper function for the new shared translation utility.
global PBAR, MISMATCH, FILENAME This maintains compatibility with existing code while using the new shared implementation.
"""
# Update config estimate mode based on global ESTIMATE global PBAR, MISMATCH, FILENAME
TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
# Update config estimate mode based on global ESTIMATE
# Call the new shared translation function TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
return sharedtranslateAI(
text=text, # Call the new shared translation function
history=history, return sharedtranslateAI(
config=TRANSLATION_CONFIG, text=text,
filename=FILENAME, history=history,
pbar=PBAR, config=TRANSLATION_CONFIG,
lock=LOCK, filename=FILENAME,
mismatchList=MISMATCH pbar=PBAR,
) lock=LOCK,
mismatchList=MISMATCH
)

764
modules/main.py
View file

@ -1,381 +1,385 @@
import sys import sys
import os import os
import traceback import traceback
import datetime import datetime
from pathlib import Path from pathlib import Path
from concurrent.futures import ThreadPoolExecutor, as_completed from concurrent.futures import ThreadPoolExecutor, as_completed
from colorama import Fore from colorama import Fore
from tqdm import tqdm from tqdm import tqdm
from dotenv import load_dotenv from dotenv import load_dotenv
# This needs to be before the module imports as some of them currently try to read and use some of these values # This needs to be before the module imports as some of them currently try to read and use some of these values
# upon import, in which case if they are unset the script will crash before we can output these messages. # upon import, in which case if they are unset the script will crash before we can output these messages.
load_dotenv() load_dotenv()
_missing_envs = [ from util.paths import migrate_root_data_files, ensure_vocab_file
env for env in [
"api", migrate_root_data_files()
"key", ensure_vocab_file()
"model", _missing_envs = [
"language", env for env in [
"timeout", "api",
"fileThreads", "key",
"threads", "model",
"width", "language",
"listWidth", "timeout",
] "fileThreads",
if os.getenv(env) is None or str(os.getenv(env))[:1] == "<" "threads",
] "width",
if _missing_envs: "listWidth",
names = ", ".join(_missing_envs) ]
tqdm.write( if os.getenv(env) is None or str(os.getenv(env))[:1] == "<"
Fore.RED ]
+ f"Missing required environment variable(s): {names}. " if _missing_envs:
+ "Set them in a .env file (see .env.example)." names = ", ".join(_missing_envs)
) tqdm.write(
Fore.RED
from modules.rpgmakermvmz import handleMVMZ, setSpeakerParseMode as setSpeakerParseMVMZ, finalizeSpeakerParse as finalizeSpeakerParseMVMZ + f"Missing required environment variable(s): {names}. "
from modules.csv import handleCSV + "Set them in a .env file (see .env.example)."
from modules.tyrano import handleTyrano )
from modules.kirikiri import handleKirikiri
from modules.json import handleJSON from modules.rpgmakermvmz import handleMVMZ, setSpeakerParseMode as setSpeakerParseMVMZ, finalizeSpeakerParse as finalizeSpeakerParseMVMZ
from modules.lune import handleLune from modules.csv import handleCSV
from modules.yuris import handleYuris from modules.tyrano import handleTyrano
from modules.nscript import handleOnscripter from modules.kirikiri import handleKirikiri
from modules.wolf import handleWOLF from modules.json import handleJSON
from modules.wolf2 import handleWOLF2 from modules.lune import handleLune
from modules.regex import handleRegex from modules.yuris import handleYuris
from modules.text import handleText from modules.nscript import handleOnscripter
from modules.renpy import handleRenpy from modules.wolf import handleWOLF
from modules.unity import handleUnity from modules.wolf2 import handleWOLF2
from modules.images import handleImages from modules.regex import handleRegex
from modules.rpgmakerplugin import handlePlugin from modules.text import handleText
from modules.srpg import handleSRPG from modules.renpy import handleRenpy
from modules.aquedi4 import handleAquedi4 from modules.unity import handleUnity
from modules.images import handleImages
# For GPT4 rate limit will be hit if you have more than 1 thread. from modules.rpgmakerplugin import handlePlugin
# 1 Thread for each file. Controls how many files are worked on at once. from modules.srpg import handleSRPG
THREADS = int(os.getenv("fileThreads")) from modules.aquedi4 import handleAquedi4
# [Display name, file extension, handle function] # For GPT4 rate limit will be hit if you have more than 1 thread.
MODULES = [ # 1 Thread for each file. Controls how many files are worked on at once.
["RPGMaker MV/MZ", ["json"], handleMVMZ], THREADS = int(os.getenv("fileThreads"))
["RPGMaker Plugins", ["js", "rb"], handlePlugin],
["CSV (From Translator++)", ["csv"], handleCSV], # [Display name, file extension, handle function]
["Tyrano", ["ks"], handleTyrano], MODULES = [
["Kirikiri", ["ks", "tjs", "ssd", "asd"], handleKirikiri], ["RPGMaker MV/MZ", ["json"], handleMVMZ],
["JSON", ["json"], handleJSON], ["RPGMaker Plugins", ["js", "rb"], handlePlugin],
["Lune", ["json"], handleLune], ["CSV (From Translator++)", ["csv"], handleCSV],
["Yuris", ["json"], handleYuris], ["Tyrano", ["ks"], handleTyrano],
["NScript", ["txt"], handleOnscripter], ["Kirikiri", ["ks", "tjs", "ssd", "asd"], handleKirikiri],
["Wolf", ["json"], handleWOLF], ["JSON", ["json"], handleJSON],
["Wolf", ["txt"], handleWOLF2], ["Lune", ["json"], handleLune],
["Regex", ["txt", "json", "script", "csv"], handleRegex], ["Yuris", ["json"], handleYuris],
["Text", ["txt", "srt"], handleText], ["NScript", ["txt"], handleOnscripter],
["Renpy", ["rpy"], handleRenpy], ["Wolf", ["json"], handleWOLF],
["Unity", ["txt"], handleUnity], ["Wolf", ["txt"], handleWOLF2],
["SRPG Studio", ["json"], handleSRPG], ["Regex", ["txt", "json", "script", "csv"], handleRegex],
["Images", [""], handleImages], ["Text", ["txt", "srt"], handleText],
["Aquedi4 Prepared JSON", [".json"], handleAquedi4], ["Renpy", ["rpy"], handleRenpy],
] ["Unity", ["txt"], handleUnity],
["SRPG Studio", ["json"], handleSRPG],
# Info Message ["Images", [""], handleImages],
tqdm.write( ["Aquedi4 Prepared JSON", [".json"], handleAquedi4],
Fore.CYAN ]
+ "-Dazed MTL Tool -"
+ Fore.RESET, # Info Message
end="\n\n", tqdm.write(
) Fore.CYAN
+ "-Dazed MTL Tool -"
+ Fore.RESET,
def main(): end="\n\n",
from util.translation import clear_cache )
estimate = ""
batch_mode = False def main():
speaker_parse = False # Deferred until after engine select from util.translation import clear_cache
while estimate == "":
estimate = input("Select Mode:\n\n 1. Translate\n 2. Estimate\n 3. Batch Translate (Anthropic Batches API, 50% off)\n") estimate = ""
match estimate: batch_mode = False
case "1": speaker_parse = False # Deferred until after engine select
estimate = False while estimate == "":
case "2": estimate = input("Select Mode:\n\n 1. Translate\n 2. Estimate\n 3. Batch Translate (Anthropic Batches API, 50% off)\n")
estimate = True match estimate:
case "3": case "1":
estimate = False estimate = False
batch_mode = True case "2":
case _: estimate = True
estimate = "" case "3":
estimate = False
resume_state = None batch_mode = True
if batch_mode: case _:
from util.translation import isClaudeNative, batchRunState estimate = ""
if not isClaudeNative(os.getenv("model", "")):
tqdm.write( resume_state = None
Fore.RED if batch_mode:
+ "Batch Translate requires a Claude model with the 'api' env var unset or pointing at anthropic.com." from util.translation import isClaudeNative, batchRunState
+ Fore.RESET if not isClaudeNative(os.getenv("model", "")):
) tqdm.write(
return Fore.RED
# An interrupted batch run can be resumed instead of re-collecting + "Batch Translate requires a Claude model with the 'api' env var unset or pointing at anthropic.com."
# (a second submission would be billed again). + Fore.RESET
resume_state = batchRunState() )
if resume_state: return
confirm = "" # An interrupted batch run can be resumed instead of re-collecting
while confirm not in ("y", "n"): # (a second submission would be billed again).
confirm = input(f"A previous batch run was interrupted ({resume_state}). Resume it? (y/n)\n").strip().lower() resume_state = batchRunState()
if confirm == "n": if resume_state:
resume_state = None confirm = ""
while confirm not in ("y", "n"):
# Clear the translation cache at the start of the run. Kept when resuming a confirm = input(f"A previous batch run was interrupted ({resume_state}). Resume it? (y/n)\n").strip().lower()
# batch so names translated during collect stay consistent with the queued if confirm == "n":
# payloads in the consume pass. resume_state = None
if not resume_state:
clear_cache() # Clear the translation cache at the start of the run. Kept when resuming a
# batch so names translated during collect stay consistent with the queued
version = "" # payloads in the consume pass.
while True: if not resume_state:
tqdm.write("Select game engine:\n") clear_cache()
for position, module in enumerate(MODULES):
tqdm.write(f"{str(position + 1).rjust(2)}. {module[0]} (.{module[1]})") version = ""
version = input() while True:
try: tqdm.write("Select game engine:\n")
version = int(version) - 1 for position, module in enumerate(MODULES):
except: tqdm.write(f"{str(position + 1).rjust(2)}. {module[0]} (.{module[1]})")
continue version = input()
if version in range(len(MODULES)): try:
break version = int(version) - 1
except:
totalCost = ( continue
Fore.RED if version in range(len(MODULES)):
+ "Translation module didn't return the total cost. Make sure the \ break
files to translate are in the /files folder and that you picked the right game engine."
) totalCost = (
Fore.RED
# If translating RPGMaker MV/MZ, prompt for speaker parse mode + "Translation module didn't return the total cost. Make sure the \
speaker_parse = False files to translate are in the /files folder and that you picked the right game engine."
if version == 0 and not estimate and not batch_mode: )
sub = ""
while sub == "": # If translating RPGMaker MV/MZ, prompt for speaker parse mode
sub = input("RPGMaker MV/MZ options:\n\n 1. Standard Translate\n 2. Parse Speakers (collect speaker names only)\n") speaker_parse = False
match sub: if version == 0 and not estimate and not batch_mode:
case "1": sub = ""
speaker_parse = False while sub == "":
case "2": sub = input("RPGMaker MV/MZ options:\n\n 1. Standard Translate\n 2. Parse Speakers (collect speaker names only)\n")
speaker_parse = True match sub:
case _: case "1":
sub = "" speaker_parse = False
if speaker_parse: case "2":
setSpeakerParseMVMZ(True) speaker_parse = True
case _:
# Open File (Threads) - recursively walk 'files' and preserve directory structure sub = ""
# Prepare per-run log file so CLI runs also write to a run-specific history file if speaker_parse:
try: setSpeakerParseMVMZ(True)
hist_dir = Path("log") / "history"
hist_dir.mkdir(parents=True, exist_ok=True) # Open File (Threads) - recursively walk 'files' and preserve directory structure
# Prepare per-run log file so CLI runs also write to a run-specific history file
# Clean up old log files, keeping only the 10 most recent try:
try: hist_dir = Path("log") / "history"
log_files = sorted(hist_dir.glob("translationHistory_*.txt"), key=lambda p: p.stat().st_mtime, reverse=True) hist_dir.mkdir(parents=True, exist_ok=True)
# Keep only the 10 most recent, delete the rest
for old_log in log_files[10:]: # Clean up old log files, keeping only the 10 most recent
try: try:
old_log.unlink() log_files = sorted(hist_dir.glob("translationHistory_*.txt"), key=lambda p: p.stat().st_mtime, reverse=True)
except Exception: # Keep only the 10 most recent, delete the rest
pass for old_log in log_files[10:]:
except Exception: try:
pass old_log.unlink()
except Exception:
fname = datetime.datetime.now().strftime("translationHistory_%Y%m%d_%H%M%S.txt") pass
run_log_path = hist_dir / fname except Exception:
# Don't create the file yet - it will be created when first log is written pass
# Store the path in environment variable
try: fname = datetime.datetime.now().strftime("translationHistory_%Y%m%d_%H%M%S.txt")
os.environ['TRANSLATION_RUN_LOG'] = str(run_log_path) run_log_path = hist_dir / fname
except Exception: # Don't create the file yet - it will be created when first log is written
pass # Store the path in environment variable
try:
# Try to create a hard link from legacy path to this run file for compatibility os.environ['TRANSLATION_RUN_LOG'] = str(run_log_path)
# This will be created when the run_log_path file is first written to except Exception:
legacy = Path("log") / "translationHistory.txt" pass
try:
if legacy.exists(): # Try to create a hard link from legacy path to this run file for compatibility
try: # This will be created when the run_log_path file is first written to
legacy.unlink() legacy = Path("log") / "translationHistory.txt"
except Exception: try:
pass if legacy.exists():
except Exception: try:
pass legacy.unlink()
except Exception: except Exception:
pass pass
except Exception:
def runFiles(estimate): pass
runCost = totalCost except Exception:
# Use single worker for estimate mode to prevent race conditions pass
max_workers = 1 if estimate else THREADS
with ThreadPoolExecutor(max_workers=max_workers) as executor: def runFiles(estimate):
futures = [] runCost = totalCost
files_root = "files" # Use single worker for estimate mode to prevent race conditions
max_workers = 1 if estimate else THREADS
# Special-case: Images engine expects a folder, not a file; schedule per directory containing assets with ThreadPoolExecutor(max_workers=max_workers) as executor:
if MODULES[version][0] == "Images": futures = []
for root, dirs, filenames in os.walk(files_root): files_root = "files"
# Skip hidden/system directories
dirs[:] = [d for d in dirs if d not in {".git", "__pycache__"}] # Special-case: Images engine expects a folder, not a file; schedule per directory containing assets
if MODULES[version][0] == "Images":
# Skip the root 'files' itself to avoid processing everything twice for root, dirs, filenames in os.walk(files_root):
# We'll still allow scheduling for root if it contains assets # Skip hidden/system directories
dirs[:] = [d for d in dirs if d not in {".git", "__pycache__"}]
# Only schedule directories that contain potential assets
has_assets = any(fn.lower().endswith((".png", ".txt")) for fn in filenames) # Skip the root 'files' itself to avoid processing everything twice
if not has_assets: # We'll still allow scheduling for root if it contains assets
continue
# Only schedule directories that contain potential assets
# Compute relative directory path and ensure translated mirror exists has_assets = any(fn.lower().endswith((".png", ".txt")) for fn in filenames)
rel_dir = os.path.relpath(root, files_root).replace(os.sep, "/") if not has_assets:
if rel_dir == ".": continue
# Represent root as empty string so handler creates files under translated/ directly
rel_dir = "" # Compute relative directory path and ensure translated mirror exists
try: rel_dir = os.path.relpath(root, files_root).replace(os.sep, "/")
target_dir = os.path.join("translated", rel_dir.replace("/", os.sep)) if rel_dir else "translated" if rel_dir == ".":
os.makedirs(target_dir, exist_ok=True) # Represent root as empty string so handler creates files under translated/ directly
except Exception: rel_dir = ""
pass try:
target_dir = os.path.join("translated", rel_dir.replace("/", os.sep)) if rel_dir else "translated"
futures.append( os.makedirs(target_dir, exist_ok=True)
executor.submit(MODULES[version][2], rel_dir, estimate) except Exception:
) pass
else:
# Gather all candidate files recursively futures.append(
for root, dirs, filenames in os.walk(files_root): executor.submit(MODULES[version][2], rel_dir, estimate)
# Skip hidden/system directories if any )
dirs[:] = [d for d in dirs if d not in {".git", "__pycache__"}] else:
# Gather all candidate files recursively
for fname in filenames: for root, dirs, filenames in os.walk(files_root):
if fname == ".gitkeep": # Skip hidden/system directories if any
continue dirs[:] = [d for d in dirs if d not in {".git", "__pycache__"}]
abs_path = os.path.join(root, fname) for fname in filenames:
# Build relative path from 'files' root using POSIX-style separators so handlers can do 'files/' + rel if fname == ".gitkeep":
rel_path = os.path.relpath(abs_path, files_root) continue
rel_path_posix = rel_path.replace(os.sep, "/")
abs_path = os.path.join(root, fname)
# Check extension match for the selected module version # Build relative path from 'files' root using POSIX-style separators so handlers can do 'files/' + rel
for m in MODULES[version][1]: rel_path = os.path.relpath(abs_path, files_root)
if rel_path_posix.endswith(m): rel_path_posix = rel_path.replace(os.sep, "/")
# Ensure the corresponding directory exists under 'translated'
rel_dir = os.path.dirname(rel_path_posix) # Check extension match for the selected module version
if rel_dir: for m in MODULES[version][1]:
try: if rel_path_posix.endswith(m):
os.makedirs(os.path.join("translated", rel_dir.replace("/", os.sep)), exist_ok=True) # Ensure the corresponding directory exists under 'translated'
except Exception: rel_dir = os.path.dirname(rel_path_posix)
# Best-effort; handler may attempt write and fail if permissions are insufficient if rel_dir:
pass try:
os.makedirs(os.path.join("translated", rel_dir.replace("/", os.sep)), exist_ok=True)
futures.append( except Exception:
executor.submit(MODULES[version][2], rel_path_posix, estimate) # Best-effort; handler may attempt write and fail if permissions are insufficient
) pass
break # Avoid double-adding if multiple ext entries match
futures.append(
for future in as_completed(futures): executor.submit(MODULES[version][2], rel_path_posix, estimate)
try: )
runCost = future.result() break # Avoid double-adding if multiple ext entries match
except Exception as e:
tracebackLineNo = str(traceback.extract_tb(sys.exc_info()[2])[-1].lineno) for future in as_completed(futures):
tqdm.write(Fore.RED + str(e) + "|" + tracebackLineNo + Fore.RESET) try:
return runCost runCost = future.result()
except Exception as e:
if batch_mode: tracebackLineNo = str(traceback.extract_tb(sys.exc_info()[2])[-1].lineno)
from util.translation import ( tqdm.write(Fore.RED + str(e) + "|" + tracebackLineNo + Fore.RESET)
set_batch_phase, return runCost
clearBatchFiles,
pendingBatchRequests, if batch_mode:
estimateBatchCost, from util.translation import (
runTranslationBatches, set_batch_phase,
) clearBatchFiles,
poll = int(os.getenv("batchPollInterval", "60") or 60) pendingBatchRequests,
run_consume = True estimateBatchCost,
runTranslationBatches,
if resume_state is None: )
clearBatchFiles() poll = int(os.getenv("batchPollInterval", "60") or 60)
run_consume = True
# Pass 1 — queue dialogue for the batch; speaker/variable strings still hit live API.
tqdm.write(Fore.CYAN + "[BATCH] Pass 1/2: collecting requests..." + Fore.RESET) if resume_state is None:
tqdm.write( clearBatchFiles()
Fore.YELLOW
+ "[BATCH] Note: speaker names and similar short strings translate at live " # Pass 1 — queue dialogue for the batch; speaker/variable strings still hit live API.
"API rates during collect (dialogue is batched after you confirm)." tqdm.write(Fore.CYAN + "[BATCH] Pass 1/2: collecting requests..." + Fore.RESET)
+ Fore.RESET tqdm.write(
) Fore.YELLOW
set_batch_phase("collect") + "[BATCH] Note: speaker names and similar short strings translate at live "
try: "API rates during collect (dialogue is batched after you confirm)."
totalCost = runFiles(False) + Fore.RESET
finally: )
set_batch_phase(None) set_batch_phase("collect")
try:
if pendingBatchRequests() == 0: totalCost = runFiles(False)
tqdm.write("[BATCH] No requests queued — nothing needed the API.") finally:
run_consume = False set_batch_phase(None)
else:
estimateBatchCost() if pendingBatchRequests() == 0:
confirm = "" tqdm.write("[BATCH] No requests queued — nothing needed the API.")
while confirm not in ("y", "n"): run_consume = False
confirm = input("Submit batch? (y/n)\n").strip().lower() else:
if confirm == "n": estimateBatchCost()
tqdm.write("[BATCH] Not submitted. The queue is kept in log/batch_requests.json.") confirm = ""
return while confirm not in ("y", "n"):
runTranslationBatches(poll) confirm = input("Submit batch? (y/n)\n").strip().lower()
elif resume_state == "submitted": if confirm == "n":
tqdm.write(Fore.CYAN + "[BATCH] Resuming the submitted batch..." + Fore.RESET) tqdm.write("[BATCH] Not submitted. The queue is kept in log/batch_requests.json.")
runTranslationBatches(poll) return
else: # "fetched" — results already downloaded, just write the files runTranslationBatches(poll)
tqdm.write(Fore.CYAN + "[BATCH] Resuming from fetched results..." + Fore.RESET) elif resume_state == "submitted":
tqdm.write(Fore.CYAN + "[BATCH] Resuming the submitted batch..." + Fore.RESET)
if run_consume: runTranslationBatches(poll)
# Pass 2 — write the translated files from the fetched results. else: # "fetched" — results already downloaded, just write the files
# Anything the batch missed falls back to the live API. tqdm.write(Fore.CYAN + "[BATCH] Resuming from fetched results..." + Fore.RESET)
tqdm.write(Fore.CYAN + "[BATCH] Pass 2/2: writing translated files..." + Fore.RESET)
set_batch_phase("consume") if run_consume:
try: # Pass 2 — write the translated files from the fetched results.
totalCost = runFiles(False) # Anything the batch missed falls back to the live API.
finally: tqdm.write(Fore.CYAN + "[BATCH] Pass 2/2: writing translated files..." + Fore.RESET)
set_batch_phase(None) set_batch_phase("consume")
else: try:
totalCost = runFiles(estimate) totalCost = runFiles(False)
finally:
# Finalize speaker parse mode by writing collected speakers to vocab set_batch_phase(None)
if speaker_parse: else:
finalizeSpeakerParseMVMZ() totalCost = runFiles(estimate)
# Delete Tmp Files # Finalize speaker parse mode by writing collected speakers to vocab
if os.path.isfile("csv.tmp"): if speaker_parse:
os.remove("csv.tmp") finalizeSpeakerParseMVMZ()
# Sweep any leftover temp files in translated/ # Delete Tmp Files
try: if os.path.isfile("csv.tmp"):
translated_dir = os.path.join("translated") os.remove("csv.tmp")
if os.path.isdir(translated_dir):
for fname in os.listdir(translated_dir): # Sweep any leftover temp files in translated/
if fname.endswith(".tmp"): try:
fpath = os.path.join(translated_dir, fname) translated_dir = os.path.join("translated")
try: if os.path.isdir(translated_dir):
os.remove(fpath) for fname in os.listdir(translated_dir):
except Exception: if fname.endswith(".tmp"):
# Best-effort cleanup; ignore files locked by other processes fpath = os.path.join(translated_dir, fname)
pass try:
except Exception: os.remove(fpath)
pass except Exception:
# Best-effort cleanup; ignore files locked by other processes
# Finish pass
if totalCost != "Fail": except Exception:
# if estimate is False: pass
# This is to encourage people to grab what's in /translated instead
# deleteFolderFiles("files") # Finish
if totalCost != "Fail":
tqdm.write(str(totalCost)) # if estimate is False:
# This is to encourage people to grab what's in /translated instead
# deleteFolderFiles("files")
def deleteFolderFiles(folderPath):
for filename in os.listdir(folderPath): tqdm.write(str(totalCost))
file_path = os.path.join(folderPath, filename)
if file_path.endswith((".json", ".ks")):
def deleteFolderFiles(folderPath):
for filename in os.listdir(folderPath):
file_path = os.path.join(folderPath, filename)
if file_path.endswith((".json", ".ks")):
os.remove(file_path) os.remove(file_path)

854
modules/nscript.py
View file

@ -1,426 +1,428 @@
# Libraries # Libraries
import json import json
import os import os
import re import re
import util.dazedwrap as dazedwrap import util.dazedwrap as dazedwrap
import threading import threading
import time import time
import traceback import traceback
import tiktoken import tiktoken
from pathlib import Path from pathlib import Path
from colorama import Fore from colorama import Fore
from dotenv import load_dotenv from dotenv import load_dotenv
from retry import retry from retry import retry
from tqdm import tqdm from tqdm import tqdm
from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
import tempfile import tempfile
# Globals # Globals
MODEL = os.getenv("model") MODEL = os.getenv("model")
TIMEOUT = int(os.getenv("timeout")) TIMEOUT = int(os.getenv("timeout"))
LANGUAGE = os.getenv("language").capitalize() LANGUAGE = os.getenv("language").capitalize()
PROMPT = Path("prompt.txt").read_text(encoding="utf-8") from util.paths import PROMPT_PATH, VOCAB_PATH
VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
LOCK = threading.Lock() PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
WIDTH = int(os.getenv("width")) VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
LISTWIDTH = int(os.getenv("listWidth")) LOCK = threading.Lock()
NOTEWIDTH = 70 WIDTH = int(os.getenv("width"))
MAXHISTORY = 10 LISTWIDTH = int(os.getenv("listWidth"))
ESTIMATE = "" NOTEWIDTH = 70
TOKENS = [0, 0] MAXHISTORY = 10
NAMESLIST = [] ESTIMATE = ""
NAMES = False # Output a list of all the character names found TOKENS = [0, 0]
BRFLAG = False # If the game uses <br> instead NAMESLIST = []
FIXTEXTWRAP = True # Overwrites textwrap NAMES = False # Output a list of all the character names found
IGNORETLTEXT = False # Ignores all translated text. BRFLAG = False # If the game uses <br> instead
MISMATCH = [] # Lists files that throw a mismatch error (Length of GPT list response is wrong) FIXTEXTWRAP = True # Overwrites textwrap
CONVERTTOWIDE = False # Default (False) IGNORETLTEXT = False # Ignores all translated text.
MISMATCH = [] # Lists files that throw a mismatch error (Length of GPT list response is wrong)
# tqdm Globals CONVERTTOWIDE = False # Default (False)
BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
POSITION = 0 # tqdm Globals
LEAVE = False BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
PBAR = None POSITION = 0
FILENAME = None LEAVE = False
PBAR = None
# Full Width FILENAME = None
ascii_to_wide = dict((i, chr(i + 0xFEE0)) for i in range(0x21, 0x7F))
ascii_to_wide.update({0x20: "\u3000", 0x2D: "\u2212"}) # space and minus # Full Width
wide_to_ascii = dict((i, chr(i - 0xFEE0)) for i in range(0xFF01, 0xFF5F)) ascii_to_wide = dict((i, chr(i + 0xFEE0)) for i in range(0x21, 0x7F))
wide_to_ascii.update({0x3000: " ", 0x2212: "-"}) # space and minus ascii_to_wide.update({0x20: "\u3000", 0x2D: "\u2212"}) # space and minus
wide_to_ascii = dict((i, chr(i - 0xFEE0)) for i in range(0xFF01, 0xFF5F))
# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex wide_to_ascii.update({0x3000: " ", 0x2212: "-"}) # space and minus
LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
# Get pricing configuration based on the model LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
PRICING_CONFIG = getPricingConfig(MODEL)
INPUTAPICOST = PRICING_CONFIG["inputAPICost"] # Get pricing configuration based on the model
OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"] PRICING_CONFIG = getPricingConfig(MODEL)
BATCHSIZE = PRICING_CONFIG["batchSize"] INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"] OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
BATCHSIZE = PRICING_CONFIG["batchSize"]
# Initialize Translation Config FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
TRANSLATION_CONFIG = TranslationConfig(
model=MODEL, # Initialize Translation Config
language=LANGUAGE, TRANSLATION_CONFIG = TranslationConfig(
prompt=PROMPT, model=MODEL,
vocab=VOCAB, language=LANGUAGE,
langRegex=LANGREGEX, prompt=PROMPT,
batchSize=BATCHSIZE, vocab=VOCAB,
maxHistory=MAXHISTORY, langRegex=LANGREGEX,
estimateMode=False # Will be set dynamically based on ESTIMATE batchSize=BATCHSIZE,
) maxHistory=MAXHISTORY,
LEAVE = False estimateMode=False # Will be set dynamically based on ESTIMATE
)
def handleOnscripter(filename, estimate): LEAVE = False
global ESTIMATE, FILENAME
ESTIMATE = estimate def handleOnscripter(filename, estimate):
FILENAME = filename global ESTIMATE, FILENAME
ESTIMATE = estimate
if ESTIMATE: FILENAME = filename
start = time.time()
translatedData = openFiles(filename) if ESTIMATE:
start = time.time()
# Print Result translatedData = openFiles(filename)
end = time.time()
tqdm.write(getResultString(translatedData, end - start, filename)) # Print Result
with LOCK: end = time.time()
TOKENS[0] += translatedData[1][0] tqdm.write(getResultString(translatedData, end - start, filename))
TOKENS[1] += translatedData[1][1] with LOCK:
TOKENS[0] += translatedData[1][0]
# Print Total TOKENS[1] += translatedData[1][1]
totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
# Print Total
# Print any errors on maps totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
if len(MISMATCH) > 0:
return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET # Print any errors on maps
else: if len(MISMATCH) > 0:
return totalString return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
else:
else: return totalString
try:
with open("translated/" + filename, "w", encoding="cp932", errors="ignore") as outFile: else:
start = time.time() try:
translatedData = openFiles(filename) with open("translated/" + filename, "w", encoding="cp932", errors="ignore") as outFile:
start = time.time()
# Print Result translatedData = openFiles(filename)
end = time.time()
outFile.writelines(translatedData[0]) # Print Result
tqdm.write(getResultString(translatedData, end - start, filename)) end = time.time()
with LOCK: outFile.writelines(translatedData[0])
TOKENS[0] += translatedData[1][0] tqdm.write(getResultString(translatedData, end - start, filename))
TOKENS[1] += translatedData[1][1] with LOCK:
except Exception: TOKENS[0] += translatedData[1][0]
traceback.print_exc() TOKENS[1] += translatedData[1][1]
return "Fail" except Exception:
traceback.print_exc()
return getResultString(["", TOKENS, None], end - start, "TOTAL") return "Fail"
return getResultString(["", TOKENS, None], end - start, "TOTAL")
def getResultString(translatedData, translationTime, filename):
# File Print String
cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL) def getResultString(translatedData, translationTime, filename):
totalTokenstring = ( # File Print String
Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]" cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
"[Output: " totalTokenstring = (
+ str(translatedData[1][1]) Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
+ "]" "[Cost: ${:,.4f}".format(cost) "[Output: "
+ "]" + str(translatedData[1][1])
) + "]" "[Cost: ${:,.4f}".format(cost)
timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]" + "]"
)
if translatedData[2] == None: timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
# Success
return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET if translatedData[2] == None:
# Success
else: return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
# Fail
try: else:
raise translatedData[2] # Fail
except Exception as e: try:
traceback.print_exc() raise translatedData[2]
errorString = str(e) + Fore.RED except Exception as e:
return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET traceback.print_exc()
errorString = str(e) + Fore.RED
return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
def openFiles(filename):
with open("files/" + filename, "r", encoding="cp932") as readFile:
translatedData = parseOnscripter(readFile, filename) def openFiles(filename):
with open("files/" + filename, "r", encoding="cp932") as readFile:
# Delete lines marked for deletion translatedData = parseOnscripter(readFile, filename)
finalData = []
for line in translatedData[0]: # Delete lines marked for deletion
if line != "\\d\n": finalData = []
finalData.append(line) for line in translatedData[0]:
translatedData[0] = finalData if line != "\\d\n":
finalData.append(line)
return translatedData translatedData[0] = finalData
return translatedData
def parseOnscripter(readFile, filename):
totalTokens = [0, 0]
def parseOnscripter(readFile, filename):
# Read File into data totalTokens = [0, 0]
data = readFile.readlines()
# Read File into data
# Create Progress Bar data = readFile.readlines()
with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
pbar.desc = filename # Create Progress Bar
with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
try: pbar.desc = filename
result = translateOnscripter(data, pbar, filename, [])
totalTokens[0] += result[0] try:
totalTokens[1] += result[1] result = translateOnscripter(data, pbar, filename, [])
except Exception as e: totalTokens[0] += result[0]
traceback.print_exc() totalTokens[1] += result[1]
return [data, totalTokens, e] except Exception as e:
return [data, totalTokens, None] traceback.print_exc()
return [data, totalTokens, e]
return [data, totalTokens, None]
def save_progress_lines(lines, filename, encoding="cp932"):
"""Atomically save current line-based translation progress."""
try: def save_progress_lines(lines, filename, encoding="cp932"):
if ESTIMATE: """Atomically save current line-based translation progress."""
return try:
os.makedirs("translated", exist_ok=True) if ESTIMATE:
tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated") return
try: os.makedirs("translated", exist_ok=True)
with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file: tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
tmp_file.writelines(lines) try:
os.replace(tmp_path, os.path.join("translated", filename)) with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file:
finally: tmp_file.writelines(lines)
if os.path.exists(tmp_path): os.replace(tmp_path, os.path.join("translated", filename))
try: finally:
os.remove(tmp_path) if os.path.exists(tmp_path):
except OSError: try:
pass os.remove(tmp_path)
except Exception: except OSError:
traceback.print_exc() pass
except Exception:
traceback.print_exc()
def translateOnscripter(data, pbar, filename, translatedList):
stringList = []
currentGroup = [] def translateOnscripter(data, pbar, filename, translatedList):
tokens = [0, 0] stringList = []
speaker = "" currentGroup = []
voice = False tokens = [0, 0]
global LOCK, ESTIMATE, PBAR speaker = ""
PBAR = pbar voice = False
i = 0 global LOCK, ESTIMATE, PBAR
PBAR = pbar
# Dialogue i = 0
while i < len(data):
# Lines # Dialogue
regex = r"^([\u3000「（【[][^\n]+)" while i < len(data):
match = re.search(regex, data[i]) # Lines
if match != None and match.group(1) != "": regex = r"^([\u3000「（【[][^\n]+)"
originalString = match.group(1) match = re.search(regex, data[i])
# Pass 1 if match != None and match.group(1) != "":
if translatedList == []: originalString = match.group(1)
# Grab Consecutive Strings # Pass 1
jaString = match.group(1) if translatedList == []:
while len(data) > i + 1 and re.match(regex, data[i + 1]): # Grab Consecutive Strings
data[i] = "" jaString = match.group(1)
i += 1 while len(data) > i + 1 and re.match(regex, data[i + 1]):
jaString = f"{jaString} {data[i]}" data[i] = ""
i += 1
# Convert from Wide jaString = f"{jaString} {data[i]}"
jaString = jaString.translate(wide_to_ascii)
# Convert from Wide
# Remove any textwrap and \u3000 and \ jaString = jaString.translate(wide_to_ascii)
jaString = jaString.replace("\n", "")
jaString = jaString.replace("\u3000", "") # Remove any textwrap and \u3000 and \
jaString = jaString.replace("\\", "") jaString = jaString.replace("\n", "")
jaString = jaString.replace("　＞", ")") jaString = jaString.replace("\u3000", "")
jaString = jaString.replace("＜　", "(") jaString = jaString.replace("\\", "")
jaString = jaString.replace("　＞", ")")
# Remove Furigana jaString = jaString.replace("＜　", "(")
furiMatch = re.findall(r"({(.+?)\/(.+?)})", jaString)
if furiMatch: # Remove Furigana
for match in furiMatch: furiMatch = re.findall(r"({(.+?)\/(.+?)})", jaString)
jaString = jaString.replace(match[0], match[2]) if furiMatch:
for match in furiMatch:
# Add String jaString = jaString.replace(match[0], match[2])
stringList.append(jaString.strip())
# Add String
# Pass 2 stringList.append(jaString.strip())
else:
# Get Text # Pass 2
if translatedList: else:
# Grab and Pop # Get Text
translatedText = translatedList[0] if translatedList:
translatedList.pop(0) # Grab and Pop
translatedText = translatedList[0]
# Set to None if empty list translatedList.pop(0)
if len(translatedList) <= 0:
translatedList = None # Set to None if empty list
if len(translatedList) <= 0:
# Textwrap & Other Text translatedList = None
translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
translatedText = translatedText.replace("\n", "\n\u3000") # Textwrap & Other Text
translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
# Split the string into lines translatedText = translatedText.replace("\n", "\n\u3000")
lines = translatedText.split("\n")
# Split the string into lines
# Add a backslash after every 3rd line lines = translatedText.split("\n")
j = 0
while j < len(lines): # Add a backslash after every 3rd line
if j == 4: j = 0
lines[j - 1] = f"{lines[j-1]}\\" while j < len(lines):
lines[j] = f"\n{lines[j]}" if j == 4:
j += 1 lines[j - 1] = f"{lines[j-1]}\\"
lines[j] = f"\n{lines[j]}"
# Join the lines back into a single string j += 1
translatedText = "\n".join(lines)
# Join the lines back into a single string
# Remove Double Spaces translatedText = "\n".join(lines)
translatedText = translatedText.replace(" ", " ")
# Remove Double Spaces
# Convert to Wide translatedText = translatedText.replace(" ", " ")
if CONVERTTOWIDE:
translatedText = translatedText.translate(ascii_to_wide) # Convert to Wide
if CONVERTTOWIDE:
# Fix Formatting translatedText = translatedText.translate(ascii_to_wide)
translatedText = fixText(translatedText)
# Fix Formatting
# Set Data translatedText = fixText(translatedText)
data[i] = data[i].replace(originalString, f"{translatedText}")
save_progress_lines(data, filename) # Set Data
i += 1 data[i] = data[i].replace(originalString, f"{translatedText}")
save_progress_lines(data, filename)
# Choices i += 1
elif "csel" in data[i] and translatedList != []:
choiceList = [] # Choices
jaString = data[i] elif "csel" in data[i] and translatedList != []:
choiceList = []
choiceList = re.findall(r"\"(.*?)\"", jaString) jaString = data[i]
if len(choiceList) > 0:
# Translate choiceList = re.findall(r"\"(.*?)\"", jaString)
response = translateAI(choiceList, "This will be a dialogue option") if len(choiceList) > 0:
translatedTextList = response[0] # Translate
tokens[0] += response[1][0] response = translateAI(choiceList, "This will be a dialogue option")
tokens[1] += response[1][1] translatedTextList = response[0]
tokens[0] += response[1][0]
# Set Data tokens[1] += response[1][1]
for j in range(len(translatedTextList)):
# Convert to Wide # Set Data
if CONVERTTOWIDE: for j in range(len(translatedTextList)):
translatedText = translatedTextList[j].translate(ascii_to_wide) # Convert to Wide
if CONVERTTOWIDE:
# Set translatedText = translatedTextList[j].translate(ascii_to_wide)
data[i] = data[i].replace(choiceList[j], translatedText)
save_progress_lines(data, filename) # Set
i += 1 data[i] = data[i].replace(choiceList[j], translatedText)
save_progress_lines(data, filename)
# Nothing relevant. Skip Line. i += 1
else:
i += 1 # Nothing relevant. Skip Line.
else:
# EOF i += 1
if len(stringList) > 0:
# Set Progress # EOF
pbar.total = len(stringList) if len(stringList) > 0:
pbar.refresh() # Set Progress
pbar.total = len(stringList)
# Translate pbar.refresh()
response = translateAI(stringList, "")
tokens[0] += response[1][0] # Translate
tokens[1] += response[1][1] response = translateAI(stringList, "")
translatedList = response[0] tokens[0] += response[1][0]
tokens[1] += response[1][1]
# Set Strings translatedList = response[0]
if len(stringList) == len(translatedList):
translateOnscripter(data, pbar, filename, translatedList) # Set Strings
if len(stringList) == len(translatedList):
# Mismatch translateOnscripter(data, pbar, filename, translatedList)
else:
with LOCK: # Mismatch
if filename not in MISMATCH: else:
MISMATCH.append(filename) with LOCK:
return tokens if filename not in MISMATCH:
MISMATCH.append(filename)
return tokens
def fixText(translatedText):
# Add Break
translatedText = translatedText.replace('"', "'") def fixText(translatedText):
translatedText = f"\u3000{translatedText}\\" # Add Break
translatedText = translatedText.replace('"', "'")
# Unconvert Codes translatedText = f"\u3000{translatedText}\\"
matchList = re.findall(r"([＄].+?)[^\w]", translatedText)
if matchList: # Unconvert Codes
for match in matchList: matchList = re.findall(r"([＄].+?)[^\w]", translatedText)
translatedText = translatedText.replace(match, match.translate(wide_to_ascii)) if matchList:
for match in matchList:
# Unconvert Color Codes translatedText = translatedText.replace(match, match.translate(wide_to_ascii))
matchList = re.findall(r"([＃][\w\d]{6})", translatedText)
if matchList: # Unconvert Color Codes
for match in matchList: matchList = re.findall(r"([＃][\w\d]{6})", translatedText)
translatedText = translatedText.replace(match, match.translate(wide_to_ascii)) if matchList:
for match in matchList:
# Unconvert Variables translatedText = translatedText.replace(match, match.translate(wide_to_ascii))
matchList = re.findall(r"([％]\w.+?)[^\w＿]", translatedText)
if matchList: # Unconvert Variables
for match in matchList: matchList = re.findall(r"([％]\w.+?)[^\w＿]", translatedText)
translatedText = translatedText.replace(match, match.translate(wide_to_ascii)) if matchList:
for match in matchList:
# Unconvert Backslashes translatedText = translatedText.replace(match, match.translate(wide_to_ascii))
matchList = re.findall(r"＼", translatedText)
if matchList: # Unconvert Backslashes
for match in matchList: matchList = re.findall(r"＼", translatedText)
translatedText = translatedText.replace(match, match.translate(wide_to_ascii)) if matchList:
for match in matchList:
return translatedText translatedText = translatedText.replace(match, match.translate(wide_to_ascii))
# Save some money and enter the character before translation return translatedText
def getSpeaker(speaker):
match speaker: # Save some money and enter the character before translation
case "ファイン": def getSpeaker(speaker):
return ["Fine", [0, 0]] match speaker:
case "": case "ファイン":
return ["", [0, 0]] return ["Fine", [0, 0]]
case _: case "":
# Find Speaker return ["", [0, 0]]
for i in range(len(NAMESLIST)): case _:
if speaker == NAMESLIST[i][0]: # Find Speaker
return [NAMESLIST[i][1], [0, 0]] for i in range(len(NAMESLIST)):
if speaker == NAMESLIST[i][0]:
# Translate and Store Speaker return [NAMESLIST[i][1], [0, 0]]
response = translateAI(
f"{speaker}", # Translate and Store Speaker
"Reply with the " + LANGUAGE + " translation of the NPC name.", response = translateAI(
False, f"{speaker}",
) "Reply with the " + LANGUAGE + " translation of the NPC name.",
response[0] = response[0].title() False,
response[0] = response[0].replace("'S", "'s") )
response[0] = response[0].replace("Speaker: ", "") response[0] = response[0].title()
response[0] = response[0].replace("'S", "'s")
# Retry if name doesn't translate for some reason response[0] = response[0].replace("Speaker: ", "")
if re.search(r"([a-zA-Z？?])", response[0]) == None:
response = translateAI( # Retry if name doesn't translate for some reason
f"{speaker}", if re.search(r"([a-zA-Z？?])", response[0]) == None:
"Reply with the " + LANGUAGE + " translation of the NPC name.", response = translateAI(
False, f"{speaker}",
) "Reply with the " + LANGUAGE + " translation of the NPC name.",
response[0] = response[0].title() False,
response[0] = response[0].replace("'S", "'s") )
response[0] = response[0].title()
speakerList = [speaker, response[0]] response[0] = response[0].replace("'S", "'s")
NAMESLIST.append(speakerList)
return response speakerList = [speaker, response[0]]
return [speaker, [0, 0]] NAMESLIST.append(speakerList)
return response
return [speaker, [0, 0]]
def translateAI(text, history, history_ctx=None):
"""
Translate text using the shared translation utility. def translateAI(text, history, history_ctx=None):
This function maintains compatibility with existing code while using the new shared implementation. """
""" Translate text using the shared translation utility.
return sharedtranslateAI( This function maintains compatibility with existing code while using the new shared implementation.
text=text, """
history=history, return sharedtranslateAI(
config=TRANSLATION_CONFIG, text=text,
filename=FILENAME, history=history,
pbar=PBAR, config=TRANSLATION_CONFIG,
lock=LOCK, filename=FILENAME,
mismatchList=MISMATCH pbar=PBAR,
) lock=LOCK,
mismatchList=MISMATCH
)

1056
modules/regex.py
View file

File diff suppressed because it is too large Load diff

704
modules/renpy.py
View file

@ -1,351 +1,353 @@
# Libraries # Libraries
import json import json
import os import os
import re import re
import util.dazedwrap as dazedwrap import util.dazedwrap as dazedwrap
import threading import threading
import time import time
import traceback import traceback
import tiktoken import tiktoken
from pathlib import Path from pathlib import Path
from colorama import Fore from colorama import Fore
from dotenv import load_dotenv from dotenv import load_dotenv
from retry import retry from retry import retry
from tqdm import tqdm from tqdm import tqdm
from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost, getPricingConfig, calculateCost from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost, getPricingConfig, calculateCost
import tempfile import tempfile
# Globals # Globals
MODEL = os.getenv("model") MODEL = os.getenv("model")
TIMEOUT = int(os.getenv("timeout")) TIMEOUT = int(os.getenv("timeout"))
LANGUAGE = os.getenv("language").capitalize() LANGUAGE = os.getenv("language").capitalize()
PROMPT = Path("prompt.txt").read_text(encoding="utf-8") from util.paths import PROMPT_PATH, VOCAB_PATH
VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
LOCK = threading.Lock() PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
WIDTH = int(os.getenv("width")) VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
LISTWIDTH = int(os.getenv("listWidth")) LOCK = threading.Lock()
NOTEWIDTH = 70 WIDTH = int(os.getenv("width"))
MAXHISTORY = 10 LISTWIDTH = int(os.getenv("listWidth"))
ESTIMATE = "" NOTEWIDTH = 70
TOKENS = [0, 0] MAXHISTORY = 10
NAMESLIST = [] ESTIMATE = ""
NAMES = False # Output a list of all the character names found TOKENS = [0, 0]
BRFLAG = False # If the game uses <br> instead NAMESLIST = []
FIXTEXTWRAP = True # Overwrites textwrap NAMES = False # Output a list of all the character names found
IGNORETLTEXT = False # Ignores all translated text. BRFLAG = False # If the game uses <br> instead
MISMATCH = [] # Lists files that throw a mismatch error (Length of GPT list response is wrong) FIXTEXTWRAP = True # Overwrites textwrap
FILENAME = None IGNORETLTEXT = False # Ignores all translated text.
MISMATCH = [] # Lists files that throw a mismatch error (Length of GPT list response is wrong)
# tqdm Globals FILENAME = None
BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
POSITION = 0 # tqdm Globals
LEAVE = False BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
PBAR = None POSITION = 0
LEAVE = False
# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex PBAR = None
LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
# Get pricing configuration based on the model LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
PRICING_CONFIG = getPricingConfig(MODEL)
INPUTAPICOST = PRICING_CONFIG["inputAPICost"] # Get pricing configuration based on the model
OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"] PRICING_CONFIG = getPricingConfig(MODEL)
BATCHSIZE = PRICING_CONFIG["batchSize"] INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"] OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
BATCHSIZE = PRICING_CONFIG["batchSize"]
# Initialize Translation Config FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
TRANSLATION_CONFIG = TranslationConfig(
model=MODEL, # Initialize Translation Config
language=LANGUAGE, TRANSLATION_CONFIG = TranslationConfig(
prompt=PROMPT, model=MODEL,
vocab=VOCAB, language=LANGUAGE,
langRegex=LANGREGEX, prompt=PROMPT,
batchSize=BATCHSIZE, vocab=VOCAB,
maxHistory=MAXHISTORY, langRegex=LANGREGEX,
estimateMode=False # Will be set dynamically based on ESTIMATE batchSize=BATCHSIZE,
) maxHistory=MAXHISTORY,
LEAVE = False estimateMode=False # Will be set dynamically based on ESTIMATE
)
def handleRenpy(filename, estimate): LEAVE = False
global ESTIMATE
global FILENAME def handleRenpy(filename, estimate):
FILENAME = filename global ESTIMATE
ESTIMATE = estimate global FILENAME
FILENAME = filename
if ESTIMATE: ESTIMATE = estimate
start = time.time()
translatedData = openFiles(filename) if ESTIMATE:
start = time.time()
# Print Result translatedData = openFiles(filename)
end = time.time()
tqdm.write(getResultString(translatedData, end - start, filename)) # Print Result
with LOCK: end = time.time()
TOKENS[0] += translatedData[1][0] tqdm.write(getResultString(translatedData, end - start, filename))
TOKENS[1] += translatedData[1][1] with LOCK:
TOKENS[0] += translatedData[1][0]
# Print Total TOKENS[1] += translatedData[1][1]
totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
# Print Total
# Print any errors on maps totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
if len(MISMATCH) > 0:
return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET # Print any errors on maps
else: if len(MISMATCH) > 0:
return totalString return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
else:
else: return totalString
try:
with open("translated/" + filename, "w", encoding="utf8", errors="ignore") as outFile: else:
start = time.time() try:
translatedData = openFiles(filename) with open("translated/" + filename, "w", encoding="utf8", errors="ignore") as outFile:
start = time.time()
# Print Result translatedData = openFiles(filename)
end = time.time()
outFile.writelines(translatedData[0]) # Print Result
tqdm.write(getResultString(translatedData, end - start, filename)) end = time.time()
with LOCK: outFile.writelines(translatedData[0])
TOKENS[0] += translatedData[1][0] tqdm.write(getResultString(translatedData, end - start, filename))
TOKENS[1] += translatedData[1][1] with LOCK:
except Exception: TOKENS[0] += translatedData[1][0]
traceback.print_exc() TOKENS[1] += translatedData[1][1]
return "Fail" except Exception:
traceback.print_exc()
return getResultString(["", TOKENS, None], end - start, "TOTAL") return "Fail"
return getResultString(["", TOKENS, None], end - start, "TOTAL")
def getResultString(translatedData, translationTime, filename):
# File Print String
cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL) def getResultString(translatedData, translationTime, filename):
totalTokenstring = ( # File Print String
Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]" cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
"[Output: " totalTokenstring = (
+ str(translatedData[1][1]) Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
+ "]" "[Cost: ${:,.4f}".format(cost) "[Output: "
+ "]" + str(translatedData[1][1])
) + "]" "[Cost: ${:,.4f}".format(cost)
timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]" + "]"
)
if translatedData[2] == None: timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
# Success
return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET if translatedData[2] == None:
# Success
else: return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
# Fail
try: else:
raise translatedData[2] # Fail
except Exception as e: try:
traceback.print_exc() raise translatedData[2]
errorString = str(e) + Fore.RED except Exception as e:
return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET traceback.print_exc()
errorString = str(e) + Fore.RED
return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
def openFiles(filename):
with open("files/" + filename, "r", encoding="utf8") as readFile:
translatedData = parseRenpy(readFile, filename) def openFiles(filename):
with open("files/" + filename, "r", encoding="utf8") as readFile:
# Delete lines marked for deletion translatedData = parseRenpy(readFile, filename)
finalData = []
for line in translatedData[0]: # Delete lines marked for deletion
if line != "\\d\n": finalData = []
finalData.append(line) for line in translatedData[0]:
translatedData[0] = finalData if line != "\\d\n":
finalData.append(line)
return translatedData translatedData[0] = finalData
return translatedData
def parseRenpy(readFile, filename):
global PBAR
totalTokens = [0, 0] def parseRenpy(readFile, filename):
global PBAR
# Read File into data totalTokens = [0, 0]
data = readFile.readlines()
# Read File into data
# Create Progress Bar data = readFile.readlines()
with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
pbar.desc = filename # Create Progress Bar
PBAR = pbar with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
pbar.desc = filename
try: PBAR = pbar
result = translateRenpy(data, filename, [])
totalTokens[0] += result[0] try:
totalTokens[1] += result[1] result = translateRenpy(data, filename, [])
except Exception as e: totalTokens[0] += result[0]
traceback.print_exc() totalTokens[1] += result[1]
return [data, totalTokens, e] except Exception as e:
return [data, totalTokens, None] traceback.print_exc()
return [data, totalTokens, e]
return [data, totalTokens, None]
def save_progress_lines(lines, filename, encoding="utf-8"):
"""Atomically save current line-based translation progress."""
try: def save_progress_lines(lines, filename, encoding="utf-8"):
if ESTIMATE: """Atomically save current line-based translation progress."""
return try:
os.makedirs("translated", exist_ok=True) if ESTIMATE:
tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated") return
try: os.makedirs("translated", exist_ok=True)
with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file: tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
tmp_file.writelines(lines) try:
os.replace(tmp_path, os.path.join("translated", filename)) with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file:
finally: tmp_file.writelines(lines)
if os.path.exists(tmp_path): os.replace(tmp_path, os.path.join("translated", filename))
try: finally:
os.remove(tmp_path) if os.path.exists(tmp_path):
except OSError: try:
pass os.remove(tmp_path)
except Exception: except OSError:
traceback.print_exc() pass
except Exception:
traceback.print_exc()
def translateRenpy(data, filename, translatedList):
stringList = []
currentGroup = [] def translateRenpy(data, filename, translatedList):
tokens = [0, 0] stringList = []
speaker = "" currentGroup = []
voice = False tokens = [0, 0]
global LOCK, ESTIMATE, FILENAME, PBAR speaker = ""
i = 0 voice = False
global LOCK, ESTIMATE, FILENAME, PBAR
while i < len(data): i = 0
voice = False
speaker = "" while i < len(data):
lineRegexNoSpeaker = r'^\s\s\s\s"(.*)"' voice = False
lineRegexSpeaker = r'^\s\s\s\s(.+?)\s"(.*)"' speaker = ""
lineRegexNoSpeaker = r'^\s\s\s\s"(.*)"'
# Grab Line lineRegexSpeaker = r'^\s\s\s\s(.+?)\s"(.*)"'
match = re.search(lineRegexSpeaker, data[i])
if match: # Grab Line
response = getSpeaker(match.group(1)) match = re.search(lineRegexSpeaker, data[i])
jaString = match.group(2) if match:
speaker = response[0] response = getSpeaker(match.group(1))
tokens[0] += response[1][0] jaString = match.group(2)
tokens[1] += response[1][1] speaker = response[0]
else: tokens[0] += response[1][0]
match = re.search(lineRegexNoSpeaker, data[i]) tokens[1] += response[1][1]
if match: else:
jaString = match.group(1) match = re.search(lineRegexNoSpeaker, data[i])
if match:
# Valid Line jaString = match.group(1)
if match and "voice" not in data[i]:
originalString = jaString # Valid Line
# Pass 1 if match and "voice" not in data[i]:
if translatedList == []: originalString = jaString
# Remove any textwrap # Pass 1
jaString = jaString.replace("\\n", " ") if translatedList == []:
# Remove any textwrap
# Add String jaString = jaString.replace("\\n", " ")
if speaker:
stringList.append(f"[{speaker}]: {jaString.strip()}") # Add String
else: if speaker:
stringList.append(jaString.strip()) stringList.append(f"[{speaker}]: {jaString.strip()}")
else:
# Pass 2 stringList.append(jaString.strip())
else:
# Get Text # Pass 2
if translatedList: else:
# Grab and Pop # Get Text
translatedText = translatedList[0] if translatedList:
translatedList.pop(0) # Grab and Pop
translatedText = translatedList[0]
# Set to None if empty list translatedList.pop(0)
if len(translatedList) <= 0:
translatedList = None # Set to None if empty list
if len(translatedList) <= 0:
# Remove speaker translatedList = None
if speaker != "":
matchSpeakerList = re.findall(r"^\[?(.+?)\]?\s?[|:]\s?", translatedText) # Remove speaker
translatedText = re.sub(r"^\[?(.+?)\]?\s?[|:]\s?", "", translatedText) if speaker != "":
matchSpeakerList = re.findall(r"^\[?(.+?)\]?\s?[|:]\s?", translatedText)
# Escape Quotes translatedText = re.sub(r"^\[?(.+?)\]?\s?[|:]\s?", "", translatedText)
translatedText = re.sub(r'[\\]*(")', '\\"', translatedText)
translatedText = re.sub(r"[\\]*(')", "\\'", translatedText) # Escape Quotes
translatedText = re.sub(r'[\\]*(")', '\\"', translatedText)
# Textwrap translatedText = re.sub(r"[\\]*(')", "\\'", translatedText)
translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
translatedText = translatedText.replace("\n", "\\n") # Textwrap
translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
# Set Data translatedText = translatedText.replace("\n", "\\n")
data[i] = data[i].replace(originalString, translatedText)
save_progress_lines(data, filename) # Set Data
i += 1 data[i] = data[i].replace(originalString, translatedText)
else: save_progress_lines(data, filename)
i += 1 i += 1
else:
# EOF i += 1
if len(stringList) > 0:
# Set Progress # EOF
PBAR.total = len(stringList) if len(stringList) > 0:
PBAR.refresh() # Set Progress
PBAR.total = len(stringList)
# Translate PBAR.refresh()
response = translateAI(stringList, "Reply with the English TL of the NPC Name")
tokens[0] += response[1][0] # Translate
tokens[1] += response[1][1] response = translateAI(stringList, "Reply with the English TL of the NPC Name")
translatedList = response[0] tokens[0] += response[1][0]
tokens[1] += response[1][1]
# Set Strings translatedList = response[0]
if len(stringList) == len(translatedList):
translateRenpy(data, filename, translatedList) # Set Strings
if len(stringList) == len(translatedList):
# Mismatch translateRenpy(data, filename, translatedList)
else:
with LOCK: # Mismatch
if FILENAME not in MISMATCH: else:
MISMATCH.append(FILENAME) with LOCK:
return tokens if FILENAME not in MISMATCH:
MISMATCH.append(FILENAME)
# Save some money and enter the character before translation return tokens
def getSpeaker(speaker):
match speaker: # Save some money and enter the character before translation
case "ファイン": def getSpeaker(speaker):
return ["Fine", [0, 0]] match speaker:
case "": case "ファイン":
return ["", [0, 0]] return ["Fine", [0, 0]]
case _: case "":
# Find Speaker return ["", [0, 0]]
for i in range(len(NAMESLIST)): case _:
if speaker == NAMESLIST[i][0]: # Find Speaker
return [NAMESLIST[i][1], [0, 0]] for i in range(len(NAMESLIST)):
if speaker == NAMESLIST[i][0]:
# Translate and Store Speaker return [NAMESLIST[i][1], [0, 0]]
response = translateAI(
f"{speaker}", # Translate and Store Speaker
"Reply with the " + LANGUAGE + " translation of the NPC name.", response = translateAI(
False, f"{speaker}",
) "Reply with the " + LANGUAGE + " translation of the NPC name.",
response[0] = response[0].title() False,
response[0] = response[0].replace("'S", "'s") )
response[0] = response[0].replace("Speaker: ", "") response[0] = response[0].title()
response[0] = response[0].replace("'S", "'s")
# Retry if name doesn't translate for some reason response[0] = response[0].replace("Speaker: ", "")
if re.search(r"([a-zA-Z？?])", response[0]) == None:
response = translateAI( # Retry if name doesn't translate for some reason
f"{speaker}", if re.search(r"([a-zA-Z？?])", response[0]) == None:
"Reply with the " + LANGUAGE + " translation of the NPC name.", response = translateAI(
False, f"{speaker}",
) "Reply with the " + LANGUAGE + " translation of the NPC name.",
response[0] = response[0].title() False,
response[0] = response[0].replace("'S", "'s") )
response[0] = response[0].title()
speakerList = [speaker, response[0]] response[0] = response[0].replace("'S", "'s")
NAMESLIST.append(speakerList)
return response speakerList = [speaker, response[0]]
return [speaker, [0, 0]] NAMESLIST.append(speakerList)
return response
def translateAI(text, history, history_ctx=None): return [speaker, [0, 0]]
"""
Legacy wrapper function for the new shared translation utility. def translateAI(text, history, history_ctx=None):
This maintains compatibility with existing code while using the new shared implementation. """
""" Legacy wrapper function for the new shared translation utility.
global PBAR, MISMATCH, FILENAME This maintains compatibility with existing code while using the new shared implementation.
"""
# Update config estimate mode based on global ESTIMATE global PBAR, MISMATCH, FILENAME
TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
# Update config estimate mode based on global ESTIMATE
# Call the new shared translation function TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
return sharedtranslateAI(
text=text, # Call the new shared translation function
history=history, return sharedtranslateAI(
config=TRANSLATION_CONFIG, text=text,
filename=FILENAME, history=history,
pbar=PBAR, config=TRANSLATION_CONFIG,
lock=LOCK, filename=FILENAME,
mismatchList=MISMATCH pbar=PBAR,
) lock=LOCK,
mismatchList=MISMATCH
)

10
modules/rpgmakermvmz.py
View file

@ -19,8 +19,10 @@ from util.translation import TranslationConfig, translateAI as sharedtranslateAI
MODEL = os.getenv("model") MODEL = os.getenv("model")
TIMEOUT = int(os.getenv("timeout")) TIMEOUT = int(os.getenv("timeout"))
LANGUAGE = os.getenv("language").capitalize() LANGUAGE = os.getenv("language").capitalize()
PROMPT = Path("prompt.txt").read_text(encoding="utf-8") from util.paths import PROMPT_PATH, VOCAB_PATH
VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
LOCK = threading.Lock() LOCK = threading.Lock()
THREAD_CTX = threading.local() THREAD_CTX = threading.local()
WIDTH = int(os.getenv("width")) WIDTH = int(os.getenv("width"))
@ -803,7 +805,7 @@ def update_vocab_section(category: str, pairs: list[tuple[str, str]]):
The existing section is replaced entirely; other sections are preserved. The existing section is replaced entirely; other sections are preserved.
""" """
try: try:
vocab_path = Path("vocab.txt") vocab_path = VOCAB_PATH
# Helper: normalized comparison to detect no-op translations # Helper: normalized comparison to detect no-op translations
def _norm(s: str) -> str: def _norm(s: str) -> str:
@ -5071,7 +5073,7 @@ def finalizeSpeakerParse():
_speakerCache[orig] = norm _speakerCache[orig] = norm
NAMESLIST.append([orig, norm]) NAMESLIST.append([orig, norm])
vocab_path = Path("vocab.txt") vocab_path = VOCAB_PATH
if not vocab_path.exists(): if not vocab_path.exists():
return return
content = vocab_path.read_text(encoding="utf-8") content = vocab_path.read_text(encoding="utf-8")

2272
modules/rpgmakerplugin.py
View file

File diff suppressed because it is too large Load diff

4524
modules/srpg.py
View file

File diff suppressed because it is too large Load diff

726
modules/text.py
View file

@ -1,363 +1,365 @@
# Libraries # Libraries
import json import json
import os import os
import re import re
import util.dazedwrap as dazedwrap import util.dazedwrap as dazedwrap
import threading import threading
import time import time
import traceback import traceback
import tiktoken import tiktoken
from pathlib import Path from pathlib import Path
from colorama import Fore from colorama import Fore
from dotenv import load_dotenv from dotenv import load_dotenv
from retry import retry from retry import retry
from tqdm import tqdm from tqdm import tqdm
from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost, getPricingConfig, calculateCost from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost, getPricingConfig, calculateCost
import tempfile import tempfile
# Globals # Globals
MODEL = os.getenv("model") MODEL = os.getenv("model")
TIMEOUT = int(os.getenv("timeout")) TIMEOUT = int(os.getenv("timeout"))
LANGUAGE = os.getenv("language").capitalize() LANGUAGE = os.getenv("language").capitalize()
PROMPT = Path("prompt.txt").read_text(encoding="utf-8") from util.paths import PROMPT_PATH, VOCAB_PATH
VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
LOCK = threading.Lock() PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
WIDTH = int(os.getenv("width")) VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
LISTWIDTH = int(os.getenv("listWidth")) LOCK = threading.Lock()
NOTEWIDTH = 70 WIDTH = int(os.getenv("width"))
MAXHISTORY = 10 LISTWIDTH = int(os.getenv("listWidth"))
ESTIMATE = "" NOTEWIDTH = 70
TOKENS = [0, 0] MAXHISTORY = 10
NAMESLIST = [] ESTIMATE = ""
NAMES = False # Output a list of all the character names found TOKENS = [0, 0]
BRFLAG = False # If the game uses <br> instead NAMESLIST = []
FIXTEXTWRAP = True # Overwrites textwrap NAMES = False # Output a list of all the character names found
IGNORETLTEXT = False # Ignores all translated text. BRFLAG = False # If the game uses <br> instead
MISMATCH = [] # Lists files that throw a mismatch error (Length of GPT list response is wrong) FIXTEXTWRAP = True # Overwrites textwrap
FILENAME = None IGNORETLTEXT = False # Ignores all translated text.
MISMATCH = [] # Lists files that throw a mismatch error (Length of GPT list response is wrong)
# tqdm Globals FILENAME = None
BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
POSITION = 0 # tqdm Globals
LEAVE = False BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
PBAR = None POSITION = 0
LEAVE = False
# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex PBAR = None
LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
# Get pricing configuration based on the model LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
PRICING_CONFIG = getPricingConfig(MODEL)
INPUTAPICOST = PRICING_CONFIG["inputAPICost"] # Get pricing configuration based on the model
OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"] PRICING_CONFIG = getPricingConfig(MODEL)
BATCHSIZE = PRICING_CONFIG["batchSize"] INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"] OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
BATCHSIZE = PRICING_CONFIG["batchSize"]
# Initialize Translation Config FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
TRANSLATION_CONFIG = TranslationConfig(
model=MODEL, # Initialize Translation Config
language=LANGUAGE, TRANSLATION_CONFIG = TranslationConfig(
prompt=PROMPT, model=MODEL,
vocab=VOCAB, language=LANGUAGE,
langRegex=LANGREGEX, prompt=PROMPT,
batchSize=BATCHSIZE, vocab=VOCAB,
maxHistory=MAXHISTORY, langRegex=LANGREGEX,
estimateMode=False # Will be set dynamically based on ESTIMATE batchSize=BATCHSIZE,
) maxHistory=MAXHISTORY,
LEAVE = False estimateMode=False # Will be set dynamically based on ESTIMATE
)
def handleText(filename, estimate): LEAVE = False
global ESTIMATE, TOKENS, FILENAME
ESTIMATE = estimate def handleText(filename, estimate):
FILENAME = filename global ESTIMATE, TOKENS, FILENAME
ESTIMATE = estimate
# Translate FILENAME = filename
start = time.time()
translatedData = openFiles(filename) # Translate
start = time.time()
# Translate translatedData = openFiles(filename)
if not estimate:
try: # Translate
with open("translated/" + filename, "w", encoding="utf-8", newline="\n") as outFile: if not estimate:
outFile.writelines(translatedData[0]) try:
except Exception: with open("translated/" + filename, "w", encoding="utf-8", newline="\n") as outFile:
traceback.print_exc() outFile.writelines(translatedData[0])
return "Fail" except Exception:
traceback.print_exc()
# Print File return "Fail"
end = time.time()
tqdm.write(getResultString(translatedData, end - start, filename)) # Print File
with LOCK: end = time.time()
TOKENS[0] += translatedData[1][0] tqdm.write(getResultString(translatedData, end - start, filename))
TOKENS[1] += translatedData[1][1] with LOCK:
TOKENS[0] += translatedData[1][0]
# Print Total TOKENS[1] += translatedData[1][1]
totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
# Print Total
# Print any errors on maps totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
if len(MISMATCH) > 0:
return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET # Print any errors on maps
else: if len(MISMATCH) > 0:
return totalString return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
else:
return totalString
def getResultString(translatedData, translationTime, filename):
# File Print String
cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL) def getResultString(translatedData, translationTime, filename):
totalTokenstring = ( # File Print String
Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]" cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
"[Output: " totalTokenstring = (
+ str(translatedData[1][1]) Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
+ "]" "[Cost: ${:,.4f}".format(cost) "[Output: "
+ "]" + str(translatedData[1][1])
) + "]" "[Cost: ${:,.4f}".format(cost)
timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]" + "]"
)
if translatedData[2] == None: timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
# Success
return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET if translatedData[2] == None:
# Success
else: return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
# Fail
try: else:
raise translatedData[2] # Fail
except Exception as e: try:
traceback.print_exc() raise translatedData[2]
errorString = str(e) + Fore.RED except Exception as e:
return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET traceback.print_exc()
errorString = str(e) + Fore.RED
return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
def openFiles(filename):
with open("files/" + filename, "r", encoding="utf8") as readFile:
translatedData = parseText(readFile, filename) def openFiles(filename):
with open("files/" + filename, "r", encoding="utf8") as readFile:
# Delete lines marked for deletion translatedData = parseText(readFile, filename)
finalData = []
for line in translatedData[0]: # Delete lines marked for deletion
if line != "\\d\n": finalData = []
finalData.append(line) for line in translatedData[0]:
translatedData[0] = finalData if line != "\\d\n":
finalData.append(line)
return translatedData translatedData[0] = finalData
return translatedData
def parseText(readFile, filename):
global PBAR
totalTokens = [0, 0] def parseText(readFile, filename):
global PBAR
# Read File into data totalTokens = [0, 0]
data = readFile.readlines()
# Read File into data
# Create Progress Bar data = readFile.readlines()
with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
pbar.desc = filename # Create Progress Bar
PBAR = pbar with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
pbar.desc = filename
try: PBAR = pbar
result = translateTxt(data, filename, [])
totalTokens[0] += result[0] try:
totalTokens[1] += result[1] result = translateTxt(data, filename, [])
except Exception as e: totalTokens[0] += result[0]
traceback.print_exc() totalTokens[1] += result[1]
return [data, totalTokens, e] except Exception as e:
return [data, totalTokens, None] traceback.print_exc()
return [data, totalTokens, e]
return [data, totalTokens, None]
def save_progress_lines(lines, filename, encoding="utf-8"):
"""Atomically save current line-based translation progress."""
try: def save_progress_lines(lines, filename, encoding="utf-8"):
if ESTIMATE: """Atomically save current line-based translation progress."""
return try:
os.makedirs("translated", exist_ok=True) if ESTIMATE:
tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated") return
try: os.makedirs("translated", exist_ok=True)
with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file: tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
tmp_file.writelines(lines) try:
os.replace(tmp_path, os.path.join("translated", filename)) with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file:
finally: tmp_file.writelines(lines)
if os.path.exists(tmp_path): os.replace(tmp_path, os.path.join("translated", filename))
try: finally:
os.remove(tmp_path) if os.path.exists(tmp_path):
except OSError: try:
pass os.remove(tmp_path)
except Exception: except OSError:
traceback.print_exc() pass
except Exception:
traceback.print_exc()
def saveCheckLines(lines, filename, tokens=None, encoding="utf-8"):
"""Save progress only when tokens indicate work or when explicitly called after a mutation."""
try: def saveCheckLines(lines, filename, tokens=None, encoding="utf-8"):
if tokens is not None: """Save progress only when tokens indicate work or when explicitly called after a mutation."""
if not (isinstance(tokens, (list, tuple)) and len(tokens) >= 2 and (tokens[0] or tokens[1])): try:
return if tokens is not None:
save_progress_lines(lines, filename, encoding=encoding) if not (isinstance(tokens, (list, tuple)) and len(tokens) >= 2 and (tokens[0] or tokens[1])):
except Exception: return
traceback.print_exc() save_progress_lines(lines, filename, encoding=encoding)
except Exception:
traceback.print_exc()
def translateTxt(data, filename, translatedList):
if translatedList:
stringList = translatedList[0] def translateTxt(data, filename, translatedList):
choiceList = translatedList[1] if translatedList:
else: stringList = translatedList[0]
stringList = [] choiceList = translatedList[1]
choiceList = [] else:
tokens = [0, 0] stringList = []
speaker = "" choiceList = []
global LOCK, ESTIMATE, FILENAME, PBAR, MISMATCH tokens = [0, 0]
i = 0 speaker = ""
global LOCK, ESTIMATE, FILENAME, PBAR, MISMATCH
# Regex i = 0
lineTextRegex = r"(?:^\[.+?\]:)?(.+)"
speakerTextRegex = r"^\[(.+?)\]" # Regex
lineTextRegex = r"(?:^\[.+?\]:)?(.+)"
while i < len(data): speakerTextRegex = r"^\[(.+?)\]"
# Speaker
match = re.search(speakerTextRegex, data[i]) while i < len(data):
if match: # Speaker
# Get Speaker match = re.search(speakerTextRegex, data[i])
speakerData = getSpeaker(match.group(1)) if match:
speaker = speakerData[0] # Get Speaker
tokens[0] += speakerData[1][0] speakerData = getSpeaker(match.group(1))
tokens[1] += speakerData[1][1] speaker = speakerData[0]
data[i] = data[i].replace(match.group(1), speaker) tokens[0] += speakerData[1][0]
tokens[1] += speakerData[1][1]
# Dialogue data[i] = data[i].replace(match.group(1), speaker)
match = re.search(lineTextRegex, data[i])
jaString = None # Dialogue
if match: match = re.search(lineTextRegex, data[i])
# Set String jaString = None
jaString = match.group(1) if match:
# Set String
# Pass 1 jaString = match.group(1)
if not translatedList:
# Strip Spaces # Pass 1
jaString = jaString.strip() if not translatedList:
# Strip Spaces
if jaString: jaString = jaString.strip()
if speaker:
stringList.append(f"[{speaker}]: {jaString}") if jaString:
else: if speaker:
stringList.append(jaString) stringList.append(f"[{speaker}]: {jaString}")
else:
# Pass 2 stringList.append(jaString)
else:
# Get Text # Pass 2
if stringList: else:
# Grab and Pop # Get Text
translatedText = stringList[0] if stringList:
stringList.pop(0) # Grab and Pop
translatedText = stringList[0]
# Set to None if empty list stringList.pop(0)
if len(stringList) <= 0:
stringList = None # Set to None if empty list
if len(stringList) <= 0:
# Remove speaker stringList = None
translatedText = re.sub(r"(^\[.+?\]\s?[|:]\s?)", "", translatedText)
# Remove speaker
# # Textwrap translatedText = re.sub(r"(^\[.+?\]\s?[|:]\s?)", "", translatedText)
# translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
# translatedText = translatedText.replace("\n", "\\n") # # Textwrap
# translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
# Set Data # translatedText = translatedText.replace("\n", "\\n")
data[i] = f"{translatedText}\n"
saveCheckLines(data, filename) # Set Data
data[i] = f"{translatedText}\n"
i += 1 saveCheckLines(data, filename)
else:
i += 1 i += 1
else:
# EOF i += 1
if not translatedList:
stringListTL = [] # EOF
choiceListTL = [] if not translatedList:
stringListTL = []
# String List choiceListTL = []
if stringList:
PBAR.total = len(stringList) # String List
PBAR.refresh() if stringList:
response = translateAI(stringList, "Reply with the English Translation") PBAR.total = len(stringList)
tokens[0] += response[1][0] PBAR.refresh()
tokens[1] += response[1][1] response = translateAI(stringList, "Reply with the English Translation")
stringListTL = response[0] tokens[0] += response[1][0]
tokens[1] += response[1][1]
if len(stringList) != len(stringListTL): stringListTL = response[0]
# Mismatch
with LOCK: if len(stringList) != len(stringListTL):
if FILENAME not in MISMATCH: # Mismatch
MISMATCH.append(FILENAME) with LOCK:
if FILENAME not in MISMATCH:
# Choice List MISMATCH.append(FILENAME)
if choiceList:
response = translateAI(choiceList, "Reply with the English TL of the Dialogue Choice") # Choice List
tokens[0] += response[1][0] if choiceList:
tokens[1] += response[1][1] response = translateAI(choiceList, "Reply with the English TL of the Dialogue Choice")
choiceListTL = response[0] tokens[0] += response[1][0]
tokens[1] += response[1][1]
if len(choiceList) != len(choiceListTL): choiceListTL = response[0]
# Mismatch
with LOCK: if len(choiceList) != len(choiceListTL):
if FILENAME not in MISMATCH: # Mismatch
MISMATCH.append(FILENAME) with LOCK:
if FILENAME not in MISMATCH:
# Set Strings MISMATCH.append(FILENAME)
translateTxt(data, filename, [stringListTL, choiceListTL])
return tokens # Set Strings
translateTxt(data, filename, [stringListTL, choiceListTL])
# Save some money and enter the character before translation return tokens
def getSpeaker(speaker):
match speaker: # Save some money and enter the character before translation
case "ファイン": def getSpeaker(speaker):
return ["Fine", [0, 0]] match speaker:
case "": case "ファイン":
return ["", [0, 0]] return ["Fine", [0, 0]]
case _: case "":
# Find Speaker return ["", [0, 0]]
for i in range(len(NAMESLIST)): case _:
if speaker == NAMESLIST[i][0]: # Find Speaker
return [NAMESLIST[i][1], [0, 0]] for i in range(len(NAMESLIST)):
if speaker == NAMESLIST[i][0]:
# Translate and Store Speaker return [NAMESLIST[i][1], [0, 0]]
response = translateAI(
f"{speaker}", # Translate and Store Speaker
"Reply with the " + LANGUAGE + " translation of the NPC name.", response = translateAI(
False, f"{speaker}",
) "Reply with the " + LANGUAGE + " translation of the NPC name.",
response[0] = response[0].title() False,
response[0] = response[0].replace("'S", "'s") )
response[0] = response[0].replace("Speaker: ", "") response[0] = response[0].title()
response[0] = response[0].replace("'S", "'s")
# Retry if name doesn't translate for some reason response[0] = response[0].replace("Speaker: ", "")
if re.search(r"([a-zA-Z？?])", response[0]) == None:
response = translateAI( # Retry if name doesn't translate for some reason
f"{speaker}", if re.search(r"([a-zA-Z？?])", response[0]) == None:
"Reply with the " + LANGUAGE + " translation of the NPC name.", response = translateAI(
False, f"{speaker}",
) "Reply with the " + LANGUAGE + " translation of the NPC name.",
response[0] = response[0].title() False,
response[0] = response[0].replace("'S", "'s") )
response[0] = response[0].title()
speakerList = [speaker, response[0]] response[0] = response[0].replace("'S", "'s")
NAMESLIST.append(speakerList)
return response speakerList = [speaker, response[0]]
return [speaker, [0, 0]] NAMESLIST.append(speakerList)
return response
def translateAI(text, history, history_ctx=None): return [speaker, [0, 0]]
"""
Legacy wrapper function for the new shared translation utility. def translateAI(text, history, history_ctx=None):
This maintains compatibility with existing code while using the new shared implementation. """
""" Legacy wrapper function for the new shared translation utility.
global PBAR, MISMATCH, FILENAME This maintains compatibility with existing code while using the new shared implementation.
"""
# Update config estimate mode based on global ESTIMATE global PBAR, MISMATCH, FILENAME
TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
# Update config estimate mode based on global ESTIMATE
# Call the new shared translation function TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
return sharedtranslateAI(
text=text, # Call the new shared translation function
history=history, return sharedtranslateAI(
config=TRANSLATION_CONFIG, text=text,
filename=FILENAME, history=history,
pbar=PBAR, config=TRANSLATION_CONFIG,
lock=LOCK, filename=FILENAME,
mismatchList=MISMATCH pbar=PBAR,
lock=LOCK,
mismatchList=MISMATCH
) )

916
modules/tyrano.py
View file

@ -1,457 +1,459 @@
# Libraries # Libraries
import json import json
import os import os
import re import re
import util.dazedwrap as dazedwrap import util.dazedwrap as dazedwrap
import threading import threading
import time import time
import traceback import traceback
import tiktoken import tiktoken
from pathlib import Path from pathlib import Path
from colorama import Fore from colorama import Fore
from dotenv import load_dotenv from dotenv import load_dotenv
from retry import retry from retry import retry
from tqdm import tqdm from tqdm import tqdm
from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
import tempfile import tempfile
# OpenAI initialization centralized in util/translation.py # OpenAI initialization centralized in util/translation.py
# Globals # Globals
MODEL = os.getenv("model") MODEL = os.getenv("model")
TIMEOUT = int(os.getenv("timeout")) TIMEOUT = int(os.getenv("timeout"))
LANGUAGE = os.getenv("language").capitalize() LANGUAGE = os.getenv("language").capitalize()
PROMPT = Path("prompt.txt").read_text(encoding="utf-8") from util.paths import PROMPT_PATH, VOCAB_PATH
VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
LOCK = threading.Lock() PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
WIDTH = int(os.getenv("width")) VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
LISTWIDTH = int(os.getenv("listWidth")) LOCK = threading.Lock()
NOTEWIDTH = 70 WIDTH = int(os.getenv("width"))
MAXHISTORY = 10 LISTWIDTH = int(os.getenv("listWidth"))
ESTIMATE = "" NOTEWIDTH = 70
TOKENS = [0, 0] MAXHISTORY = 10
NAMESLIST = [] ESTIMATE = ""
NAMES = False # Output a list of all the character names found TOKENS = [0, 0]
BRFLAG = False # If the game uses <br> instead NAMESLIST = []
FIXTEXTWRAP = True # Overwrites textwrap NAMES = False # Output a list of all the character names found
IGNORETLTEXT = False # Ignores all translated text. BRFLAG = False # If the game uses <br> instead
MISMATCH = [] # Lists files that throw a mismatch error (Length of GPT list response is wrong) FIXTEXTWRAP = True # Overwrites textwrap
FILENAME = None IGNORETLTEXT = False # Ignores all translated text.
MISMATCH = [] # Lists files that throw a mismatch error (Length of GPT list response is wrong)
# tqdm Globals FILENAME = None
BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
POSITION = 0 # tqdm Globals
LEAVE = False BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
PBAR = None POSITION = 0
LEAVE = False
# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex PBAR = None
LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
# Get pricing configuration based on the model LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
PRICING_CONFIG = getPricingConfig(MODEL)
INPUTAPICOST = PRICING_CONFIG["inputAPICost"] # Get pricing configuration based on the model
OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"] PRICING_CONFIG = getPricingConfig(MODEL)
BATCHSIZE = PRICING_CONFIG["batchSize"] INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"] OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
BATCHSIZE = PRICING_CONFIG["batchSize"]
# Initialize Translation Config FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
TRANSLATION_CONFIG = TranslationConfig(
model=MODEL, # Initialize Translation Config
language=LANGUAGE, TRANSLATION_CONFIG = TranslationConfig(
prompt=PROMPT, model=MODEL,
vocab=VOCAB, language=LANGUAGE,
langRegex=LANGREGEX, prompt=PROMPT,
batchSize=BATCHSIZE, vocab=VOCAB,
maxHistory=MAXHISTORY, langRegex=LANGREGEX,
estimateMode=False # Will be set dynamically based on ESTIMATE batchSize=BATCHSIZE,
) maxHistory=MAXHISTORY,
LEAVE = False estimateMode=False # Will be set dynamically based on ESTIMATE
)
def handleTyrano(filename, estimate): LEAVE = False
global ESTIMATE, TOKENS, FILENAME
ESTIMATE = estimate def handleTyrano(filename, estimate):
FILENAME = filename global ESTIMATE, TOKENS, FILENAME
ESTIMATE = estimate
# Translate FILENAME = filename
start = time.time()
translatedData = openFiles(filename) # Translate
start = time.time()
# Translate translatedData = openFiles(filename)
if not estimate:
try: # Translate
with open("translated/" + filename, "w", encoding="utf-8", newline="\n") as outFile: if not estimate:
outFile.writelines(translatedData[0]) try:
except Exception: with open("translated/" + filename, "w", encoding="utf-8", newline="\n") as outFile:
traceback.print_exc() outFile.writelines(translatedData[0])
return "Fail" except Exception:
traceback.print_exc()
# Print File return "Fail"
end = time.time()
tqdm.write(getResultString(translatedData, end - start, filename)) # Print File
with LOCK: end = time.time()
TOKENS[0] += translatedData[1][0] tqdm.write(getResultString(translatedData, end - start, filename))
TOKENS[1] += translatedData[1][1] with LOCK:
TOKENS[0] += translatedData[1][0]
# Print Total TOKENS[1] += translatedData[1][1]
totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
# Print Total
# Print any errors on maps totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
if len(MISMATCH) > 0:
return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET # Print any errors on maps
else: if len(MISMATCH) > 0:
return totalString return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
else:
return totalString
def getResultString(translatedData, translationTime, filename):
# File Print String
cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL) def getResultString(translatedData, translationTime, filename):
totalTokenstring = ( # File Print String
Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]" cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
"[Output: " totalTokenstring = (
+ str(translatedData[1][1]) Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
+ "]" "[Cost: ${:,.4f}".format(cost) "[Output: "
+ "]" + str(translatedData[1][1])
) + "]" "[Cost: ${:,.4f}".format(cost)
timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]" + "]"
)
if translatedData[2] == None: timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
# Success
return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET if translatedData[2] == None:
# Success
else: return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
# Fail
try: else:
raise translatedData[2] # Fail
except Exception as e: try:
traceback.print_exc() raise translatedData[2]
errorString = str(e) + Fore.RED except Exception as e:
return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET traceback.print_exc()
errorString = str(e) + Fore.RED
return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
def openFiles(filename):
with open("files/" + filename, "r", encoding="utf8") as readFile:
translatedData = parseTyrano(readFile, filename) def openFiles(filename):
with open("files/" + filename, "r", encoding="utf8") as readFile:
# Delete lines marked for deletion translatedData = parseTyrano(readFile, filename)
finalData = []
for line in translatedData[0]: # Delete lines marked for deletion
if line != "\\d\n": finalData = []
finalData.append(line) for line in translatedData[0]:
translatedData[0] = finalData if line != "\\d\n":
finalData.append(line)
return translatedData translatedData[0] = finalData
return translatedData
def parseTyrano(readFile, filename):
global PBAR
totalTokens = [0, 0] def parseTyrano(readFile, filename):
global PBAR
# Read File into data totalTokens = [0, 0]
data = readFile.readlines()
# Read File into data
# Create Progress Bar data = readFile.readlines()
with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
pbar.desc = filename # Create Progress Bar
PBAR = pbar with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
pbar.desc = filename
try: PBAR = pbar
result = translateTyrano(data, filename, [])
totalTokens[0] += result[0] try:
totalTokens[1] += result[1] result = translateTyrano(data, filename, [])
except Exception as e: totalTokens[0] += result[0]
traceback.print_exc() totalTokens[1] += result[1]
return [data, totalTokens, e] except Exception as e:
return [data, totalTokens, None] traceback.print_exc()
return [data, totalTokens, e]
return [data, totalTokens, None]
def save_progress_lines(lines, filename, encoding="utf-8"):
"""Atomically save current line-based translation progress."""
try: def save_progress_lines(lines, filename, encoding="utf-8"):
if ESTIMATE: """Atomically save current line-based translation progress."""
return try:
os.makedirs("translated", exist_ok=True) if ESTIMATE:
tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated") return
try: os.makedirs("translated", exist_ok=True)
with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file: tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
tmp_file.writelines(lines) try:
os.replace(tmp_path, os.path.join("translated", filename)) with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file:
finally: tmp_file.writelines(lines)
if os.path.exists(tmp_path): os.replace(tmp_path, os.path.join("translated", filename))
try: finally:
os.remove(tmp_path) if os.path.exists(tmp_path):
except OSError: try:
pass os.remove(tmp_path)
except Exception: except OSError:
traceback.print_exc() pass
except Exception:
traceback.print_exc()
def saveCheckLines(lines, filename, tokens=None, encoding="utf-8"):
"""Save progress only when tokens indicate work or when tokens is None but we know a mutation occurred."""
try: def saveCheckLines(lines, filename, tokens=None, encoding="utf-8"):
# If explicit tokens provided, gate on tokens; otherwise assume we were called on actual mutation """Save progress only when tokens indicate work or when tokens is None but we know a mutation occurred."""
if tokens is not None: try:
if not (isinstance(tokens, (list, tuple)) and len(tokens) >= 2 and (tokens[0] or tokens[1])): # If explicit tokens provided, gate on tokens; otherwise assume we were called on actual mutation
return if tokens is not None:
save_progress_lines(lines, filename, encoding=encoding) if not (isinstance(tokens, (list, tuple)) and len(tokens) >= 2 and (tokens[0] or tokens[1])):
except Exception: return
traceback.print_exc() save_progress_lines(lines, filename, encoding=encoding)
except Exception:
traceback.print_exc()
def translateTyrano(data, filename, translatedList):
if translatedList:
stringList = translatedList[0] def translateTyrano(data, filename, translatedList):
choiceList = translatedList[1] if translatedList:
else: stringList = translatedList[0]
stringList = [] choiceList = translatedList[1]
choiceList = [] else:
tokens = [0, 0] stringList = []
speaker = "" choiceList = []
stringListTL = [] tokens = [0, 0]
choiceListTL = [] speaker = ""
global LOCK, ESTIMATE, FILENAME, PBAR, MISMATCH stringListTL = []
i = 0 choiceListTL = []
global LOCK, ESTIMATE, FILENAME, PBAR, MISMATCH
while i < len(data): i = 0
voice = False
lineRegexNoSpeaker = r"^([^\[#;*@\n]+)\[p\]|^([^\[#;*@\n]+)\[l\]\[[rp]\]|^([^\[#;*@\n]+)\[[rpl]\]|^([^\[#;*@_\n]+)\n$" while i < len(data):
lineRegexSpeaker = r"^#(.*)" voice = False
furiganaRegex = r"(\[ruby\stext=(.*?)\])" lineRegexNoSpeaker = r"^([^\[#;*@\n]+)\[p\]|^([^\[#;*@\n]+)\[l\]\[[rp]\]|^([^\[#;*@\n]+)\[[rpl]\]|^([^\[#;*@_\n]+)\n$"
choiceRegex = r'\[glink.+?text="(.*?)"' lineRegexSpeaker = r"^#(.*)"
furiganaRegex = r"(\[ruby\stext=(.*?)\])"
# Speaker choiceRegex = r'\[glink.+?text="(.*?)"'
match = re.search(lineRegexSpeaker, data[i])
if match: # Speaker
if match.group(1): match = re.search(lineRegexSpeaker, data[i])
response = getSpeaker(match.group(1)) if match:
speaker = response[0] if match.group(1):
tokens[0] += response[1][0] response = getSpeaker(match.group(1))
tokens[1] += response[1][1] speaker = response[0]
data[i] = data[i].replace(match.group(1), speaker) tokens[0] += response[1][0]
else: tokens[1] += response[1][1]
speaker = None data[i] = data[i].replace(match.group(1), speaker)
else:
# Furigana speaker = None
match = re.search(r"^\[ruby\stext", data[i])
furiganaList = [] # Furigana
if match: match = re.search(r"^\[ruby\stext", data[i])
# Check next line and combine furiganaList = []
while match: if match:
furiganaList.append(data[i].replace("\n", "")) # Check next line and combine
del data[i] while match:
match = re.search(r"^\[ruby\stext", data[i]) furiganaList.append(data[i].replace("\n", ""))
jaString = "".join(furiganaList) del data[i]
match = re.search(r"^\[ruby\stext", data[i])
# Ruby Text jaString = "".join(furiganaList)
furiganaList = re.findall(furiganaRegex, jaString)
for furigana in furiganaList: # Ruby Text
jaString = jaString.replace(furigana[0], furigana[1]) furiganaList = re.findall(furiganaRegex, jaString)
for furigana in furiganaList:
data.insert(i, f"{jaString}[r]") jaString = jaString.replace(furigana[0], furigana[1])
# Dialogue data.insert(i, f"{jaString}[r]")
match = re.search(lineRegexNoSpeaker, data[i])
jaString = None # Dialogue
if match: match = re.search(lineRegexNoSpeaker, data[i])
# Find which group matched jaString = None
for group_num in range(1, 5): if match:
try: # Find which group matched
if match.group(group_num): for group_num in range(1, 5):
jaString = match.group(group_num) try:
break if match.group(group_num):
except IndexError: jaString = match.group(group_num)
break break
except IndexError:
# Skip if no valid string found break
if not jaString:
i += 1 # Skip if no valid string found
continue if not jaString:
i += 1
# Combine w/ next line if necessary continue
repeatRegex = r"(.+?)\[[rpl]\]"
if i + 1 < len(data): # Combine w/ next line if necessary
match = re.search(repeatRegex, data[i + 1]) repeatRegex = r"(.+?)\[[rpl]\]"
while match and "[p]" not in data[i]: if i + 1 < len(data):
jaString = jaString + match.group(1) match = re.search(repeatRegex, data[i + 1])
jaString = jaString.replace("_　", "") while match and "[p]" not in data[i]:
if "[p]" in data[i + 1]: jaString = jaString + match.group(1)
data[i] = f"{jaString}[p]\n" jaString = jaString.replace("_　", "")
else: if "[p]" in data[i + 1]:
data[i] = jaString data[i] = f"{jaString}[p]\n"
del data[i + 1] else:
if i + 1 < len(data): data[i] = jaString
match = re.search(repeatRegex, data[i + 1]) del data[i + 1]
else: if i + 1 < len(data):
break match = re.search(repeatRegex, data[i + 1])
else:
originalString = jaString break
# Pass 1 originalString = jaString
if not translatedList:
# Remove any textwrap and commands # Pass 1
jaString = jaString.replace("[r]", " ") if not translatedList:
jaString = jaString.replace("[l]", "") # Remove any textwrap and commands
jaString = jaString.replace("[r]", " ")
# Ruby Text jaString = jaString.replace("[l]", "")
furiganaList = re.findall(furiganaRegex, jaString)
for furigana in furiganaList: # Ruby Text
jaString = jaString.replace(furigana[0], furigana[1]) furiganaList = re.findall(furiganaRegex, jaString)
for furigana in furiganaList:
# Strip Spaces jaString = jaString.replace(furigana[0], furigana[1])
jaString = jaString.strip()
# Strip Spaces
if jaString: jaString = jaString.strip()
if speaker:
stringList.append(f"[{speaker}]: {jaString}") if jaString:
else: if speaker:
stringList.append(jaString) stringList.append(f"[{speaker}]: {jaString}")
else:
# Pass 2 stringList.append(jaString)
else:
# Get Text # Pass 2
if stringList: else:
# Grab and Pop # Get Text
translatedText = stringList[0] if stringList:
stringList.pop(0) # Grab and Pop
translatedText = stringList[0]
# Set to None if empty list stringList.pop(0)
if len(stringList) <= 0:
stringList = None # Set to None if empty list
if len(stringList) <= 0:
# Remove speaker stringList = None
if speaker != "":
matchSpeakerList = re.findall(r"^\[?(.+?)\]?\s?[|:]\s?", translatedText) # Remove speaker
translatedText = re.sub(r"^\[?(.+?)\]?\s?[|:]\s?", "", translatedText) if speaker != "":
matchSpeakerList = re.findall(r"^\[?(.+?)\]?\s?[|:]\s?", translatedText)
# Avoid Crashes translatedText = re.sub(r"^\[?(.+?)\]?\s?[|:]\s?", "", translatedText)
translatedText = translatedText.replace("[", "(")
translatedText = translatedText.replace("]", ")") # Avoid Crashes
translatedText = translatedText.replace("[", "(")
# Textwrap translatedText = translatedText.replace("]", ")")
translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
translatedText = translatedText.replace("\n", "[r]") # Textwrap
translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
# Set Data translatedText = translatedText.replace("\n", "[r]")
data[i] = data[i].replace(originalString, translatedText)
# Save progress after each line change (we mutated, so tokens gate optional) # Set Data
saveCheckLines(data, filename) data[i] = data[i].replace(originalString, translatedText)
# Save progress after each line change (we mutated, so tokens gate optional)
# Choices saveCheckLines(data, filename)
match = re.search(choiceRegex, data[i])
if match: # Choices
# Pass 1 match = re.search(choiceRegex, data[i])
if not translatedList: if match:
choiceList.append(match.group(1)) # Pass 1
match = re.search(choiceRegex, data[i + 1]) if not translatedList:
choiceList.append(match.group(1))
# Pass 2 match = re.search(choiceRegex, data[i + 1])
else:
# Grab and Pop # Pass 2
translatedText = choiceList[0] else:
choiceList.pop(0) # Grab and Pop
translatedText = choiceList[0]
# Replace Spaces choiceList.pop(0)
translatedText = translatedText.replace(" ", "\u3000")
# Replace Spaces
# Set translatedText = translatedText.replace(" ", "\u3000")
data[i] = data[i].replace(match.group(1), translatedText)
saveCheckLines(data, filename) # Set
data[i] = data[i].replace(match.group(1), translatedText)
i += 1 saveCheckLines(data, filename)
else:
i += 1 i += 1
else:
# EOF i += 1
if not translatedList:
# String List # EOF
if stringList: if not translatedList:
PBAR.total = len(stringList) # String List
PBAR.refresh() if stringList:
response = translateAI(stringList, "Reply with the English Translation") PBAR.total = len(stringList)
tokens[0] += response[1][0] PBAR.refresh()
tokens[1] += response[1][1] response = translateAI(stringList, "Reply with the English Translation")
stringListTL = response[0] tokens[0] += response[1][0]
tokens[1] += response[1][1]
if len(stringList) != len(stringListTL): stringListTL = response[0]
# Mismatch
with LOCK: if len(stringList) != len(stringListTL):
if FILENAME not in MISMATCH: # Mismatch
MISMATCH.append(FILENAME) with LOCK:
if FILENAME not in MISMATCH:
# Choice List MISMATCH.append(FILENAME)
if choiceList:
response = translateAI(choiceList, "Reply with the English TL of the Dialogue Choice") # Choice List
tokens[0] += response[1][0] if choiceList:
tokens[1] += response[1][1] response = translateAI(choiceList, "Reply with the English TL of the Dialogue Choice")
choiceListTL = response[0] tokens[0] += response[1][0]
tokens[1] += response[1][1]
if len(choiceList) != len(choiceListTL): choiceListTL = response[0]
# Mismatch
with LOCK: if len(choiceList) != len(choiceListTL):
if FILENAME not in MISMATCH: # Mismatch
MISMATCH.append(FILENAME) with LOCK:
if FILENAME not in MISMATCH:
# Recursive call for Pass 2 with translated strings MISMATCH.append(FILENAME)
result = translateTyrano(data, filename, [stringListTL, choiceListTL])
tokens[0] += result[0] # Recursive call for Pass 2 with translated strings
tokens[1] += result[1] result = translateTyrano(data, filename, [stringListTL, choiceListTL])
tokens[0] += result[0]
return tokens tokens[1] += result[1]
# Save some money and enter the character before translation return tokens
def getSpeaker(speaker):
match speaker: # Save some money and enter the character before translation
case "ファイン": def getSpeaker(speaker):
return ["Fine", [0, 0]] match speaker:
case "": case "ファイン":
return ["", [0, 0]] return ["Fine", [0, 0]]
case _: case "":
# Find Speaker return ["", [0, 0]]
for i in range(len(NAMESLIST)): case _:
if speaker == NAMESLIST[i][0]: # Find Speaker
return [NAMESLIST[i][1], [0, 0]] for i in range(len(NAMESLIST)):
if speaker == NAMESLIST[i][0]:
# Translate and Store Speaker return [NAMESLIST[i][1], [0, 0]]
response = translateAI(
f"{speaker}", # Translate and Store Speaker
"Reply with the " + LANGUAGE + " translation of the NPC name.", response = translateAI(
False, f"{speaker}",
) "Reply with the " + LANGUAGE + " translation of the NPC name.",
response[0] = response[0].title() False,
response[0] = response[0].replace("'S", "'s") )
response[0] = response[0].replace("Speaker: ", "") response[0] = response[0].title()
response[0] = response[0].replace("'S", "'s")
# Retry if name doesn't translate for some reason response[0] = response[0].replace("Speaker: ", "")
if re.search(r"([a-zA-Z？?])", response[0]) == None:
response = translateAI( # Retry if name doesn't translate for some reason
f"{speaker}", if re.search(r"([a-zA-Z？?])", response[0]) == None:
"Reply with the " + LANGUAGE + " translation of the NPC name.", response = translateAI(
False, f"{speaker}",
) "Reply with the " + LANGUAGE + " translation of the NPC name.",
response[0] = response[0].title() False,
response[0] = response[0].replace("'S", "'s") )
response[0] = response[0].title()
speakerList = [speaker, response[0]] response[0] = response[0].replace("'S", "'s")
NAMESLIST.append(speakerList)
return response speakerList = [speaker, response[0]]
return [speaker, [0, 0]] NAMESLIST.append(speakerList)
return response
def translateAI(text, history, history_ctx=None): return [speaker, [0, 0]]
"""
Legacy wrapper function for the new shared translation utility. def translateAI(text, history, history_ctx=None):
This maintains compatibility with existing code while using the new shared implementation. """
""" Legacy wrapper function for the new shared translation utility.
global PBAR, MISMATCH, FILENAME This maintains compatibility with existing code while using the new shared implementation.
"""
# Update config estimate mode based on global ESTIMATE global PBAR, MISMATCH, FILENAME
TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
# Update config estimate mode based on global ESTIMATE
# Call the new shared translation function TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
return sharedtranslateAI(
text=text, # Call the new shared translation function
history=history, return sharedtranslateAI(
config=TRANSLATION_CONFIG, text=text,
filename=FILENAME, history=history,
pbar=PBAR, config=TRANSLATION_CONFIG,
lock=LOCK, filename=FILENAME,
mismatchList=MISMATCH pbar=PBAR,
) lock=LOCK,
mismatchList=MISMATCH
)

708
modules/unity.py
View file

@ -1,353 +1,355 @@
# Libraries # Libraries
import json import json
import os import os
import re import re
import util.dazedwrap as dazedwrap import util.dazedwrap as dazedwrap
import threading import threading
import time import time
import traceback import traceback
import tiktoken import tiktoken
from pathlib import Path from pathlib import Path
from colorama import Fore from colorama import Fore
from dotenv import load_dotenv from dotenv import load_dotenv
from retry import retry from retry import retry
from tqdm import tqdm from tqdm import tqdm
from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost, getPricingConfig, calculateCost from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost, getPricingConfig, calculateCost
import tempfile import tempfile
# OpenAI initialization centralized in util/translation.py # OpenAI initialization centralized in util/translation.py
# Globals # Globals
MODEL = os.getenv("model") MODEL = os.getenv("model")
TIMEOUT = int(os.getenv("timeout")) TIMEOUT = int(os.getenv("timeout"))
LANGUAGE = os.getenv("language").capitalize() LANGUAGE = os.getenv("language").capitalize()
PROMPT = Path("prompt.txt").read_text(encoding="utf-8") from util.paths import PROMPT_PATH, VOCAB_PATH
VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
LOCK = threading.Lock() PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
WIDTH = int(os.getenv("width")) VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
LISTWIDTH = int(os.getenv("listWidth")) LOCK = threading.Lock()
NOTEWIDTH = 70 WIDTH = int(os.getenv("width"))
MAXHISTORY = 10 LISTWIDTH = int(os.getenv("listWidth"))
ESTIMATE = "" NOTEWIDTH = 70
TOKENS = [0, 0] MAXHISTORY = 10
NAMESLIST = [] ESTIMATE = ""
NAMES = False # Output a list of all the character names found TOKENS = [0, 0]
BRFLAG = False # If the game uses <br> instead NAMESLIST = []
FIXTEXTWRAP = True # Overwrites textwrap NAMES = False # Output a list of all the character names found
IGNORETLTEXT = False # Ignores all translated text. BRFLAG = False # If the game uses <br> instead
MISMATCH = [] # Lists files that throw a mismatch error (Length of GPT list response is wrong) FIXTEXTWRAP = True # Overwrites textwrap
IGNORETLTEXT = False # Ignores all translated text.
# tqdm Globals MISMATCH = [] # Lists files that throw a mismatch error (Length of GPT list response is wrong)
BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
POSITION = 0 # tqdm Globals
LEAVE = False BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
PBAR = None POSITION = 0
FILENAME = None LEAVE = False
PBAR = None
# Full Width FILENAME = None
ascii_to_wide = dict((i, chr(i + 0xFEE0)) for i in range(0x21, 0x7F))
ascii_to_wide.update({0x20: "\u3000", 0x2D: "\u2212"}) # space and minus # Full Width
wide_to_ascii = dict((i, chr(i - 0xFEE0)) for i in range(0xFF01, 0xFF5F)) ascii_to_wide = dict((i, chr(i + 0xFEE0)) for i in range(0x21, 0x7F))
wide_to_ascii.update({0x3000: " ", 0x2212: "-"}) # space and minus ascii_to_wide.update({0x20: "\u3000", 0x2D: "\u2212"}) # space and minus
wide_to_ascii = dict((i, chr(i - 0xFEE0)) for i in range(0xFF01, 0xFF5F))
# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex wide_to_ascii.update({0x3000: " ", 0x2212: "-"}) # space and minus
LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
# Get pricing configuration based on the model LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
PRICING_CONFIG = getPricingConfig(MODEL)
INPUTAPICOST = PRICING_CONFIG["inputAPICost"] # Get pricing configuration based on the model
OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"] PRICING_CONFIG = getPricingConfig(MODEL)
BATCHSIZE = PRICING_CONFIG["batchSize"] INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"] OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
BATCHSIZE = PRICING_CONFIG["batchSize"]
# Initialize Translation Config FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
TRANSLATION_CONFIG = TranslationConfig(
model=MODEL, # Initialize Translation Config
language=LANGUAGE, TRANSLATION_CONFIG = TranslationConfig(
prompt=PROMPT, model=MODEL,
vocab=VOCAB, language=LANGUAGE,
langRegex=LANGREGEX, prompt=PROMPT,
batchSize=BATCHSIZE, vocab=VOCAB,
maxHistory=MAXHISTORY, langRegex=LANGREGEX,
estimateMode=False # Will be set dynamically based on ESTIMATE batchSize=BATCHSIZE,
) maxHistory=MAXHISTORY,
LEAVE = False estimateMode=False # Will be set dynamically based on ESTIMATE
)
def handleUnity(filename, estimate): LEAVE = False
global ESTIMATE, FILENAME
ESTIMATE = estimate def handleUnity(filename, estimate):
FILENAME = filename global ESTIMATE, FILENAME
ESTIMATE = estimate
if ESTIMATE: FILENAME = filename
start = time.time()
translatedData = openFiles(filename) if ESTIMATE:
start = time.time()
# Print Result translatedData = openFiles(filename)
end = time.time()
tqdm.write(getResultString(translatedData, end - start, filename)) # Print Result
with LOCK: end = time.time()
TOKENS[0] += translatedData[1][0] tqdm.write(getResultString(translatedData, end - start, filename))
TOKENS[1] += translatedData[1][1] with LOCK:
TOKENS[0] += translatedData[1][0]
# Print Total TOKENS[1] += translatedData[1][1]
totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
# Print Total
# Print any errors on maps totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
if len(MISMATCH) > 0:
return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET # Print any errors on maps
else: if len(MISMATCH) > 0:
return totalString return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
else:
else: return totalString
try:
with open("translated/" + filename, "w", encoding="utf8", errors="ignore") as outFile: else:
start = time.time() try:
translatedData = openFiles(filename) with open("translated/" + filename, "w", encoding="utf8", errors="ignore") as outFile:
start = time.time()
# Print Result translatedData = openFiles(filename)
end = time.time()
outFile.writelines(translatedData[0]) # Print Result
tqdm.write(getResultString(translatedData, end - start, filename)) end = time.time()
with LOCK: outFile.writelines(translatedData[0])
TOKENS[0] += translatedData[1][0] tqdm.write(getResultString(translatedData, end - start, filename))
TOKENS[1] += translatedData[1][1] with LOCK:
except Exception: TOKENS[0] += translatedData[1][0]
traceback.print_exc() TOKENS[1] += translatedData[1][1]
return "Fail" except Exception:
traceback.print_exc()
return getResultString(["", TOKENS, None], end - start, "TOTAL") return "Fail"
return getResultString(["", TOKENS, None], end - start, "TOTAL")
def getResultString(translatedData, translationTime, filename):
# File Print String
cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL) def getResultString(translatedData, translationTime, filename):
totalTokenstring = ( # File Print String
Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]" cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
"[Output: " totalTokenstring = (
+ str(translatedData[1][1]) Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
+ "]" "[Cost: ${:,.4f}".format(cost) "[Output: "
+ "]" + str(translatedData[1][1])
) + "]" "[Cost: ${:,.4f}".format(cost)
timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]" + "]"
)
if translatedData[2] == None: timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
# Success
return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET if translatedData[2] == None:
# Success
else: return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
# Fail
try: else:
raise translatedData[2] # Fail
except Exception as e: try:
traceback.print_exc() raise translatedData[2]
errorString = str(e) + Fore.RED except Exception as e:
return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET traceback.print_exc()
errorString = str(e) + Fore.RED
return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
def openFiles(filename):
with open("files/" + filename, "r", encoding="utf8") as readFile:
translatedData = parseUnity(readFile, filename) def openFiles(filename):
with open("files/" + filename, "r", encoding="utf8") as readFile:
# Delete lines marked for deletion translatedData = parseUnity(readFile, filename)
finalData = []
for line in translatedData[0]: # Delete lines marked for deletion
if line != "\\d\n": finalData = []
finalData.append(line) for line in translatedData[0]:
translatedData[0] = finalData if line != "\\d\n":
finalData.append(line)
return translatedData translatedData[0] = finalData
return translatedData
def parseUnity(readFile, filename):
totalTokens = [0, 0]
def parseUnity(readFile, filename):
# Read File into data totalTokens = [0, 0]
data = readFile.readlines()
# Read File into data
# Create Progress Bar data = readFile.readlines()
with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
pbar.desc = filename # Create Progress Bar
with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
try: pbar.desc = filename
result = translateUnity(data, pbar, filename, [])
totalTokens[0] += result[0] try:
totalTokens[1] += result[1] result = translateUnity(data, pbar, filename, [])
except Exception as e: totalTokens[0] += result[0]
traceback.print_exc() totalTokens[1] += result[1]
return [data, totalTokens, e] except Exception as e:
return [data, totalTokens, None] traceback.print_exc()
return [data, totalTokens, e]
return [data, totalTokens, None]
def save_progress_lines(lines, filename, encoding="utf-8"):
"""Atomically save current line-based translation progress."""
try: def save_progress_lines(lines, filename, encoding="utf-8"):
if ESTIMATE: """Atomically save current line-based translation progress."""
return try:
os.makedirs("translated", exist_ok=True) if ESTIMATE:
tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated") return
try: os.makedirs("translated", exist_ok=True)
with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file: tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
tmp_file.writelines(lines) try:
os.replace(tmp_path, os.path.join("translated", filename)) with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file:
finally: tmp_file.writelines(lines)
if os.path.exists(tmp_path): os.replace(tmp_path, os.path.join("translated", filename))
try: finally:
os.remove(tmp_path) if os.path.exists(tmp_path):
except OSError: try:
pass os.remove(tmp_path)
except Exception: except OSError:
traceback.print_exc() pass
except Exception:
traceback.print_exc()
def saveCheckLines(lines, filename, tokens=None, encoding="utf-8"):
"""Save progress only when tokens indicate work or when explicitly called after a mutation."""
try: def saveCheckLines(lines, filename, tokens=None, encoding="utf-8"):
if tokens is not None: """Save progress only when tokens indicate work or when explicitly called after a mutation."""
if not (isinstance(tokens, (list, tuple)) and len(tokens) >= 2 and (tokens[0] or tokens[1])): try:
return if tokens is not None:
save_progress_lines(lines, filename, encoding=encoding) if not (isinstance(tokens, (list, tuple)) and len(tokens) >= 2 and (tokens[0] or tokens[1])):
except Exception: return
traceback.print_exc() save_progress_lines(lines, filename, encoding=encoding)
except Exception:
traceback.print_exc()
def translateUnity(data, pbar, filename, translatedList):
stringList = []
tokens = [0, 0] def translateUnity(data, pbar, filename, translatedList):
global LOCK, ESTIMATE, PBAR stringList = []
PBAR = pbar tokens = [0, 0]
i = 0 global LOCK, ESTIMATE, PBAR
PBAR = pbar
# Dialogue i = 0
while i < len(data):
# Lines # Dialogue
regex = r"(.*?)=(.*)" while i < len(data):
match = re.search(regex, data[i]) # Lines
if match: regex = r"(.*?)=(.*)"
leftString = match.group(1) match = re.search(regex, data[i])
rightString = match.group(2) if match:
leftString = match.group(1)
# Validate Japanese Text rightString = match.group(2)
if not re.search(LANGREGEX, leftString) and IGNORETLTEXT:
i += 1 # Validate Japanese Text
continue if not re.search(LANGREGEX, leftString) and IGNORETLTEXT:
i += 1
# Pass 1 continue
if translatedList == []:
jaString = leftString # Pass 1
if translatedList == []:
# Remove textwrap jaString = leftString
# jaString = jaString.replace("\\n", " ")
# Remove textwrap
# Add String # jaString = jaString.replace("\\n", " ")
stringList.append(jaString.strip())
# Add String
# Pass 2 stringList.append(jaString.strip())
else:
# Get Text # Pass 2
if translatedList: else:
# Grab and Pop # Get Text
translatedText = translatedList[0] if translatedList:
translatedList.pop(0) # Grab and Pop
translatedText = translatedList[0]
# Set to None if empty list translatedList.pop(0)
if len(translatedList) <= 0:
translatedList = None # Set to None if empty list
if len(translatedList) <= 0:
# Textwrap translatedList = None
# translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
# translatedText = translatedText.replace("\n", "\\n") # Textwrap
# translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
# Remove Double Spaces and = # translatedText = translatedText.replace("\n", "\\n")
translatedText = translatedText.replace(" ", " ")
translatedText = translatedText.replace("=", "->") # Remove Double Spaces and =
translatedText = translatedText.replace(" ", " ")
# Set Data translatedText = translatedText.replace("=", "->")
data[i] = f"{leftString}={translatedText}\n"
saveCheckLines(data, filename) # Set Data
i += 1 data[i] = f"{leftString}={translatedText}\n"
saveCheckLines(data, filename)
# Nothing relevant. Skip Line. i += 1
else:
i += 1 # Nothing relevant. Skip Line.
else:
# EOF i += 1
if len(stringList) > 0:
# Set Progress # EOF
pbar.total = len(stringList) if len(stringList) > 0:
pbar.refresh() # Set Progress
pbar.total = len(stringList)
# Translate pbar.refresh()
response = translateAI(stringList, "")
tokens[0] += response[1][0] # Translate
tokens[1] += response[1][1] response = translateAI(stringList, "")
translatedList = response[0] tokens[0] += response[1][0]
tokens[1] += response[1][1]
# Set Strings translatedList = response[0]
if len(stringList) == len(translatedList):
translateUnity(data, pbar, filename, translatedList) # Set Strings
if len(stringList) == len(translatedList):
# Mismatch translateUnity(data, pbar, filename, translatedList)
else:
with LOCK: # Mismatch
if filename not in MISMATCH: else:
MISMATCH.append(filename) with LOCK:
return tokens if filename not in MISMATCH:
MISMATCH.append(filename)
# Save some money and enter the character before translation return tokens
def getSpeaker(speaker):
match speaker: # Save some money and enter the character before translation
case "ファイン": def getSpeaker(speaker):
return ["Fine", [0, 0]] match speaker:
case "": case "ファイン":
return ["", [0, 0]] return ["Fine", [0, 0]]
case _: case "":
# Find Speaker return ["", [0, 0]]
for i in range(len(NAMESLIST)): case _:
if speaker == NAMESLIST[i][0]: # Find Speaker
return [NAMESLIST[i][1], [0, 0]] for i in range(len(NAMESLIST)):
if speaker == NAMESLIST[i][0]:
# Translate and Store Speaker return [NAMESLIST[i][1], [0, 0]]
response = translateAI(
f"{speaker}", # Translate and Store Speaker
"Reply with the " + LANGUAGE + " translation of the NPC name.", response = translateAI(
False, f"{speaker}",
) "Reply with the " + LANGUAGE + " translation of the NPC name.",
response[0] = response[0].title() False,
response[0] = response[0].replace("'S", "'s") )
response[0] = response[0].replace("Speaker: ", "") response[0] = response[0].title()
response[0] = response[0].replace("'S", "'s")
# Retry if name doesn't translate for some reason response[0] = response[0].replace("Speaker: ", "")
if re.search(r"([a-zA-Z？?])", response[0]) == None:
response = translateAI( # Retry if name doesn't translate for some reason
f"{speaker}", if re.search(r"([a-zA-Z？?])", response[0]) == None:
"Reply with the " + LANGUAGE + " translation of the NPC name.", response = translateAI(
False, f"{speaker}",
) "Reply with the " + LANGUAGE + " translation of the NPC name.",
response[0] = response[0].title() False,
response[0] = response[0].replace("'S", "'s") )
response[0] = response[0].title()
speakerList = [speaker, response[0]] response[0] = response[0].replace("'S", "'s")
NAMESLIST.append(speakerList)
return response speakerList = [speaker, response[0]]
return [speaker, [0, 0]] NAMESLIST.append(speakerList)
return response
def translateAI(text, history, history_ctx=None): return [speaker, [0, 0]]
"""
Legacy wrapper function for the new shared translation utility. def translateAI(text, history, history_ctx=None):
This maintains compatibility with existing code while using the new shared implementation. """
""" Legacy wrapper function for the new shared translation utility.
global PBAR, MISMATCH, FILENAME This maintains compatibility with existing code while using the new shared implementation.
"""
# Update config estimate mode based on global ESTIMATE global PBAR, MISMATCH, FILENAME
TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
# Update config estimate mode based on global ESTIMATE
# Call the new shared translation function TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
return sharedtranslateAI(
text=text, # Call the new shared translation function
history=history, return sharedtranslateAI(
config=TRANSLATION_CONFIG, text=text,
filename=FILENAME, history=history,
pbar=PBAR, config=TRANSLATION_CONFIG,
lock=LOCK, filename=FILENAME,
mismatchList=MISMATCH pbar=PBAR,
) lock=LOCK,
mismatchList=MISMATCH
)

5400
modules/wolf.py
View file

File diff suppressed because it is too large Load diff

842
modules/wolf2.py
View file

@ -1,420 +1,422 @@
# Libraries # Libraries
import json import json
import io import io
import os import os
import re import re
import util.dazedwrap as dazedwrap import util.dazedwrap as dazedwrap
import threading import threading
import time import time
import traceback import traceback
import tiktoken import tiktoken
from pathlib import Path from pathlib import Path
from colorama import Fore from colorama import Fore
from dotenv import load_dotenv from dotenv import load_dotenv
from retry import retry from retry import retry
from tqdm import tqdm from tqdm import tqdm
from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
import tempfile import tempfile
# OpenAI initialization centralized in util/translation.py # OpenAI initialization centralized in util/translation.py
# Globals # Globals
MODEL = os.getenv("model") MODEL = os.getenv("model")
TIMEOUT = int(os.getenv("timeout")) TIMEOUT = int(os.getenv("timeout"))
LANGUAGE = os.getenv("language").capitalize() LANGUAGE = os.getenv("language").capitalize()
PROMPT = Path("prompt.txt").read_text(encoding="utf-8") from util.paths import PROMPT_PATH, VOCAB_PATH
VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
LOCK = threading.Lock() PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
WIDTH = int(os.getenv("width")) VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
LISTWIDTH = int(os.getenv("listWidth")) LOCK = threading.Lock()
NOTEWIDTH = 70 WIDTH = int(os.getenv("width"))
MAXHISTORY = 10 LISTWIDTH = int(os.getenv("listWidth"))
ESTIMATE = "" NOTEWIDTH = 70
TOKENS = [0, 0] MAXHISTORY = 10
NAMESLIST = [] ESTIMATE = ""
NAMES = False # Output a list of all the character names found TOKENS = [0, 0]
BRFLAG = False # If the game uses <br> instead NAMESLIST = []
FIXTEXTWRAP = True # Overwrites textwrap NAMES = False # Output a list of all the character names found
IGNORETLTEXT = False # Ignores all translated text. BRFLAG = False # If the game uses <br> instead
MISMATCH = [] # Lists files that throw a mismatch error (Length of GPT list response is wrong) FIXTEXTWRAP = True # Overwrites textwrap
FILENAME = "" # Current file being processed, used by translateAI wrapper IGNORETLTEXT = False # Ignores all translated text.
MISMATCH = [] # Lists files that throw a mismatch error (Length of GPT list response is wrong)
# tqdm Globals FILENAME = "" # Current file being processed, used by translateAI wrapper
BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
POSITION = 0 # tqdm Globals
LEAVE = False BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
POSITION = 0
# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex LEAVE = False
LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
# Get pricing configuration based on the model LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
PRICING_CONFIG = getPricingConfig(MODEL)
INPUTAPICOST = PRICING_CONFIG["inputAPICost"] # Get pricing configuration based on the model
OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"] PRICING_CONFIG = getPricingConfig(MODEL)
BATCHSIZE = PRICING_CONFIG["batchSize"] INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"] OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
BATCHSIZE = PRICING_CONFIG["batchSize"]
# Initialize Translation Config FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
TRANSLATION_CONFIG = TranslationConfig(
model=MODEL, # Initialize Translation Config
language=LANGUAGE, TRANSLATION_CONFIG = TranslationConfig(
prompt=PROMPT, model=MODEL,
vocab=VOCAB, language=LANGUAGE,
langRegex=LANGREGEX, prompt=PROMPT,
batchSize=BATCHSIZE, vocab=VOCAB,
maxHistory=MAXHISTORY, langRegex=LANGREGEX,
estimateMode=False # Will be set dynamically based on ESTIMATE batchSize=BATCHSIZE,
) maxHistory=MAXHISTORY,
LEAVE = False estimateMode=False # Will be set dynamically based on ESTIMATE
)
def handleWOLF2(filename, estimate): LEAVE = False
global ESTIMATE, FILENAME
ESTIMATE = estimate def handleWOLF2(filename, estimate):
FILENAME = filename global ESTIMATE, FILENAME
ESTIMATE = estimate
if ESTIMATE: FILENAME = filename
start = time.time()
translatedData = openFiles(filename) if ESTIMATE:
start = time.time()
# Print Result translatedData = openFiles(filename)
end = time.time()
tqdm.write(getResultString(translatedData, end - start, filename)) # Print Result
with LOCK: end = time.time()
TOKENS[0] += translatedData[1][0] tqdm.write(getResultString(translatedData, end - start, filename))
TOKENS[1] += translatedData[1][1] with LOCK:
TOKENS[0] += translatedData[1][0]
# Print Total TOKENS[1] += translatedData[1][1]
totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
# Print Total
# Print any errors on maps totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
if len(MISMATCH) > 0:
return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET # Print any errors on maps
else: if len(MISMATCH) > 0:
return totalString return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
else:
else: return totalString
try:
with open("translated/" + filename, "w", encoding="shift_jis", errors="ignore") as outFile: else:
start = time.time() try:
translatedData = openFiles(filename) with open("translated/" + filename, "w", encoding="shift_jis", errors="ignore") as outFile:
start = time.time()
# Print Result translatedData = openFiles(filename)
end = time.time()
outFile.writelines(translatedData[0]) # Print Result
tqdm.write(getResultString(translatedData, end - start, filename)) end = time.time()
with LOCK: outFile.writelines(translatedData[0])
TOKENS[0] += translatedData[1][0] tqdm.write(getResultString(translatedData, end - start, filename))
TOKENS[1] += translatedData[1][1] with LOCK:
except Exception: TOKENS[0] += translatedData[1][0]
traceback.print_exc() TOKENS[1] += translatedData[1][1]
return "Fail" except Exception:
traceback.print_exc()
return getResultString(["", TOKENS, None], end - start, "TOTAL") return "Fail"
return getResultString(["", TOKENS, None], end - start, "TOTAL")
def getResultString(translatedData, translationTime, filename):
# File Print String
cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL) def getResultString(translatedData, translationTime, filename):
totalTokenstring = ( # File Print String
Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]" cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
"[Output: " totalTokenstring = (
+ str(translatedData[1][1]) Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
+ "]" "[Cost: ${:,.4f}".format(cost) "[Output: "
+ "]" + str(translatedData[1][1])
) + "]" "[Cost: ${:,.4f}".format(cost)
timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]" + "]"
)
if translatedData[2] == None: timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
# Success
return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET if translatedData[2] == None:
# Success
else: return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
# Fail
try: else:
raise translatedData[2] # Fail
except Exception as e: try:
traceback.print_exc() raise translatedData[2]
errorString = str(e) + Fore.RED except Exception as e:
return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET traceback.print_exc()
errorString = str(e) + Fore.RED
return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
def openFiles(filename):
# Use a robust reader to handle cp932/Shift_JIS variants and occasional bad bytes
def read_file_lines_with_fallback(path: str): def openFiles(filename):
encodings = ["cp932", "shift_jis", "utf-8-sig", "utf-8"] # Use a robust reader to handle cp932/Shift_JIS variants and occasional bad bytes
for enc in encodings: def read_file_lines_with_fallback(path: str):
try: encodings = ["cp932", "shift_jis", "utf-8-sig", "utf-8"]
with open(path, "r", encoding=enc) as f: for enc in encodings:
return f.readlines() try:
except UnicodeDecodeError: with open(path, "r", encoding=enc) as f:
continue return f.readlines()
# Last resort: ignore undecodable bytes under cp932 except UnicodeDecodeError:
with open(path, "rb") as f: continue
raw = f.read() # Last resort: ignore undecodable bytes under cp932
try: with open(path, "rb") as f:
text = raw.decode("cp932", errors="ignore") raw = f.read()
except Exception: try:
text = raw.decode("latin-1", errors="ignore") text = raw.decode("cp932", errors="ignore")
return text.splitlines(keepends=True) except Exception:
text = raw.decode("latin-1", errors="ignore")
path = os.path.join("files", filename) return text.splitlines(keepends=True)
lines = read_file_lines_with_fallback(path)
path = os.path.join("files", filename)
# Keep parseWOLF API by wrapping lines in a file-like object lines = read_file_lines_with_fallback(path)
with io.StringIO("".join(lines)) as readFile:
translatedData = parseWOLF(readFile, filename) # Keep parseWOLF API by wrapping lines in a file-like object
with io.StringIO("".join(lines)) as readFile:
# Delete lines marked for deletion translatedData = parseWOLF(readFile, filename)
finalData = []
for line in translatedData[0]: # Delete lines marked for deletion
if line != "\\d\n": finalData = []
finalData.append(line) for line in translatedData[0]:
translatedData[0] = finalData if line != "\\d\n":
finalData.append(line)
return translatedData translatedData[0] = finalData
return translatedData
def parseWOLF(readFile, filename):
totalTokens = [0, 0]
def parseWOLF(readFile, filename):
# Read File into data totalTokens = [0, 0]
data = readFile.readlines()
# Read File into data
# Create Progress Bar data = readFile.readlines()
with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
pbar.desc = filename # Create Progress Bar
with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
try: pbar.desc = filename
result = translateWOLF(data, [], pbar, filename)
totalTokens[0] += result[0] try:
totalTokens[1] += result[1] result = translateWOLF(data, [], pbar, filename)
except Exception as e: totalTokens[0] += result[0]
traceback.print_exc() totalTokens[1] += result[1]
return [data, totalTokens, e] except Exception as e:
return [data, totalTokens, None] traceback.print_exc()
return [data, totalTokens, e]
return [data, totalTokens, None]
def save_progress_lines(lines, filename, encoding="shift_jis"):
"""Atomically save current line-based translation progress."""
try: def save_progress_lines(lines, filename, encoding="shift_jis"):
if ESTIMATE: """Atomically save current line-based translation progress."""
return try:
os.makedirs("translated", exist_ok=True) if ESTIMATE:
tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated") return
try: os.makedirs("translated", exist_ok=True)
with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file: tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
tmp_file.writelines(lines) try:
os.replace(tmp_path, os.path.join("translated", filename)) with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file:
finally: tmp_file.writelines(lines)
if os.path.exists(tmp_path): os.replace(tmp_path, os.path.join("translated", filename))
try: finally:
os.remove(tmp_path) if os.path.exists(tmp_path):
except OSError: try:
pass os.remove(tmp_path)
except Exception: except OSError:
traceback.print_exc() pass
except Exception:
traceback.print_exc()
def saveCheckLines(lines, filename, tokens=None, encoding="shift_jis"):
try:
if tokens is not None: def saveCheckLines(lines, filename, tokens=None, encoding="shift_jis"):
if not (isinstance(tokens, (list, tuple)) and len(tokens) >= 2 and (tokens[0] or tokens[1])): try:
return if tokens is not None:
save_progress_lines(lines, filename, encoding=encoding) if not (isinstance(tokens, (list, tuple)) and len(tokens) >= 2 and (tokens[0] or tokens[1])):
except Exception: return
traceback.print_exc() save_progress_lines(lines, filename, encoding=encoding)
except Exception:
traceback.print_exc()
def translateWOLF(data, translatedList, pbar, filename):
stringList = []
currentGroup = [] def translateWOLF(data, translatedList, pbar, filename):
tokens = [0, 0] stringList = []
speaker = "" currentGroup = []
global LOCK, ESTIMATE, PBAR tokens = [0, 0]
PBAR = pbar speaker = ""
i = 0 global LOCK, ESTIMATE, PBAR
PBAR = pbar
while i < len(data): i = 0
# Speaker
matchList = re.findall(r"^([^/]*)：", data[i]) while i < len(data):
if len(matchList) != 0: # Speaker
response = getSpeaker(matchList[0]) matchList = re.findall(r"^([^/]*)：", data[i])
speaker = response[0] if len(matchList) != 0:
tokens[0] += response[1][0] response = getSpeaker(matchList[0])
tokens[1] += response[1][1] speaker = response[0]
data[i] = data[i].replace(matchList[0], f"{speaker}") tokens[0] += response[1][0]
saveCheckLines(data, filename) tokens[1] += response[1][1]
i += 1 data[i] = data[i].replace(matchList[0], f"{speaker}")
else: saveCheckLines(data, filename)
speaker = "" i += 1
else:
# Options speaker = ""
if "//選択肢" in data[i]:
i += 1 # Options
choiceList = [] if "//選択肢" in data[i]:
initialIndex = i i += 1
while "//" in data[i] and "の場合" not in data[i]: choiceList = []
choiceList.append(re.search(r"\/\/(.*)", data[i]).group(1)) initialIndex = i
i += 1 while "//" in data[i] and "の場合" not in data[i]:
choiceList.append(re.search(r"\/\/(.*)", data[i]).group(1))
# Translate i += 1
response = translateAI(choiceList, "This will be a dialogue option")
tokens[0] += response[1][0] # Translate
tokens[1] += response[1][1] response = translateAI(choiceList, "This will be a dialogue option")
choiceListTL = response[0] tokens[0] += response[1][0]
tokens[1] += response[1][1]
# Set Data choiceListTL = response[0]
if len(choiceList) == len(choiceListTL):
# Set Data # Set Data
i = initialIndex if len(choiceList) == len(choiceListTL):
while "//" in data[i] and "の場合" not in data[i]: # Set Data
choiceListTL[0] = choiceListTL[0].replace(", ", "、") i = initialIndex
data[i] = f"//{choiceListTL[0]}\n" while "//" in data[i] and "の場合" not in data[i]:
choiceListTL.pop(0) choiceListTL[0] = choiceListTL[0].replace(", ", "、")
i += 1 data[i] = f"//{choiceListTL[0]}\n"
saveCheckLines(data, filename) choiceListTL.pop(0)
i += 1
# Mismatch saveCheckLines(data, filename)
else:
with LOCK: # Mismatch
if filename not in MISMATCH: else:
MISMATCH.append(filename) with LOCK:
if filename not in MISMATCH:
# Lines MISMATCH.append(filename)
if r"/" not in data[i] and "@" not in data[i] and data[i] != "\n":
# Pass 1 # Lines
if translatedList == []: if r"/" not in data[i] and "@" not in data[i] and data[i] != "\n":
# Grab Consecutive Strings # Pass 1
currentGroup.append(data[i]) if translatedList == []:
i += 1 # Grab Consecutive Strings
while i < len(data) and r"/" not in data[i] and "@" not in data[i] and data[i] != "\n": currentGroup.append(data[i])
currentGroup.append(data[i]) i += 1
i += 1 while i < len(data) and r"/" not in data[i] and "@" not in data[i] and data[i] != "\n":
currentGroup.append(data[i])
# Join up 401 groups for better translation. i += 1
if len(currentGroup) > 0:
jaString = "".join(currentGroup) # Join up 401 groups for better translation.
currentGroup = [] if len(currentGroup) > 0:
jaString = "".join(currentGroup)
# Remove any textwrap currentGroup = []
jaString = jaString.replace("\n", " ")
# Remove any textwrap
# Add Speaker (If there is one) jaString = jaString.replace("\n", " ")
if speaker != "":
jaString = f"[{speaker}]: {jaString}" # Add Speaker (If there is one)
if speaker != "":
# Add String jaString = f"[{speaker}]: {jaString}"
stringList.append(jaString)
i += 1 # Add String
stringList.append(jaString)
# Pass 2 i += 1
else:
# Insert Strings # Pass 2
while i < len(data) and r"/" not in data[i] and "@" not in data[i] and data[i] != "\n": else:
data.pop(i) # Insert Strings
while i < len(data) and r"/" not in data[i] and "@" not in data[i] and data[i] != "\n":
# Get Text data.pop(i)
translatedText = translatedList[0]
translatedList.pop(0) # Get Text
translatedText = translatedList[0]
if len(translatedList) <= 0: translatedList.pop(0)
translatedList = None
if len(translatedList) <= 0:
# Remove speaker translatedList = None
matchSpeakerList = re.findall(r"^(\[.+?\]\s?[|:]\s?)\s?", translatedText)
if len(matchSpeakerList) > 0: # Remove speaker
translatedText = translatedText.replace(matchSpeakerList[0], "") matchSpeakerList = re.findall(r"^(\[.+?\]\s?[|:]\s?)\s?", translatedText)
if len(matchSpeakerList) > 0:
# Textwrap translatedText = translatedText.replace(matchSpeakerList[0], "")
translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
# Textwrap
# Set Data translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
data.insert(i, f"{translatedText}\n")
saveCheckLines(data, filename) # Set Data
i += 1 data.insert(i, f"{translatedText}\n")
saveCheckLines(data, filename)
# Nothing relevant. Skip Line. i += 1
else:
i += 1 # Nothing relevant. Skip Line.
else:
# EOF i += 1
if len(stringList) > 0:
# Set Progress # EOF
pbar.total = len(stringList) if len(stringList) > 0:
pbar.refresh() # Set Progress
pbar.total = len(stringList)
# Translate pbar.refresh()
response = translateAI(stringList, "")
tokens[0] += response[1][0] # Translate
tokens[1] += response[1][1] response = translateAI(stringList, "")
translatedList = response[0] tokens[0] += response[1][0]
tokens[1] += response[1][1]
# Set Strings translatedList = response[0]
if len(stringList) == len(translatedList):
translateWOLF(data, translatedList, pbar, filename) # Set Strings
if len(stringList) == len(translatedList):
# Mismatch translateWOLF(data, translatedList, pbar, filename)
else:
with LOCK: # Mismatch
if filename not in MISMATCH: else:
MISMATCH.append(filename) with LOCK:
return tokens if filename not in MISMATCH:
MISMATCH.append(filename)
# Save some money and enter the character before translation return tokens
def getSpeaker(speaker):
match speaker: # Save some money and enter the character before translation
case "ファイン": def getSpeaker(speaker):
return ["Fine", [0, 0]] match speaker:
case "": case "ファイン":
return ["", [0, 0]] return ["Fine", [0, 0]]
case _: case "":
# Find Speaker return ["", [0, 0]]
for i in range(len(NAMESLIST)): case _:
if speaker == NAMESLIST[i][0]: # Find Speaker
return [NAMESLIST[i][1], [0, 0]] for i in range(len(NAMESLIST)):
if speaker == NAMESLIST[i][0]:
# Translate and Store Speaker return [NAMESLIST[i][1], [0, 0]]
response = translateAI(
f"{speaker}", # Translate and Store Speaker
"Reply with the " + LANGUAGE + " translation of the NPC name.", response = translateAI(
False, f"{speaker}",
) "Reply with the " + LANGUAGE + " translation of the NPC name.",
response[0] = response[0].title() False,
response[0] = response[0].replace("'S", "'s") )
response[0] = response[0].replace("Speaker: ", "") response[0] = response[0].title()
response[0] = response[0].replace("'S", "'s")
# Retry if name doesn't translate for some reason response[0] = response[0].replace("Speaker: ", "")
if re.search(r"([a-zA-Z？?])", response[0]) == None:
response = translateAI( # Retry if name doesn't translate for some reason
f"{speaker}", if re.search(r"([a-zA-Z？?])", response[0]) == None:
"Reply with the " + LANGUAGE + " translation of the NPC name.", response = translateAI(
False, f"{speaker}",
) "Reply with the " + LANGUAGE + " translation of the NPC name.",
response[0] = response[0].title() False,
response[0] = response[0].replace("'S", "'s") )
response[0] = response[0].title()
speakerList = [speaker, response[0]] response[0] = response[0].replace("'S", "'s")
NAMESLIST.append(speakerList)
return response speakerList = [speaker, response[0]]
return [speaker, [0, 0]] NAMESLIST.append(speakerList)
return response
def translateAI(text, history, history_ctx=None): return [speaker, [0, 0]]
"""
Legacy wrapper function for the new shared translation utility. def translateAI(text, history, history_ctx=None):
This maintains compatibility with existing code while using the new shared implementation. """
""" Legacy wrapper function for the new shared translation utility.
global PBAR, MISMATCH, FILENAME This maintains compatibility with existing code while using the new shared implementation.
"""
# Update config estimate mode based on global ESTIMATE global PBAR, MISMATCH, FILENAME
TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
# Update config estimate mode based on global ESTIMATE
# Call the new shared translation function TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
return sharedtranslateAI(
text=text, # Call the new shared translation function
history=history, return sharedtranslateAI(
config=TRANSLATION_CONFIG, text=text,
filename=FILENAME, history=history,
pbar=PBAR, config=TRANSLATION_CONFIG,
lock=LOCK, filename=FILENAME,
mismatchList=MISMATCH pbar=PBAR,
) lock=LOCK,
mismatchList=MISMATCH
)

6
modules/yuris.py
View file

@ -21,8 +21,10 @@ from util.translation import (
MODEL = os.getenv("model") MODEL = os.getenv("model")
LANGUAGE = os.getenv("language").capitalize() LANGUAGE = os.getenv("language").capitalize()
PROMPT = Path("prompt.txt").read_text(encoding="utf-8") from util.paths import PROMPT_PATH, VOCAB_PATH
VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
LOCK = threading.Lock() LOCK = threading.Lock()
WIDTH = int(os.getenv("width")) WIDTH = int(os.getenv("width"))
MAXHISTORY = 10 MAXHISTORY = 10

13
scripts/launch.sh Normal file
View file

@ -0,0 +1,13 @@
#!/usr/bin/env bash
# Launcher for DazedMTLTool.desktop (finds venv, then starts the GUI).
ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$ROOT" || exit 1
PYTHON="python3"
if [[ -x "$ROOT/.venv/bin/python" ]]; then
PYTHON="$ROOT/.venv/bin/python"
elif [[ -x "$ROOT/venv/bin/python" ]]; then
PYTHON="$ROOT/venv/bin/python"
fi
exec "$PYTHON" "$ROOT/scripts/start_gui.py"

146
start_gui.py → scripts/start_gui.py
View file

@ -1,72 +1,74 @@
#!/usr/bin/env python3 #!/usr/bin/env python3
""" """Launch script for DazedMTLTool GUI."""
Launch script for DazedMTLTool GUI
""" import sys
from pathlib import Path
import sys
import os PROJECT_ROOT = Path(__file__).resolve().parent.parent
from pathlib import Path sys.path.insert(0, str(PROJECT_ROOT))
# Add the project root to Python path
project_root = Path(__file__).parent def check_dependencies():
sys.path.insert(0, str(project_root)) """Check if required dependencies are installed."""
missing_deps = []
def check_dependencies():
"""Check if required dependencies are installed.""" try:
missing_deps = [] import PyQt5 # noqa: F401
except ImportError:
try: missing_deps.append("PyQt5")
import PyQt5
except ImportError: try:
missing_deps.append("PyQt5") from dotenv import load_dotenv # noqa: F401
except ImportError:
try: missing_deps.append("python-dotenv")
from dotenv import load_dotenv
except ImportError: if missing_deps:
missing_deps.append("python-dotenv") print("Missing dependencies:")
for dep in missing_deps:
if missing_deps: print(f" - {dep}")
print("Missing dependencies:") print("\nPlease install them using:")
for dep in missing_deps: print(" pip install -r requirements.txt")
print(f" - {dep}") return False
print("\nPlease install them using:")
print(" pip install -r requirements_gui.txt") return True
return False
return True def main():
"""Main entry point."""
def main(): print("DazedMTLTool GUI Launcher")
"""Main entry point.""" print("=" * 40)
print("DazedMTLTool GUI Launcher")
print("=" * 40) from util.paths import ensure_vocab_file, migrate_root_data_files
# Check dependencies migrate_root_data_files()
if not check_dependencies(): ensure_vocab_file()
sys.exit(1)
if not check_dependencies():
try: sys.exit(1)
from util.ace.update_tools import seed_ace_tools
seed_ace_tools() try:
except Exception as exc: from util.ace.update_tools import seed_ace_tools
print(f"Warning: Ace tool setup failed ({exc}). Ace features may be unavailable.") seed_ace_tools()
except Exception as exc:
try: print(f"Warning: Ace tool setup failed ({exc}). Ace features may be unavailable.")
from util.forge.update_tools import seed_forge_plugins
seed_forge_plugins() try:
except Exception as exc: from util.forge.update_tools import seed_forge_plugins
print(f"Warning: Forge plugin setup failed ({exc}). Playtest Forge may be unavailable.") seed_forge_plugins()
except Exception as exc:
# Import and run GUI print(f"Warning: Forge plugin setup failed ({exc}). Playtest Forge may be unavailable.")
try:
from gui.main import main as gui_main try:
gui_main() from gui.main import main as gui_main
except ImportError as e: gui_main()
print(f"Error importing GUI modules: {e}") except ImportError as e:
print("Make sure all GUI files are in the 'gui' directory") print(f"Error importing GUI modules: {e}")
sys.exit(1) print("Make sure all GUI files are in the 'gui' directory")
except Exception as e: sys.exit(1)
print(f"Error starting GUI: {e}") except Exception as e:
sys.exit(1) print(f"Error starting GUI: {e}")
sys.exit(1)
if __name__ == "__main__":
main()
if __name__ == "__main__":
main()

6
set_defaults.py → util/defaults.py
View file

@ -1,7 +1,9 @@
import re import re
from copy import deepcopy from copy import deepcopy
# Canonical defaults used by the GUI and the `set_defaults` script. from util.paths import PROJECT_ROOT
# Canonical defaults used by the GUI and the pre-commit hook.
# Stored as booleans for easier consumption by the GUI. # Stored as booleans for easier consumption by the GUI.
DEFAULTS = { DEFAULTS = {
'FIRSTLINESPEAKERS': False, 'FIRSTLINESPEAKERS': False,
@ -56,4 +58,4 @@ def set_defaults(file_path):
if __name__ == "__main__": if __name__ == "__main__":
set_defaults('modules/rpgmakermvmz.py') set_defaults(PROJECT_ROOT / "modules/rpgmakermvmz.py")

72
util/linux_desktop.py Normal file
View file

@ -0,0 +1,72 @@
"""Install/update the Linux .desktop entry for taskbar and window icons."""
from __future__ import annotations
import os
import subprocess
import sys
from pathlib import Path
from util.paths import ICON_PATH, PROJECT_ROOT
DESKTOP_ID = "DazedMTLTool"
LAUNCH_SCRIPT = PROJECT_ROOT / "scripts" / "launch.sh"
def installed_desktop_path() -> Path:
data_home = os.environ.get("XDG_DATA_HOME", "")
if data_home:
base = Path(data_home)
else:
base = Path.home() / ".local" / "share"
return base / "applications" / f"{DESKTOP_ID}.desktop"
def _desktop_content(root: Path, icon: Path, launch: Path) -> str:
return (
"[Desktop Entry]\n"
"Type=Application\n"
f"Name={DESKTOP_ID}\n"
"GenericName=Translation Tool\n"
"Comment=AI translation tool for visual novels and RPG games\n"
f"Exec={launch}\n"
f"Icon={icon}\n"
f"Path={root}\n"
"Terminal=false\n"
"Categories=Development;Utility;\n"
f"StartupWMClass={DESKTOP_ID}\n"
)
def ensure_linux_desktop_entry() -> None:
"""Write ~/.local/share/applications/DazedMTLTool.desktop with absolute paths."""
if not sys.platform.startswith("linux"):
return
if not LAUNCH_SCRIPT.is_file() or not ICON_PATH.is_file():
return
desktop_path = installed_desktop_path()
content = _desktop_content(PROJECT_ROOT, ICON_PATH, LAUNCH_SCRIPT)
try:
if desktop_path.is_file() and desktop_path.read_text(encoding="utf-8") == content:
return
desktop_path.parent.mkdir(parents=True, exist_ok=True)
desktop_path.write_text(content, encoding="utf-8")
desktop_path.chmod(0o755)
except OSError as exc:
print(f"Warning: could not install Linux desktop entry ({exc})")
return
_refresh_desktop_database(desktop_path.parent)
def _refresh_desktop_database(applications_dir: Path) -> None:
for cmd in (
["update-desktop-database", str(applications_dir)],
["kbuildsycoca5", "--noincremental"],
):
try:
subprocess.run(cmd, check=False, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
except OSError:
pass

45
util/paths.py Normal file
View file

@ -0,0 +1,45 @@
"""Canonical project paths (repo root, data files, config)."""
from __future__ import annotations
from pathlib import Path
PROJECT_ROOT = Path(__file__).resolve().parent.parent
DATA_DIR = PROJECT_ROOT / "data"
VOCAB_PATH = DATA_DIR / "vocab.txt"
VOCAB_BASE_PATH = DATA_DIR / "vocab_base.txt"
PROMPT_PATH = DATA_DIR / "prompt.txt"
LAST_UPDATE_SHA_PATH = DATA_DIR / "last_update_sha.txt"
ENV_PATH = PROJECT_ROOT / ".env"
ICON_PATH = PROJECT_ROOT / "assets" / "icon.png"
_ROOT_DATA_FILES = (
"vocab.txt",
"vocab_base.txt",
"prompt.txt",
"last_update_sha.txt",
)
def migrate_root_data_files() -> None:
"""Move legacy root-level data files into data/ on first run."""
DATA_DIR.mkdir(parents=True, exist_ok=True)
for name in _ROOT_DATA_FILES:
src = PROJECT_ROOT / name
dst = DATA_DIR / name
if src.is_file() and not dst.exists():
src.rename(dst)
migrate_root_data_files()
def ensure_vocab_file() -> None:
"""Create data/vocab.txt from vocab_base.txt when missing."""
migrate_root_data_files()
if VOCAB_PATH.is_file():
return
if VOCAB_BASE_PATH.is_file():
VOCAB_PATH.write_text(VOCAB_BASE_PATH.read_text(encoding="utf-8"), encoding="utf-8")
else:
VOCAB_PATH.write_text("", encoding="utf-8")

6119
util/translation.py
View file

File diff suppressed because it is too large Load diff