Surely cleaning up all this mess wont break anything. surely

.gitignore

@@ -1,32 +1,32 @@
-
-.env
-*.tmp
-*.json
-*.js
-*.yaml
-*.yml
-*.txt
-*.log
-logs/
-*.rpy
-*.csv
-*.ks
-*.cid
-*.png
-*.script
-__pycache__
-
-!assets/*.png
-!.pre-commit-config.yaml
-!prompt.txt
-!vocab_base.txt
-!requirements.txt
-!util/tl_inspector/TLInspector.js
-!util/forge/Forge_MZ.js
-!util/forge/Forge_MV.js
-!util/forge/upstream/Forge_MZ.js
-!util/forge/upstream/Forge_MV.js
-util/forge/.forge_version.json
-util/ace/*.exe
-util/ace/.tools_version.json
-!util/ace/offline/*.exe
+
+.env
+*.tmp
+*.json
+*.js
+*.yaml
+*.yml
+*.txt
+*.log
+logs/
+*.rpy
+*.csv
+*.ks
+*.cid
+*.png
+*.script
+__pycache__
+
+!assets/*.png
+!.pre-commit-config.yaml
+!data/prompt.txt
+!data/vocab_base.txt
+!requirements.txt
+!util/tl_inspector/TLInspector.js
+!util/forge/Forge_MZ.js
+!util/forge/Forge_MV.js
+!util/forge/upstream/Forge_MZ.js
+!util/forge/upstream/Forge_MV.js
+util/forge/.forge_version.json
+util/ace/*.exe
+util/ace/.tools_version.json
+!util/ace/offline/*.exe

.pre-commit-config.yaml

@@ -3,6 +3,6 @@
   'hooks':
   - 'id': 'set-defaults'
     'name': 'Set Defaults in rpgmakermvmz.py'
-    'entry': 'python set_defaults.py'
+    'entry': 'python -m util.defaults'
     'language': 'python'
     'files': '^modules/rpgmakermvmz.py$'

DazedMTLTool.desktop

@@ -0,0 +1,10 @@
+[Desktop Entry]
+Type=Application
+Name=DazedMTLTool
+GenericName=Translation Tool
+Comment=AI translation tool for visual novels and RPG games
+Exec=bash -c 'exec "$(cd "$(dirname "%k")" && pwd)/scripts/launch.sh"'
+Icon=%k/assets/icon.png
+Terminal=false
+Categories=Development;Utility;
+StartupWMClass=DazedMTLTool

README.md

@@ -124,12 +124,11 @@ This means Python wasn't added to your PATH. You have two options:
 
 ### 3. Launch the GUI
 
-**Double-click `START.bat`**. It will:
-- Create a virtual environment automatically.
-- Install all dependencies.
-- Launch the GUI.
+**Windows:** Double-click `START.bat`. It will create a virtual environment, install dependencies, and open the GUI.
 
-That's it! From now on, just double-click `START.bat` to open the tool.
+**Linux/macOS:** Run `./START.sh`, or double-click `DazedMTLTool.desktop` (choose **Allow Launching** when your file manager asks). From then on, either method works.
+
+That's it! Use the same launcher each time you want to open the tool.
 
 ---
 
@@ -160,10 +159,10 @@ Specialized tabs with extra options for those specific engines.
 
 ## Vocab & Prompt
 
-### vocab.txt
+### data/vocab.txt
 This file gives the AI context about your game — character names, genders, recurring terms, etc. The better your vocab file, the more consistent the translation.
 
-Open `vocab.txt` (or copy `vocab.txt.example` to `vocab.txt` if it doesn't exist) and add entries like:
+On first run, `data/vocab.txt` is created automatically from `data/vocab_base.txt` if it does not exist yet. Add entries like:
 
 ```plaintext
 # Game Characters
@@ -176,14 +175,14 @@ Format: Japanese name, English name in parentheses, then gender.
 
 > **Note:** A very large vocab file can increase API costs and potentially reduce quality. Focus on the most important characters and terms.
 
-### prompt.txt
-This is the system prompt sent to the AI. A default `prompt.txt` is included and works well for most games. You generally don't need to edit it unless you want to customize the translation style.
+### data/prompt.txt
+This is the system prompt sent to the AI. A default `data/prompt.txt` is included and works well for most games. You generally don't need to edit it unless you want to customize the translation style.
 
 ---
 
 ## Tips
 
-- **Check `log/translations.txt`** after a run to see what was translated. You can copy useful terms from it into `vocab.txt` for consistency in future runs.
+- **Check `log/translations.txt`** after a run to see what was translated. You can copy useful terms from it into `data/vocab.txt` for consistency in future runs.
 - **Start small** — Translate a few files first to make sure the output looks good before doing the whole game.
 - **Wordwrap** — If text overflows or looks awkward in-game, adjust the `width` setting in `.env` or the Config tab. `60` is a good default for most RPG Maker games.
 - **Version control** — Using [Git](https://git-scm.com/) with the game folder is highly recommended. It lets you track every change the translation makes, compare with original files, and roll back if needed.
@@ -300,8 +299,8 @@ Here's the recommended step-by-step process for translating an RPG Maker MV/MZ g
 
 | Step | Action |
 |------|--------|
-| **1** | **Parse speakers → vocab.txt** — Use the Parse Speakers feature to pull character names from the game files into `vocab.txt`. |
-| **2** | **Identify speaker genders** — Figure out which characters are male/female and update `vocab.txt` accordingly. This helps the AI use correct pronouns. |
+| **1** | **Parse speakers → vocab** — Use the Parse Speakers feature to pull character names from the game files into `data/vocab.txt`. |
+| **2** | **Identify speaker genders** — Figure out which characters are male/female and update `data/vocab.txt` accordingly. This helps the AI use correct pronouns. |
 | **3** | **Translate Actors.json, MapInfos.json** — These are small files with character and map names. Good to do first. |
 | **4** | **Translate Items, System, Weapons, etc.** — All the data files that aren't maps or events. Place them in `files/`, translate, then copy results back. |
 | **5** | **Find speaker names** — Enable CODE 101 (Speakers), check for bracketed names, or use the "First Line = Speaker" option to capture speaker names properly. |

START.bat

@@ -184,28 +184,29 @@ echo    Launching DazedMTLTool GUI...
 echo ==========================================
 echo.
 
-:: Ensure vocab.txt exists (create from example if available)
-if not exist "vocab.txt" (
-    if exist "vocab.txt.example" (
-        echo vocab.txt not found - creating from vocab.txt.example...
-        copy /Y "vocab.txt.example" "vocab.txt" >nul 2>&1
+:: Ensure data/vocab.txt exists (create from vocab_base.txt if available)
+if not exist "data\\vocab.txt" (
+    if not exist "data" mkdir data
+    if exist "data\\vocab_base.txt" (
+        echo data\vocab.txt not found - creating from data\vocab_base.txt...
+        copy /Y "data\vocab_base.txt" "data\vocab.txt" >nul 2>&1
         if errorlevel 1 (
-            echo ERROR: Failed to copy vocab.txt.example to vocab.txt.
+            echo ERROR: Failed to copy data\vocab_base.txt to data\vocab.txt.
         ) else (
-            echo Created vocab.txt from vocab.txt.example
+            echo Created data\vocab.txt from data\vocab_base.txt
         )
     ) else (
-        echo vocab.txt and vocab.txt.example not found - creating empty vocab.txt to avoid import errors...
-        type NUL > "vocab.txt"
+        echo data\vocab.txt not found - creating empty file to avoid import errors...
+        type NUL > "data\vocab.txt"
         if errorlevel 1 (
-            echo ERROR: Failed to create empty vocab.txt.
+            echo ERROR: Failed to create empty data\vocab.txt.
         ) else (
-            echo Created empty vocab.txt
+            echo Created empty data\vocab.txt
         )
     )
 )
 
-python start_gui.py
+python scripts/start_gui.py
 
 :: Check if GUI launched successfully
 if errorlevel 1 (

START.sh

@@ -127,23 +127,24 @@ activate_venv() {
 }
 
 ensure_vocab_file() {
-    if [[ -f "vocab.txt" ]]; then
+    mkdir -p data
+    if [[ -f "data/vocab.txt" ]]; then
         return 0
     fi
 
-    if [[ -f "vocab.txt.example" ]]; then
-        echo "vocab.txt not found - creating from vocab.txt.example..."
-        if cp "vocab.txt.example" "vocab.txt"; then
-            echo "Created vocab.txt from vocab.txt.example"
+    if [[ -f "data/vocab_base.txt" ]]; then
+        echo "data/vocab.txt not found - creating from data/vocab_base.txt..."
+        if cp "data/vocab_base.txt" "data/vocab.txt"; then
+            echo "Created data/vocab.txt from data/vocab_base.txt"
         else
-            echo "ERROR: Failed to copy vocab.txt.example to vocab.txt."
+            echo "ERROR: Failed to copy data/vocab_base.txt to data/vocab.txt."
         fi
     else
-        echo "vocab.txt and vocab.txt.example not found - creating empty vocab.txt to avoid import errors..."
-        if : > "vocab.txt"; then
-            echo "Created empty vocab.txt"
+        echo "data/vocab.txt not found - creating empty file to avoid import errors..."
+        if : > "data/vocab.txt"; then
+            echo "Created empty data/vocab.txt"
         else
-            echo "ERROR: Failed to create empty vocab.txt."
+            echo "ERROR: Failed to create empty data/vocab.txt."
         fi
     fi
 }
@@ -227,7 +228,7 @@ echo
 
 ensure_vocab_file
 
-if ! python start_gui.py; then
+if ! exec python "$SCRIPT_DIR/scripts/start_gui.py"; then
     echo
     echo "ERROR: Failed to launch GUI."
     echo "Check the error messages above."

gui/main.py

@@ -19,20 +19,16 @@ from PyQt5.QtWidgets import (
     QTextEdit, QSplitter, QGroupBox, QStatusBar, QStackedWidget, QToolButton,
     QDialog
 )
-from PyQt5.QtCore import Qt, QThread, pyqtSignal, QTimer, QSettings
-from PyQt5.QtGui import QIcon, QFont, QPixmap, QScreen
+from PyQt5.QtCore import Qt, QThread, pyqtSignal, QTimer, QSettings, QCoreApplication
+from PyQt5.QtGui import QIcon, QFont, QPixmap, QScreen, QGuiApplication
 
 
-def _project_root() -> Path:
-    """Directory containing start_gui.py / requirements.txt (parent of gui/)."""
-    return Path(__file__).resolve().parent.parent
+from util.paths import PROJECT_ROOT, ICON_PATH, LAST_UPDATE_SHA_PATH
 
 
 def load_application_icon() -> QIcon:
     """Prefer ICO on Windows; PNG is fine for all platforms. Empty if missing."""
-    root = _project_root()
-    for rel in ("assets/icon.ico", "assets/icon.png"):
-        path = root / rel
+    for path in (PROJECT_ROOT / "assets/icon.ico", ICON_PATH):
         if path.is_file():
             icon = QIcon(str(path))
             if not icon.isNull():
@@ -126,14 +122,13 @@ class UpdateThread(QThread):
     REPO_USER   = "DazedAnon"
     REPO_NAME   = "DazedMTLTool"
     REPO_BRANCH = "main"
-    SHA_FILE    = "last_update_sha.txt"
+    SHA_FILE    = str(LAST_UPDATE_SHA_PATH)
 
     # Paths (relative, top-level) that should never be touched during update
-    PROTECTED = {".env", "venv", "log", "files", "translated",
-                 "vocab.txt", "last_update_sha.txt"}
+    PROTECTED = {".env", "venv", "log", "files", "translated", "data"}
 
     # GitLab zip archives do not preserve Unix execute bits; restore after apply.
-    EXECUTABLE_SUFFIXES = {".sh"}
+    EXECUTABLE_SUFFIXES = {".sh", ".desktop"}
 
     progress  = pyqtSignal(str)           # status message
     finished  = pyqtSignal(bool, str)     # (success, message)
@@ -902,6 +897,14 @@ def main():
         # Set high DPI scale factor policy
         if hasattr(QApplication, 'setHighDpiScaleFactorRoundingPolicy'):
             QApplication.setHighDpiScaleFactorRoundingPolicy(Qt.HighDpiScaleFactorRoundingPolicy.PassThrough)
+
+        if sys.platform.startswith("linux"):
+            from util.linux_desktop import ensure_linux_desktop_entry
+            ensure_linux_desktop_entry()
+            QGuiApplication.setDesktopFileName("DazedMTLTool")
+
+        QCoreApplication.setOrganizationName("DazedTranslations")
+        QCoreApplication.setApplicationName("DazedMTLTool")
         
         app = QApplication(sys.argv)
 
@@ -934,9 +937,7 @@ def main():
         return 1
     
     # Set application properties
-    app.setApplicationName("DazedMTLTool")
     app.setApplicationVersion("1.0")
-    app.setOrganizationName("DazedTranslations")
     
     # Apply dark theme with cleaner, more compact styling
     app.setStyleSheet("""

gui/rpgmaker_tab.py

@@ -19,8 +19,7 @@ except ImportError:
     ConfigIntegration = None
 try:
     # Prefer importing the project's canonical defaults if available
-    import set_defaults
-    CANONICAL_DEFAULTS = getattr(set_defaults, 'DEFAULTS', None)
+    from util.defaults import DEFAULTS as CANONICAL_DEFAULTS
 except Exception:
     CANONICAL_DEFAULTS = None
 

gui/srpg_tab.py

@@ -12,8 +12,7 @@ import re
 from gui.platform_glyph import platform_glyph
 
 try:
-    import set_defaults
-    CANONICAL_DEFAULTS = getattr(set_defaults, 'DEFAULTS', None)
+    from util.defaults import DEFAULTS as CANONICAL_DEFAULTS
 except Exception:
     CANONICAL_DEFAULTS = None
 

gui/wolf_tab.py

@@ -11,8 +11,7 @@ from PyQt5.QtCore import pyqtSignal
 import re
 from gui.platform_glyph import platform_glyph
 try:
-    import set_defaults
-    CANONICAL_DEFAULTS = getattr(set_defaults, 'DEFAULTS', None)
+    from util.defaults import DEFAULTS as CANONICAL_DEFAULTS
 except Exception:
     CANONICAL_DEFAULTS = None
 

gui/workflow_tab.py

@@ -22,6 +22,8 @@ import sys
 import threading
 from pathlib import Path
 
+from util.paths import VOCAB_BASE_PATH, VOCAB_PATH
+
 import jsbeautifier
 
 from PyQt5.QtCore import Qt, QEvent, QSettings, QThread, QTimer, pyqtSignal
@@ -3576,7 +3578,7 @@ class WorkflowTab(QWidget):
 
     def _read_vocab_speakers(self) -> list[tuple[str, str]]:
         """Parse the '# Speakers' section from vocab.txt and return (orig, tl) pairs."""
-        vocab_path = Path("vocab.txt")
+        vocab_path = VOCAB_PATH
         if not vocab_path.exists():
             return []
         try:
@@ -3606,7 +3608,7 @@ class WorkflowTab(QWidget):
         return results
 
     def _reload_vocab(self):
-        vocab_path = Path("vocab.txt")
+        vocab_path = VOCAB_PATH
         try:
             if vocab_path.exists():
                 text = vocab_path.read_text(encoding="utf-8")
@@ -3623,10 +3625,10 @@ class WorkflowTab(QWidget):
     def _save_vocab(self):
         try:
             game_text = self.vocab_editor.toPlainText().rstrip("\n")
-            base_path = Path("vocab_base.txt")
+            base_path = VOCAB_BASE_PATH
             base_text = base_path.read_text(encoding="utf-8") if base_path.exists() else ""
             combined = game_text + "\n\n" + self._BASE_SEPARATOR + base_text
-            Path("vocab.txt").write_text(combined, encoding="utf-8")
+            VOCAB_PATH.write_text(combined, encoding="utf-8")
             self._log("✅ vocab.txt saved (base terms from vocab_base.txt appended).")
         except Exception as exc:
             self._log(f"❌ Could not save vocab.txt: {exc}")
@@ -3795,7 +3797,7 @@ class WorkflowTab(QWidget):
             self._log("⚠  No game folder set. Complete Step 0 first.")
             return
 
-        src = Path("vocab.txt")
+        src = VOCAB_PATH
         if not src.exists():
             self._log("⚠  vocab.txt not found — save it in Step 3 first.")
             return

modules/aquedi4.py

@@ -1,215 +1,217 @@
-# modules/aquedi4.py
-
-# Libraries
-import json
-import os
-import re
-import util.dazedwrap as dazedwrap
-import threading
-import time
-import traceback
-import tiktoken
-from pathlib import Path
-from colorama import Fore
-from dotenv import load_dotenv
-from retry import retry
-from tqdm import tqdm
-from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
-# from modules.json import translateSimpleKeyValueJSON
-import tempfile
-
-# Boilerplate Globals
-MODEL = os.getenv("model")
-TIMEOUT = int(os.getenv("timeout"))
-LANGUAGE = os.getenv("language").capitalize()
-PROMPT = Path("prompt.txt").read_text(encoding="utf-8")
-VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
-LOCK = threading.Lock()
-ESTIMATE = ""
-TOKENS = [0, 0]
-MISMATCH = []
-FILENAME = None
-PBAR = None
-BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
-
-# Flag to control line break replacement
-REPLACE_LINEBREAKS_WITH_PIPES = True
-
-LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
-TRANSLATION_CONFIG = TranslationConfig(
-    model=MODEL,
-    language=LANGUAGE,
-    prompt=PROMPT,
-    vocab=VOCAB,
-    langRegex=LANGREGEX,
-    batchSize=int(os.getenv("batchsize", 10)),
-    maxHistory=10,
-    estimateMode=False
-)
-# End Boilerplate
-
-def handleAquedi4(filename, estimate):
-    """Main handler for Aquedi4 JSON files."""
-    global ESTIMATE, TOKENS, FILENAME
-    ESTIMATE = estimate
-    FILENAME = filename
-
-    try:
-        start = time.time()
-        translatedData = openFiles(filename)
-
-        if not estimate:
-            # Write final result after translation is complete
-            with open("translated/" + filename, "w", encoding="utf-8", newline="\n") as outFile:
-                json.dump(translatedData[0], outFile, ensure_ascii=False, indent=4)
-        
-        end = time.time()
-        tqdm.write(getResultString(translatedData, end - start, filename))
-        with LOCK:
-            TOKENS[0] += translatedData[1][0]
-            TOKENS[1] += translatedData[1][1]
-
-    except Exception as e:
-        traceback.print_exc()
-        return "Fail"
-
-    return getResultString(["", TOKENS, None], end - start, "TOTAL")
-
-
-def openFiles(filename):
-    """Opens and parses the JSON file."""
-    with open("files/" + filename, "r", encoding="UTF-8-sig") as f:
-        data = json.load(f)
-        translatedData = parseAquedi4JSON(data, filename)
-    return translatedData
-
-
-def parseAquedi4JSON(data, filename):
-    """Parses and translates the simple key-value JSON."""
-    totalTokens = [0, 0]
-    totalLines = 0
-    totalLines = len(data)
-    global LOCK, PBAR
-
-    stringList = []
-    keyList = []
-    if isinstance(data, dict) and all(isinstance(v, str) for v in data.values()):
-        for key, value in data.items():
-            if re.search(LANGREGEX, key):
-                stringList.append(key)
-                keyList.append(key)
-    
-    with tqdm(total=len(stringList), bar_format=BAR_FORMAT, position=0, leave=False) as pbar:
-        pbar.desc = filename
-        PBAR = pbar
-
-        # from modules.json import translateSimpleKeyValueJSON
-        
-        try:
-            if stringList:
-                result = translateSimpleKeyValueJSON(data, filename, stringList, keyList)
-                totalTokens[0] += result[0]
-                totalTokens[1] += result[1]
-            else:
-                pbar.write(f"Warning: {filename} is not in the expected simple key-value format. Skipping.")
-        except Exception as e:
-            traceback.print_exc()
-            return [data, totalTokens, e]
-            
-    return [data, totalTokens, None]
-
-
-def translateSimpleKeyValueJSON(data, filename, stringList, keyList):
-    """
-    Translate simple key-value JSON format where keys contain Japanese text
-    and values are placeholder strings like "TODO".
-    
-    Format example:
-    {
-        "\\r\\nスタビライザー": "TODO",
-        "\\r\\nプラグインなし": "TODO"
-    }
-    """
-    global LOCK, ESTIMATE, FILENAME, PBAR, MISMATCH
-    tokens = [0, 0]
-
-    preparedStringList = []
-    if REPLACE_LINEBREAKS_WITH_PIPES:
-        for s in stringList:
-            preparedStringList.append(s.replace("\r\n", "|").replace("\n", "|"))
-    else:
-        preparedStringList = stringList
-    
-    # Translate all keys if any were found
-    if preparedStringList:
-        PBAR.total = len(preparedStringList)
-        PBAR.refresh()
-        
-        response = translateAI(preparedStringList, "Reply with the English Translation")
-        tokens[0] += response[1][0]
-        tokens[1] += response[1][1]
-        translatedList = response[0]
-        
-        # Check for mismatch
-        if len(preparedStringList) != len(translatedList):
-            with LOCK:
-                if FILENAME not in MISMATCH:
-                    MISMATCH.append(FILENAME)
-        
-        # Pass 2: Update the values with translations
-        for i, original_key in enumerate(keyList):
-            if i < len(translatedList):
-                translatedText = translatedList[i]
-
-                if REPLACE_LINEBREAKS_WITH_PIPES:
-                    translatedText = translatedText.replace("|", "\r\n")
-                # Set the translated text as the value
-                data[original_key] = translatedText
-                
-                # Save progress after each translation
-                save_progress_json(data, filename)
-    
-    return tokens
-
-def save_progress_json(data, filename):
-    """Save current JSON translation progress."""
-    try:
-        if ESTIMATE: return
-        os.makedirs("translated", exist_ok=True)
-        tmp_path = os.path.join("translated", f"{filename}.tmp")
-        final_path = os.path.join("translated", filename)
-        with open(tmp_path, "w", encoding="utf-8", newline="\n") as outFile:
-            json.dump(data, outFile, ensure_ascii=False, indent=4)
-        os.replace(tmp_path, final_path)
-    except Exception:
-        traceback.print_exc()
-
-
-def getResultString(translatedData, translationTime, filename):
-    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
-    totalTokenstring = (
-        Fore.YELLOW + f"[Input: {translatedData[1][0]}]"
-        f"[Output: {translatedData[1][1]}]"
-        f"[Cost: ${cost:,.4f}]"
-    )
-    timeString = Fore.BLUE + f"[{round(translationTime, 1)}s]"
-
-    if translatedData[2] is None:
-        return f"{filename}: {totalTokenstring}{timeString}" + Fore.GREEN + " \u2713 " + Fore.RESET
-    else:
-        errorString = str(translatedData[2]) + Fore.RED
-        return f"{filename}: {totalTokenstring}{timeString}" + Fore.RED + " \u2717 " + errorString + Fore.RESET
-
-def translateAI(text, history, history_ctx=None):
-    """Legacy wrapper for the shared translation utility."""
-    global PBAR, MISMATCH, FILENAME
-    TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
-    return sharedtranslateAI(
-        text=text,
-        history=history,
-        config=TRANSLATION_CONFIG,
-        filename=FILENAME,
-        pbar=PBAR,
-        lock=LOCK,
-        mismatchList=MISMATCH
+# modules/aquedi4.py
+
+# Libraries
+import json
+import os
+import re
+import util.dazedwrap as dazedwrap
+import threading
+import time
+import traceback
+import tiktoken
+from pathlib import Path
+from colorama import Fore
+from dotenv import load_dotenv
+from retry import retry
+from tqdm import tqdm
+from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
+# from modules.json import translateSimpleKeyValueJSON
+import tempfile
+
+# Boilerplate Globals
+MODEL = os.getenv("model")
+TIMEOUT = int(os.getenv("timeout"))
+LANGUAGE = os.getenv("language").capitalize()
+from util.paths import PROMPT_PATH, VOCAB_PATH
+
+PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
+VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
+LOCK = threading.Lock()
+ESTIMATE = ""
+TOKENS = [0, 0]
+MISMATCH = []
+FILENAME = None
+PBAR = None
+BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
+
+# Flag to control line break replacement
+REPLACE_LINEBREAKS_WITH_PIPES = True
+
+LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
+TRANSLATION_CONFIG = TranslationConfig(
+    model=MODEL,
+    language=LANGUAGE,
+    prompt=PROMPT,
+    vocab=VOCAB,
+    langRegex=LANGREGEX,
+    batchSize=int(os.getenv("batchsize", 10)),
+    maxHistory=10,
+    estimateMode=False
+)
+# End Boilerplate
+
+def handleAquedi4(filename, estimate):
+    """Main handler for Aquedi4 JSON files."""
+    global ESTIMATE, TOKENS, FILENAME
+    ESTIMATE = estimate
+    FILENAME = filename
+
+    try:
+        start = time.time()
+        translatedData = openFiles(filename)
+
+        if not estimate:
+            # Write final result after translation is complete
+            with open("translated/" + filename, "w", encoding="utf-8", newline="\n") as outFile:
+                json.dump(translatedData[0], outFile, ensure_ascii=False, indent=4)
+        
+        end = time.time()
+        tqdm.write(getResultString(translatedData, end - start, filename))
+        with LOCK:
+            TOKENS[0] += translatedData[1][0]
+            TOKENS[1] += translatedData[1][1]
+
+    except Exception as e:
+        traceback.print_exc()
+        return "Fail"
+
+    return getResultString(["", TOKENS, None], end - start, "TOTAL")
+
+
+def openFiles(filename):
+    """Opens and parses the JSON file."""
+    with open("files/" + filename, "r", encoding="UTF-8-sig") as f:
+        data = json.load(f)
+        translatedData = parseAquedi4JSON(data, filename)
+    return translatedData
+
+
+def parseAquedi4JSON(data, filename):
+    """Parses and translates the simple key-value JSON."""
+    totalTokens = [0, 0]
+    totalLines = 0
+    totalLines = len(data)
+    global LOCK, PBAR
+
+    stringList = []
+    keyList = []
+    if isinstance(data, dict) and all(isinstance(v, str) for v in data.values()):
+        for key, value in data.items():
+            if re.search(LANGREGEX, key):
+                stringList.append(key)
+                keyList.append(key)
+    
+    with tqdm(total=len(stringList), bar_format=BAR_FORMAT, position=0, leave=False) as pbar:
+        pbar.desc = filename
+        PBAR = pbar
+
+        # from modules.json import translateSimpleKeyValueJSON
+        
+        try:
+            if stringList:
+                result = translateSimpleKeyValueJSON(data, filename, stringList, keyList)
+                totalTokens[0] += result[0]
+                totalTokens[1] += result[1]
+            else:
+                pbar.write(f"Warning: {filename} is not in the expected simple key-value format. Skipping.")
+        except Exception as e:
+            traceback.print_exc()
+            return [data, totalTokens, e]
+            
+    return [data, totalTokens, None]
+
+
+def translateSimpleKeyValueJSON(data, filename, stringList, keyList):
+    """
+    Translate simple key-value JSON format where keys contain Japanese text
+    and values are placeholder strings like "TODO".
+    
+    Format example:
+    {
+        "\\r\\nスタビライザー": "TODO",
+        "\\r\\nプラグインなし": "TODO"
+    }
+    """
+    global LOCK, ESTIMATE, FILENAME, PBAR, MISMATCH
+    tokens = [0, 0]
+
+    preparedStringList = []
+    if REPLACE_LINEBREAKS_WITH_PIPES:
+        for s in stringList:
+            preparedStringList.append(s.replace("\r\n", "|").replace("\n", "|"))
+    else:
+        preparedStringList = stringList
+    
+    # Translate all keys if any were found
+    if preparedStringList:
+        PBAR.total = len(preparedStringList)
+        PBAR.refresh()
+        
+        response = translateAI(preparedStringList, "Reply with the English Translation")
+        tokens[0] += response[1][0]
+        tokens[1] += response[1][1]
+        translatedList = response[0]
+        
+        # Check for mismatch
+        if len(preparedStringList) != len(translatedList):
+            with LOCK:
+                if FILENAME not in MISMATCH:
+                    MISMATCH.append(FILENAME)
+        
+        # Pass 2: Update the values with translations
+        for i, original_key in enumerate(keyList):
+            if i < len(translatedList):
+                translatedText = translatedList[i]
+
+                if REPLACE_LINEBREAKS_WITH_PIPES:
+                    translatedText = translatedText.replace("|", "\r\n")
+                # Set the translated text as the value
+                data[original_key] = translatedText
+                
+                # Save progress after each translation
+                save_progress_json(data, filename)
+    
+    return tokens
+
+def save_progress_json(data, filename):
+    """Save current JSON translation progress."""
+    try:
+        if ESTIMATE: return
+        os.makedirs("translated", exist_ok=True)
+        tmp_path = os.path.join("translated", f"{filename}.tmp")
+        final_path = os.path.join("translated", filename)
+        with open(tmp_path, "w", encoding="utf-8", newline="\n") as outFile:
+            json.dump(data, outFile, ensure_ascii=False, indent=4)
+        os.replace(tmp_path, final_path)
+    except Exception:
+        traceback.print_exc()
+
+
+def getResultString(translatedData, translationTime, filename):
+    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
+    totalTokenstring = (
+        Fore.YELLOW + f"[Input: {translatedData[1][0]}]"
+        f"[Output: {translatedData[1][1]}]"
+        f"[Cost: ${cost:,.4f}]"
+    )
+    timeString = Fore.BLUE + f"[{round(translationTime, 1)}s]"
+
+    if translatedData[2] is None:
+        return f"{filename}: {totalTokenstring}{timeString}" + Fore.GREEN + " \u2713 " + Fore.RESET
+    else:
+        errorString = str(translatedData[2]) + Fore.RED
+        return f"{filename}: {totalTokenstring}{timeString}" + Fore.RED + " \u2717 " + errorString + Fore.RESET
+
+def translateAI(text, history, history_ctx=None):
+    """Legacy wrapper for the shared translation utility."""
+    global PBAR, MISMATCH, FILENAME
+    TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
+    return sharedtranslateAI(
+        text=text,
+        history=history,
+        config=TRANSLATION_CONFIG,
+        filename=FILENAME,
+        pbar=PBAR,
+        lock=LOCK,
+        mismatchList=MISMATCH
     )

modules/csv.py

@@ -1,458 +1,460 @@
-# Libraries
-import json
-import os
-import re
-import util.dazedwrap as dazedwrap
-import threading
-import time
-import traceback
-import tiktoken
-import csv
-from pathlib import Path
-from colorama import Fore
-from dotenv import load_dotenv
-from retry import retry
-from tqdm import tqdm
-from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
-import tempfile
-
-# Globals
-MODEL = os.getenv("model")
-TIMEOUT = int(os.getenv("timeout"))
-LANGUAGE = os.getenv("language").capitalize()
-PROMPT = Path("prompt.txt").read_text(encoding="utf-8")
-VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
-LOCK = threading.Lock()
-WIDTH = int(os.getenv("width"))
-LISTWIDTH = int(os.getenv("listWidth"))
-NOTEWIDTH = int(os.getenv("noteWidth"))
-MAXHISTORY = 10
-ESTIMATE = ""
-TOKENS = [0, 0]
-NAMESLIST = []
-NAMES = False  # Output a list of all the character names found
-BRFLAG = False  # If the game uses <br> instead
-FIXTEXTWRAP = True  # Overwrites textwrap
-IGNORETLTEXT = True  # Ignores all translated text.
-MISMATCH = []  # Lists files that thdata a mismatch error (Length of GPT list response is wrong)
-FILENAME = None
-BRACKETNAMES = False
-
-# CSV Configuration Settings (configurable via GUI)
-CSV_DELIMITER = "	"  # CSV delimiter character (comma, semicolon, tab)
-SOURCE_COLUMN = 2  # Which column has the source text to translate (0-indexed)
-TARGET_COLUMN = 3  # Which column to write translations to
-SPEAKER_COLUMN = 1  # Which column has speaker names (-1 = none)
-SKIP_HEADER_ROW = False  # Skip the first row (header)
-USE_TARGET_IF_NOT_EMPTY = False  # Use target column text if not empty (T++ style)
-WRITE_TO_NEXT_COLUMN = False  # Write to column after target instead of overwriting
-PARSE_NAME_TAGS = False  # Parse :name[] tags in text
-PARSE_M_MARKERS = False  # Parse \M markers in text
-REMOVE_FURIGANA = True  # Remove furigana annotations ＜＝＞
-SKIP_COMMENT_ROWS = False  # Skip rows starting with 'comment'
-
-# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
-LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
-
-# Get pricing configuration based on the model
-PRICING_CONFIG = getPricingConfig(MODEL)
-INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
-OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
-BATCHSIZE = PRICING_CONFIG["batchSize"]
-FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
-
-# tqdm Globals
-BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
-POSITION = 0
-LEAVE = False
-PBAR = None
-ENCODING = "utf8"
-
-# Initialize Translation Config
-TRANSLATION_CONFIG = TranslationConfig(
-    model=MODEL,
-    language=LANGUAGE,
-    prompt=PROMPT,
-    vocab=VOCAB,
-    langRegex=LANGREGEX,
-    batchSize=BATCHSIZE,
-    maxHistory=MAXHISTORY,
-    estimateMode=False  # Will be set dynamically based on ESTIMATE
-)
-LEAVE = False
-
-def handleCSV(filename, estimate):
-    global ESTIMATE, TOKENS, FILENAME
-    ESTIMATE = estimate
-    FILENAME = filename
-
-    if not ESTIMATE:
-        with open("translated/" + filename, "w+t", newline="", encoding=ENCODING, errors="xmlcharrefreplace") as writeFile:
-            # Translate
-            start = time.time()
-            translatedData = openFiles(filename, writeFile)
-
-            # Print Result
-            end = time.time()
-            tqdm.write(getResultString(translatedData, end - start, filename))
-            with LOCK:
-                TOKENS[0] += translatedData[1][0]
-                TOKENS[1] += translatedData[1][1]
-    else:
-        # Translate
-        start = time.time()
-        translatedData = openFilesEstimate(filename)
-
-        # Print Result
-        end = time.time()
-        tqdm.write(getResultString(translatedData, end - start, filename))
-        with LOCK:
-            TOKENS[0] += translatedData[1][0]
-            TOKENS[1] += translatedData[1][1]
-
-    # Print Total
-    totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
-
-    # Print any errors on maps
-    if len(MISMATCH) > 0:
-        return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
-    else:
-        return totalString
-
-
-def openFiles(filename, writeFile):
-    with open("files/" + filename, "r", encoding=ENCODING) as readFile, writeFile:
-        translatedData = parseCSV(readFile, writeFile, filename)
-
-    return translatedData
-
-
-def openFilesEstimate(filename):
-    with open("files/" + filename, "r", encoding="utf8") as readFile:
-        translatedData = parseCSV(readFile, "", filename)
-
-    return translatedData
-
-
-def getResultString(translatedData, translationTime, filename):
-    # File Print String
-    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
-    totalTokenstring = (
-        Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
-        "[Output: "
-        + str(translatedData[1][1])
-        + "]" "[Cost: ${:,.4f}".format(cost)
-        + "]"
-    )
-    timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
-
-    if translatedData[2] is None:
-        # Success
-        return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
-    else:
-        # Fail
-        try:
-            raise translatedData[2]
-        except Exception as e:
-            traceback.print_exc()
-            errorString = str(e) + Fore.RED
-            return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
-
-
-def parseCSV(readFile, writeFile, filename):
-    totalTokens = [0, 0]
-    totalLines = 0
-    global LOCK
-
-    # Get total for progress bar
-    totalLines = len(readFile.readlines())
-    readFile.seek(0)
-
-    reader = csv.reader(readFile, delimiter=CSV_DELIMITER)
-    if not ESTIMATE:
-        writer = csv.writer(
-            writeFile,
-            delimiter=CSV_DELIMITER,
-        )
-    else:
-        writer = ""
-
-    with tqdm(bar_format=BAR_FORMAT, position=POSITION, total=totalLines, leave=LEAVE) as pbar:
-        pbar.desc = filename
-        pbar.total = totalLines
-
-        # Grab All Rows
-        data = []
-        for row in reader:
-            data.append(row)
-
-        try:
-            response = translateCSV(data, pbar, writeFile, writer, filename, None)
-            totalTokens[0] = response[0]
-            totalTokens[1] = response[1]
-        except Exception:
-            traceback.print_exc()
-    return [data, totalTokens, None]
-
-
-def flush_progress_csv(writeFile, writer, rows):
-    """Flush current CSV progress to the already-open output file (Windows-safe)."""
-    try:
-        if ESTIMATE or writeFile is None:
-            return
-        with LOCK:
-            writeFile.seek(0)
-            # Recreate writer at current position to avoid state issues
-            tmp_writer = csv.writer(writeFile, delimiter=CSV_DELIMITER)
-            tmp_writer.writerows(rows)
-            writeFile.truncate()
-            writeFile.flush()
-            os.fsync(writeFile.fileno())
-    except Exception:
-        traceback.print_exc()
-
-
-def translateCSV(data, pbar, writeFile, writer, filename, translatedList):
-    """
-    Unified CSV translation function using configurable settings.
-    
-    Uses global settings:
-    - SOURCE_COLUMN: column index for source text
-    - TARGET_COLUMN: column index to write translations
-    - SPEAKER_COLUMN: column index for speaker names (-1 = none)
-    - SKIP_HEADER_ROW: whether to skip first row
-    - USE_TARGET_IF_NOT_EMPTY: use existing target text if present (T++ style)
-    - WRITE_TO_NEXT_COLUMN: write to column after target
-    - PARSE_NAME_TAGS: parse :name[] tags
-    - PARSE_M_MARKERS: parse \\M markers
-    - REMOVE_FURIGANA: remove furigana
-    - SKIP_COMMENT_ROWS: skip rows with 'comment' in first column
-    """
-    global LOCK, ESTIMATE, PBAR
-    PBAR = pbar
-    translatedText = ""
-    totalTokens = [0, 0]
-    i = 0
-    stringList = []
-
-    try:
-        # Translate
-        while i < len(data):
-            # Skip header row if configured
-            if SKIP_HEADER_ROW and i == 0:
-                i += 1
-                continue
-            
-            # Skip comment rows if configured
-            if SKIP_COMMENT_ROWS and len(data[i]) > 0 and 'comment' in str(data[i][0]).lower():
-                i += 1
-                continue
-            
-            # Check if row has enough columns
-            if len(data[i]) <= SOURCE_COLUMN:
-                i += 1
-                continue
-            
-            # Get source text
-            jaString = ""
-            speaker = ""
-            
-            # If USE_TARGET_IF_NOT_EMPTY is enabled (T++ style), check target column first
-            if USE_TARGET_IF_NOT_EMPTY and len(data[i]) > TARGET_COLUMN and data[i][TARGET_COLUMN]:
-                jaString = data[i][TARGET_COLUMN]
-            else:
-                jaString = data[i][SOURCE_COLUMN] if data[i][SOURCE_COLUMN] else ""
-            
-            # Skip empty strings
-            if not jaString:
-                i += 1
-                continue
-            
-            # Handle speaker column if configured
-            if SPEAKER_COLUMN >= 0 and len(data[i]) > SPEAKER_COLUMN and data[i][SPEAKER_COLUMN]:
-                speakerResponse = getSpeaker(data[i][SPEAKER_COLUMN])
-                totalTokens[0] += speakerResponse[1][0]
-                totalTokens[1] += speakerResponse[1][1]
-                speaker = speakerResponse[0]
-                data[i][SPEAKER_COLUMN] = speaker
-            
-            # Parse :name[] tags if configured
-            if PARSE_NAME_TAGS and ':name' in jaString:
-                match = re.search(r":name\[([^\]]+?)\]\n([\w\W]*)", jaString)
-                if match:
-                    # Translate speaker name
-                    response = getSpeaker(match.group(1))
-                    speaker = response[0]
-                    totalTokens[0] += response[1][0]
-                    totalTokens[1] += response[1][1]
-                    data[i][SOURCE_COLUMN] = data[i][SOURCE_COLUMN].replace(match.group(1), speaker)
-                    
-                    # Extract text portion
-                    jaString = match.group(2)
-                    # Remove voice markers
-                    voMatch = re.search(r"\\[vfF]+\[.+]", jaString)
-                    if voMatch:
-                        jaString = jaString.replace(voMatch.group(0), "")
-            
-            # Parse \M markers if configured
-            if PARSE_M_MARKERS and '\\M' in jaString:
-                match = re.search(r"\\M.+\n([\w\W]*)", jaString)
-                if match:
-                    jaString = match.group(1)
-                    voMatch = re.search(r"\\[vfF]+\[.+]", jaString)
-                    if voMatch:
-                        jaString = jaString.replace(voMatch.group(0), "")
-            
-            # Remove furigana if configured
-            if REMOVE_FURIGANA:
-                jaString = re.sub(r"＜(.*)＝.*＞", r"\1", jaString)
-            
-            # Store original for replacement
-            ojaString = jaString
-            
-            # Remove textwrap
-            jaString = jaString.replace("\\n", " ")
-            jaString = jaString.replace("\n", " ")
-            
-            # Pass 1: Collect strings
-            if not translatedList:
-                if speaker:
-                    stringList.append(f"[{speaker}]: {jaString}")
-                else:
-                    stringList.append(jaString)
-            
-            # Pass 2: Apply translations
-            else:
-                # Grab and pop translation
-                translatedText = translatedList[0]
-                translatedList.pop(0)
-                
-                # Remove speaker prefix from translation if present
-                if speaker:
-                    translatedText = re.sub(r"^\[?(.+?)\]?\s?[|:]\s?", "", translatedText)
-                
-                # Add wordwrap
-                translatedText = dazedwrap.wrapText(translatedText, WIDTH)
-                translatedText = translatedText.replace("\n", "\\n")
-                
-                # Determine target column
-                actual_target = TARGET_COLUMN + 1 if WRITE_TO_NEXT_COLUMN else TARGET_COLUMN
-                
-                # Ensure row has enough columns
-                while len(data[i]) <= actual_target:
-                    data[i].append("")
-                
-                # Set data
-                if PARSE_NAME_TAGS or PARSE_M_MARKERS:
-                    # Replace original text portion
-                    data[i][actual_target] = data[i][actual_target].replace(ojaString, translatedText) if data[i][actual_target] else translatedText
-                else:
-                    data[i][actual_target] = translatedText
-                
-                flush_progress_csv(writeFile, writer, data)
-            
-            # Iterate
-            i += 1
-
-        # EOF - Process collected strings
-        if len(stringList) > 0:
-            # Set Progress
-            pbar.total = len(stringList)
-            pbar.refresh()
-
-            # Translate
-            response = translateAI(stringList, "")
-            totalTokens[0] += response[1][0]
-            totalTokens[1] += response[1][1]
-            translatedList = response[0]
-
-            # Set Strings
-            if len(stringList) == len(translatedList):
-                translateCSV(data, pbar, writeFile, writer, filename, translatedList)
-            # Mismatch
-            else:
-                with LOCK:
-                    if filename not in MISMATCH:
-                        MISMATCH.append(filename)
-
-            # Write all Data
-            with LOCK:
-                if not ESTIMATE:
-                    for row in data:
-                        writer.writerow(row)
-            flush_progress_csv(writeFile, writer, data)
-
-    except Exception:
-        traceback.print_exc()
-
-        # Write all Data
-        with LOCK:
-            if not ESTIMATE:
-                for row in data:
-                    writer.writerow(row)
-        flush_progress_csv(writeFile, writer, data)
-        return totalTokens
-
-    return totalTokens
-
-# Save some money and enter the character before translation
-def getSpeaker(speaker):
-    match speaker:
-        case "ファイン":
-            return ["Fine", [0, 0]]
-        case "":
-            return ["", [0, 0]]
-        case _:
-            # Find Speaker
-            for i in range(len(NAMESLIST)):
-                if speaker == NAMESLIST[i][0]:
-                    return [NAMESLIST[i][1], [0, 0]]
-                
-            if speaker == "？？？":
-                return ["???", [0, 0]]
-
-            # Translate and Store Speaker
-            response = translateAI(
-                f"{speaker}",
-                "Reply with the " + LANGUAGE + " translation of the NPC name.",
-                False,
-            )
-            response[0] = response[0].title()
-            response[0] = response[0].replace("'S", "'s")
-            response[0] = response[0].replace("Speaker: ", "")
-
-            # Retry if name doesn't translate for some reason
-            if re.search(r"([a-zA-Z？?])", response[0]) == None:
-                response = translateAI(
-                    f"{speaker}",
-                    "Reply with the " + LANGUAGE + " translation of the NPC name.",
-                    False,
-                )
-                response[0] = response[0].title()
-                response[0] = response[0].replace("'S", "'s")
-
-            speakerList = [speaker, response[0]]
-            NAMESLIST.append(speakerList)
-            return response
-    return [speaker, [0, 0]]
-
-def translateAI(text, history, history_ctx=None):
-    """
-    Legacy wrapper function for the new shared translation utility.
-    This maintains compatibility with existing code while using the new shared implementation.
-    """
-    global PBAR, MISMATCH, FILENAME
-    
-    # Update config estimate mode based on global ESTIMATE
-    TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
-    
-    # Call the new shared translation function
-    return sharedtranslateAI(
-        text=text,
-        history=history,
-        config=TRANSLATION_CONFIG,
-        filename=FILENAME,
-        pbar=PBAR,
-        lock=LOCK,
-        mismatchList=MISMATCH
-    )
+# Libraries
+import json
+import os
+import re
+import util.dazedwrap as dazedwrap
+import threading
+import time
+import traceback
+import tiktoken
+import csv
+from pathlib import Path
+from colorama import Fore
+from dotenv import load_dotenv
+from retry import retry
+from tqdm import tqdm
+from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
+import tempfile
+
+# Globals
+MODEL = os.getenv("model")
+TIMEOUT = int(os.getenv("timeout"))
+LANGUAGE = os.getenv("language").capitalize()
+from util.paths import PROMPT_PATH, VOCAB_PATH
+
+PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
+VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
+LOCK = threading.Lock()
+WIDTH = int(os.getenv("width"))
+LISTWIDTH = int(os.getenv("listWidth"))
+NOTEWIDTH = int(os.getenv("noteWidth"))
+MAXHISTORY = 10
+ESTIMATE = ""
+TOKENS = [0, 0]
+NAMESLIST = []
+NAMES = False  # Output a list of all the character names found
+BRFLAG = False  # If the game uses <br> instead
+FIXTEXTWRAP = True  # Overwrites textwrap
+IGNORETLTEXT = True  # Ignores all translated text.
+MISMATCH = []  # Lists files that thdata a mismatch error (Length of GPT list response is wrong)
+FILENAME = None
+BRACKETNAMES = False
+
+# CSV Configuration Settings (configurable via GUI)
+CSV_DELIMITER = "	"  # CSV delimiter character (comma, semicolon, tab)
+SOURCE_COLUMN = 2  # Which column has the source text to translate (0-indexed)
+TARGET_COLUMN = 3  # Which column to write translations to
+SPEAKER_COLUMN = 1  # Which column has speaker names (-1 = none)
+SKIP_HEADER_ROW = False  # Skip the first row (header)
+USE_TARGET_IF_NOT_EMPTY = False  # Use target column text if not empty (T++ style)
+WRITE_TO_NEXT_COLUMN = False  # Write to column after target instead of overwriting
+PARSE_NAME_TAGS = False  # Parse :name[] tags in text
+PARSE_M_MARKERS = False  # Parse \M markers in text
+REMOVE_FURIGANA = True  # Remove furigana annotations ＜＝＞
+SKIP_COMMENT_ROWS = False  # Skip rows starting with 'comment'
+
+# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
+LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
+
+# Get pricing configuration based on the model
+PRICING_CONFIG = getPricingConfig(MODEL)
+INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
+OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
+BATCHSIZE = PRICING_CONFIG["batchSize"]
+FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
+
+# tqdm Globals
+BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
+POSITION = 0
+LEAVE = False
+PBAR = None
+ENCODING = "utf8"
+
+# Initialize Translation Config
+TRANSLATION_CONFIG = TranslationConfig(
+    model=MODEL,
+    language=LANGUAGE,
+    prompt=PROMPT,
+    vocab=VOCAB,
+    langRegex=LANGREGEX,
+    batchSize=BATCHSIZE,
+    maxHistory=MAXHISTORY,
+    estimateMode=False  # Will be set dynamically based on ESTIMATE
+)
+LEAVE = False
+
+def handleCSV(filename, estimate):
+    global ESTIMATE, TOKENS, FILENAME
+    ESTIMATE = estimate
+    FILENAME = filename
+
+    if not ESTIMATE:
+        with open("translated/" + filename, "w+t", newline="", encoding=ENCODING, errors="xmlcharrefreplace") as writeFile:
+            # Translate
+            start = time.time()
+            translatedData = openFiles(filename, writeFile)
+
+            # Print Result
+            end = time.time()
+            tqdm.write(getResultString(translatedData, end - start, filename))
+            with LOCK:
+                TOKENS[0] += translatedData[1][0]
+                TOKENS[1] += translatedData[1][1]
+    else:
+        # Translate
+        start = time.time()
+        translatedData = openFilesEstimate(filename)
+
+        # Print Result
+        end = time.time()
+        tqdm.write(getResultString(translatedData, end - start, filename))
+        with LOCK:
+            TOKENS[0] += translatedData[1][0]
+            TOKENS[1] += translatedData[1][1]
+
+    # Print Total
+    totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
+
+    # Print any errors on maps
+    if len(MISMATCH) > 0:
+        return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
+    else:
+        return totalString
+
+
+def openFiles(filename, writeFile):
+    with open("files/" + filename, "r", encoding=ENCODING) as readFile, writeFile:
+        translatedData = parseCSV(readFile, writeFile, filename)
+
+    return translatedData
+
+
+def openFilesEstimate(filename):
+    with open("files/" + filename, "r", encoding="utf8") as readFile:
+        translatedData = parseCSV(readFile, "", filename)
+
+    return translatedData
+
+
+def getResultString(translatedData, translationTime, filename):
+    # File Print String
+    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
+    totalTokenstring = (
+        Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
+        "[Output: "
+        + str(translatedData[1][1])
+        + "]" "[Cost: ${:,.4f}".format(cost)
+        + "]"
+    )
+    timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
+
+    if translatedData[2] is None:
+        # Success
+        return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
+    else:
+        # Fail
+        try:
+            raise translatedData[2]
+        except Exception as e:
+            traceback.print_exc()
+            errorString = str(e) + Fore.RED
+            return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
+
+
+def parseCSV(readFile, writeFile, filename):
+    totalTokens = [0, 0]
+    totalLines = 0
+    global LOCK
+
+    # Get total for progress bar
+    totalLines = len(readFile.readlines())
+    readFile.seek(0)
+
+    reader = csv.reader(readFile, delimiter=CSV_DELIMITER)
+    if not ESTIMATE:
+        writer = csv.writer(
+            writeFile,
+            delimiter=CSV_DELIMITER,
+        )
+    else:
+        writer = ""
+
+    with tqdm(bar_format=BAR_FORMAT, position=POSITION, total=totalLines, leave=LEAVE) as pbar:
+        pbar.desc = filename
+        pbar.total = totalLines
+
+        # Grab All Rows
+        data = []
+        for row in reader:
+            data.append(row)
+
+        try:
+            response = translateCSV(data, pbar, writeFile, writer, filename, None)
+            totalTokens[0] = response[0]
+            totalTokens[1] = response[1]
+        except Exception:
+            traceback.print_exc()
+    return [data, totalTokens, None]
+
+
+def flush_progress_csv(writeFile, writer, rows):
+    """Flush current CSV progress to the already-open output file (Windows-safe)."""
+    try:
+        if ESTIMATE or writeFile is None:
+            return
+        with LOCK:
+            writeFile.seek(0)
+            # Recreate writer at current position to avoid state issues
+            tmp_writer = csv.writer(writeFile, delimiter=CSV_DELIMITER)
+            tmp_writer.writerows(rows)
+            writeFile.truncate()
+            writeFile.flush()
+            os.fsync(writeFile.fileno())
+    except Exception:
+        traceback.print_exc()
+
+
+def translateCSV(data, pbar, writeFile, writer, filename, translatedList):
+    """
+    Unified CSV translation function using configurable settings.
+    
+    Uses global settings:
+    - SOURCE_COLUMN: column index for source text
+    - TARGET_COLUMN: column index to write translations
+    - SPEAKER_COLUMN: column index for speaker names (-1 = none)
+    - SKIP_HEADER_ROW: whether to skip first row
+    - USE_TARGET_IF_NOT_EMPTY: use existing target text if present (T++ style)
+    - WRITE_TO_NEXT_COLUMN: write to column after target
+    - PARSE_NAME_TAGS: parse :name[] tags
+    - PARSE_M_MARKERS: parse \\M markers
+    - REMOVE_FURIGANA: remove furigana
+    - SKIP_COMMENT_ROWS: skip rows with 'comment' in first column
+    """
+    global LOCK, ESTIMATE, PBAR
+    PBAR = pbar
+    translatedText = ""
+    totalTokens = [0, 0]
+    i = 0
+    stringList = []
+
+    try:
+        # Translate
+        while i < len(data):
+            # Skip header row if configured
+            if SKIP_HEADER_ROW and i == 0:
+                i += 1
+                continue
+            
+            # Skip comment rows if configured
+            if SKIP_COMMENT_ROWS and len(data[i]) > 0 and 'comment' in str(data[i][0]).lower():
+                i += 1
+                continue
+            
+            # Check if row has enough columns
+            if len(data[i]) <= SOURCE_COLUMN:
+                i += 1
+                continue
+            
+            # Get source text
+            jaString = ""
+            speaker = ""
+            
+            # If USE_TARGET_IF_NOT_EMPTY is enabled (T++ style), check target column first
+            if USE_TARGET_IF_NOT_EMPTY and len(data[i]) > TARGET_COLUMN and data[i][TARGET_COLUMN]:
+                jaString = data[i][TARGET_COLUMN]
+            else:
+                jaString = data[i][SOURCE_COLUMN] if data[i][SOURCE_COLUMN] else ""
+            
+            # Skip empty strings
+            if not jaString:
+                i += 1
+                continue
+            
+            # Handle speaker column if configured
+            if SPEAKER_COLUMN >= 0 and len(data[i]) > SPEAKER_COLUMN and data[i][SPEAKER_COLUMN]:
+                speakerResponse = getSpeaker(data[i][SPEAKER_COLUMN])
+                totalTokens[0] += speakerResponse[1][0]
+                totalTokens[1] += speakerResponse[1][1]
+                speaker = speakerResponse[0]
+                data[i][SPEAKER_COLUMN] = speaker
+            
+            # Parse :name[] tags if configured
+            if PARSE_NAME_TAGS and ':name' in jaString:
+                match = re.search(r":name\[([^\]]+?)\]\n([\w\W]*)", jaString)
+                if match:
+                    # Translate speaker name
+                    response = getSpeaker(match.group(1))
+                    speaker = response[0]
+                    totalTokens[0] += response[1][0]
+                    totalTokens[1] += response[1][1]
+                    data[i][SOURCE_COLUMN] = data[i][SOURCE_COLUMN].replace(match.group(1), speaker)
+                    
+                    # Extract text portion
+                    jaString = match.group(2)
+                    # Remove voice markers
+                    voMatch = re.search(r"\\[vfF]+\[.+]", jaString)
+                    if voMatch:
+                        jaString = jaString.replace(voMatch.group(0), "")
+            
+            # Parse \M markers if configured
+            if PARSE_M_MARKERS and '\\M' in jaString:
+                match = re.search(r"\\M.+\n([\w\W]*)", jaString)
+                if match:
+                    jaString = match.group(1)
+                    voMatch = re.search(r"\\[vfF]+\[.+]", jaString)
+                    if voMatch:
+                        jaString = jaString.replace(voMatch.group(0), "")
+            
+            # Remove furigana if configured
+            if REMOVE_FURIGANA:
+                jaString = re.sub(r"＜(.*)＝.*＞", r"\1", jaString)
+            
+            # Store original for replacement
+            ojaString = jaString
+            
+            # Remove textwrap
+            jaString = jaString.replace("\\n", " ")
+            jaString = jaString.replace("\n", " ")
+            
+            # Pass 1: Collect strings
+            if not translatedList:
+                if speaker:
+                    stringList.append(f"[{speaker}]: {jaString}")
+                else:
+                    stringList.append(jaString)
+            
+            # Pass 2: Apply translations
+            else:
+                # Grab and pop translation
+                translatedText = translatedList[0]
+                translatedList.pop(0)
+                
+                # Remove speaker prefix from translation if present
+                if speaker:
+                    translatedText = re.sub(r"^\[?(.+?)\]?\s?[|:]\s?", "", translatedText)
+                
+                # Add wordwrap
+                translatedText = dazedwrap.wrapText(translatedText, WIDTH)
+                translatedText = translatedText.replace("\n", "\\n")
+                
+                # Determine target column
+                actual_target = TARGET_COLUMN + 1 if WRITE_TO_NEXT_COLUMN else TARGET_COLUMN
+                
+                # Ensure row has enough columns
+                while len(data[i]) <= actual_target:
+                    data[i].append("")
+                
+                # Set data
+                if PARSE_NAME_TAGS or PARSE_M_MARKERS:
+                    # Replace original text portion
+                    data[i][actual_target] = data[i][actual_target].replace(ojaString, translatedText) if data[i][actual_target] else translatedText
+                else:
+                    data[i][actual_target] = translatedText
+                
+                flush_progress_csv(writeFile, writer, data)
+            
+            # Iterate
+            i += 1
+
+        # EOF - Process collected strings
+        if len(stringList) > 0:
+            # Set Progress
+            pbar.total = len(stringList)
+            pbar.refresh()
+
+            # Translate
+            response = translateAI(stringList, "")
+            totalTokens[0] += response[1][0]
+            totalTokens[1] += response[1][1]
+            translatedList = response[0]
+
+            # Set Strings
+            if len(stringList) == len(translatedList):
+                translateCSV(data, pbar, writeFile, writer, filename, translatedList)
+            # Mismatch
+            else:
+                with LOCK:
+                    if filename not in MISMATCH:
+                        MISMATCH.append(filename)
+
+            # Write all Data
+            with LOCK:
+                if not ESTIMATE:
+                    for row in data:
+                        writer.writerow(row)
+            flush_progress_csv(writeFile, writer, data)
+
+    except Exception:
+        traceback.print_exc()
+
+        # Write all Data
+        with LOCK:
+            if not ESTIMATE:
+                for row in data:
+                    writer.writerow(row)
+        flush_progress_csv(writeFile, writer, data)
+        return totalTokens
+
+    return totalTokens
+
+# Save some money and enter the character before translation
+def getSpeaker(speaker):
+    match speaker:
+        case "ファイン":
+            return ["Fine", [0, 0]]
+        case "":
+            return ["", [0, 0]]
+        case _:
+            # Find Speaker
+            for i in range(len(NAMESLIST)):
+                if speaker == NAMESLIST[i][0]:
+                    return [NAMESLIST[i][1], [0, 0]]
+                
+            if speaker == "？？？":
+                return ["???", [0, 0]]
+
+            # Translate and Store Speaker
+            response = translateAI(
+                f"{speaker}",
+                "Reply with the " + LANGUAGE + " translation of the NPC name.",
+                False,
+            )
+            response[0] = response[0].title()
+            response[0] = response[0].replace("'S", "'s")
+            response[0] = response[0].replace("Speaker: ", "")
+
+            # Retry if name doesn't translate for some reason
+            if re.search(r"([a-zA-Z？?])", response[0]) == None:
+                response = translateAI(
+                    f"{speaker}",
+                    "Reply with the " + LANGUAGE + " translation of the NPC name.",
+                    False,
+                )
+                response[0] = response[0].title()
+                response[0] = response[0].replace("'S", "'s")
+
+            speakerList = [speaker, response[0]]
+            NAMESLIST.append(speakerList)
+            return response
+    return [speaker, [0, 0]]
+
+def translateAI(text, history, history_ctx=None):
+    """
+    Legacy wrapper function for the new shared translation utility.
+    This maintains compatibility with existing code while using the new shared implementation.
+    """
+    global PBAR, MISMATCH, FILENAME
+    
+    # Update config estimate mode based on global ESTIMATE
+    TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
+    
+    # Call the new shared translation function
+    return sharedtranslateAI(
+        text=text,
+        history=history,
+        config=TRANSLATION_CONFIG,
+        filename=FILENAME,
+        pbar=PBAR,
+        lock=LOCK,
+        mismatchList=MISMATCH
+    )

modules/images.py

@@ -1,439 +1,441 @@
-# Libraries
-from PIL import Image, ImageDraw, ImageFont
-import json
-import os
-import re
-import threading
-import time
-import traceback
-import tiktoken
-from pathlib import Path
-from colorama import Fore
-from dotenv import load_dotenv
-from retry import retry
-from tqdm import tqdm
-from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
-
-# Globals
-MODEL = os.getenv("model")
-TIMEOUT = int(os.getenv("timeout"))
-LANGUAGE = os.getenv("language").capitalize()
-PROMPT = Path("prompt.txt").read_text(encoding="utf-8")
-VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
-LOCK = threading.Lock()
-PBAR = None
-WIDTH = int(os.getenv("width"))
-LISTWIDTH = int(os.getenv("listWidth"))
-NOTEWIDTH = int(os.getenv("noteWidth"))
-MAXHISTORY = 10
-ESTIMATE = ""
-TOKENS = [0, 0]
-NAMESLIST = []
-MISMATCH = []  # Lists files that throw a mismatch error (Length of GPT list response is wrong)
-FILENAME = None
-
-# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
-LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
-
-# Get pricing configuration based on the model
-PRICING_CONFIG = getPricingConfig(MODEL)
-INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
-OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
-BATCHSIZE = PRICING_CONFIG["batchSize"]
-FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
-
-# Initialize Translation Config
-TRANSLATION_CONFIG = TranslationConfig(
-    model=MODEL,
-    language=LANGUAGE,
-    prompt=PROMPT,
-    vocab=VOCAB,
-    langRegex=LANGREGEX,
-    batchSize=BATCHSIZE,
-    maxHistory=MAXHISTORY,
-    estimateMode=False  # Will be set dynamically based on ESTIMATE
-)
-LEAVE = False
-
-# tqdm Globals
-BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
-POSITION = 0
-LEAVE = False
-
-
-def handleImages(folderName, estimate):
-    global ESTIMATE, TOKENS, FILENAME
-    ESTIMATE = estimate
-    FILENAME = folderName
-    start = time.time()
-
-    # Translate Strings
-    translatedData = openFiles(f"files/{folderName}")
-
-    # Custom Names
-    # customList = [[], []]
-    # customList = processImagesDir("Custom", customList)
-
-    # Write TL To Images
-    try:
-        translatedList, originalList, dimensionsList = translatedData[0]
-        for i in range(len(translatedList)):
-            try:
-                # Create image from string
-                image = stringToImageOutline(translatedList[i], dimensionsList[i][0], dimensionsList[i][1])
-                # Save image using the corresponding original filename
-                image.save(rf"translated/{folderName}/{originalList[i]}.png", quality=100)
-            except Exception as e:
-                # Log error if image saving fails
-                PBAR.write(f"Error processing {translatedList[i]}: {str(e)}")
-    except IndexError:
-        PBAR.write("Translated data is incomplete. Please check your input.")
-
-    # Print File
-    end = time.time()
-    tqdm.write(getResultString(translatedData, end - start, folderName))
-    with LOCK:
-        TOKENS[0] += translatedData[1][0]
-        TOKENS[1] += translatedData[1][1]
-
-    # Print Total
-    totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
-
-    # Print any errors on maps
-    if len(MISMATCH) > 0:
-        return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
-    else:
-        return totalString
-
-
-def openFiles(folderName):
-    global PBAR
-
-    if os.path.isdir(folderName):
-        imageList = [[], [], []]
-        imageList = processImagesDir(folderName, imageList)
-
-        # Start Translation
-        with tqdm(
-            bar_format=BAR_FORMAT,
-            position=POSITION,
-            leave=LEAVE,
-            desc=folderName,
-            total=len(imageList[0]),
-        ) as PBAR:
-            translatedData = translateImages(imageList)
-            translatedData = [
-                [translatedData[0], imageList[2], imageList[1]],
-                translatedData[1],
-                translatedData[2],
-            ]
-
-        return translatedData
-    else:
-        print("The provided directory path does not exist.")
-
-
-def getResultString(translatedData, translationTime, filename):
-    # File Print String
-    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
-    totalTokenstring = (
-        Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
-        "[Output: "
-        + str(translatedData[1][1])
-        + "]" "[Cost: ${:,.4f}".format(cost)
-        + "]"
-    )
-    timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
-
-    if translatedData[2] is None:
-        # Success
-        return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
-    else:
-        # Fail
-        try:
-            raise translatedData[2]
-        except Exception as e:
-            traceback.print_exc()
-            errorString = str(e) + Fore.RED
-            return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
-
-
-def getFontSize(text, image_width, image_height, font_path):
-    # Start with a high font size and keep reducing it until the text fits within the image bounds
-    font_size = min(image_width, image_height)
-
-    while font_size > 0:
-        font = ImageFont.truetype(font_path, font_size)
-        text_bbox = ImageDraw.Draw(Image.new("RGB", (1, 1))).textbbox((0, 0), text, font=font)
-        text_width = text_bbox[2] - text_bbox[0]
-        text_height = text_bbox[3] - text_bbox[1]
-
-        if text_width <= image_width and text_height <= image_height:
-            return font_size
-        font_size -= 1
-
-    return font_size
-
-
-def stringToImage(text, width, height, font_path="fonts/TsunagiGothic.ttf", scale_factor=4):
-    # Increase the resolution
-    scaled_width = int(width * scale_factor)
-    scaled_height = int(height * scale_factor)
-
-    # Find the appropriate font size for the scaled up image
-    font_size = getFontSize(text, scaled_width, scaled_height, font_path)
-    if font_size == 0:
-        raise ValueError("Text is too long to fit in the supplied dimensions.")
-
-    # Create a new image with the scaled width and height and a transparent background
-    image = Image.new("RGBA", (scaled_width, scaled_height), (255, 255, 255, 0))
-
-    # Create a drawing context
-    draw = ImageDraw.Draw(image)
-
-    # Load the appropriate font
-    font = ImageFont.truetype(font_path, font_size)
-
-    # Calculate the size of the text to center it
-    text_bbox = draw.textbbox((0, 0), text, font=font)
-    text_width = text_bbox[2] - text_bbox[0]
-    text_height = text_bbox[3] - text_bbox[1] + 20
-    x = 0
-
-    x = (scaled_width - text_width) // 2
-    y = (scaled_height - text_height) // 2
-
-    # Draw the text on the image
-    draw.text((x, y), text, font=font, fill=(255, 255, 255, 255))
-
-    # Resize back to the original dimensions to get a clearer text rendering
-    image = image.resize(
-        (width, height),
-        Image.LANCZOS,
-    )
-
-    return image
-
-
-from PIL import Image, ImageDraw, ImageFont
-
-
-def stringToImageOutline(text, width, height, font_path="fonts/TsunagiGothic.ttf", scale_factor=4):
-    # Outline
-    outline_color = (255, 255, 255, 255)
-    text_color = (0, 0, 0, 255)
-    outline_thickness = 4
-
-    # Increase the resolution
-    scaled_width = int(width * scale_factor)
-    scaled_height = int(height * scale_factor)
-
-    # Find the appropriate font size for the scaled up image
-    font_size = getFontSize(text, scaled_width, scaled_height, font_path)
-    if font_size == 0:
-        raise ValueError("Text is too long to fit in the supplied dimensions.")
-
-    # Create a new image with the scaled width and height and a transparent background
-    image = Image.new("RGBA", (scaled_width, scaled_height), (255, 255, 255, 0))
-
-    # Create a drawing context
-    draw = ImageDraw.Draw(image)
-
-    # Load the appropriate font
-    font = ImageFont.truetype(font_path, font_size)
-
-    # Calculate the size of the text to center it
-    text_bbox = draw.textbbox((0, 0), text, font=font)
-    text_width = text_bbox[2] - text_bbox[0]
-    text_height = text_bbox[3] - text_bbox[1] + 20
-    x = (scaled_width - text_width) // 2
-    y = (scaled_height - text_height) // 2
-
-    # Draw the text outline by applying the text multiple times with small offsets
-    for dx in range(-outline_thickness, outline_thickness + 1):
-        for dy in range(-outline_thickness, outline_thickness + 1):
-            if dx != 0 or dy != 0:
-                draw.text((x + dx, y + dy), text, font=font, fill=outline_color)
-
-    # Draw the main text
-    draw.text((x, y), text, font=font, fill=text_color)
-
-    # Resize back to the original dimensions to get a clearer text rendering
-    image = image.resize((width, height), Image.LANCZOS)
-
-    return image
-
-
-def stringToImageBox(text, width, height, font_path="fonts/TsunagiGothic.ttf", scale_factor=4):
-    # Increase the resolution
-    scaled_width = int(width * scale_factor)
-    scaled_height = int(height * scale_factor)
-
-    # Padding around the text
-    padding = 10
-
-    # Calculate the dimensions available for text placement
-    available_width = scaled_width - 2 * padding
-    available_height = scaled_height - 2 * padding
-
-    # Determine the best font size to fit within the available dimensions
-    font_size = getFontSize(text, available_width, available_height, font_path)
-    if font_size <= 0:
-        raise ValueError("Text is too long to fit in the supplied dimensions.")
-
-    # Create a new image with increased resolution
-    image = Image.new("RGBA", (scaled_width, scaled_height), (255, 255, 255, 0))
-    draw = ImageDraw.Draw(image)
-
-    # Load the calculated font
-    font = ImageFont.truetype(font_path, font_size)
-
-    # Calculate the size and bounding box of the text
-    text_bbox = draw.textbbox((0, 0), text, font=font)
-    text_width = text_bbox[2] - text_bbox[0]
-    text_height = text_bbox[3] - text_bbox[1] + 20
-
-    # Determine centered position for the text while considering padding
-    # Additional adjustment ensures text appears centrally aligned
-    x = (scaled_width - text_width) // 2
-    y = (scaled_height - text_height) // 2
-
-    # Draw a black box with a white outline that fits the image dimensions precisely
-    draw.rectangle([0, 0, scaled_width - 1, scaled_height - 1], outline=(255, 255, 255, 255), width=1)
-
-    # Fill the inside box with black color
-    draw.rectangle([1, 1, scaled_width - 2, scaled_height - 2], fill=(0, 0, 0, 255))
-
-    # Render the text within the image
-    draw.text((x, y), text, font=font, fill=(255, 255, 255, 255))
-
-    # Shrink the image back to original dimensions with high-quality interpolation
-    image = image.resize(
-        (width, height),
-        Image.LANCZOS,
-    )
-
-    return image
-
-
-def getImageDimensions(file_path):
-    try:
-        with Image.open(file_path) as img:
-            width, height = img.size
-            return width, height
-    except Exception as e:
-        print(f"Error reading {file_path}: {e}")
-        return None, None
-
-
-def processImagesDir(directory_path, imageList):
-    for file_name in os.listdir(directory_path):
-        # .png and Japanese
-        if ".png" in file_name:
-            file_path = os.path.join(directory_path, file_name)
-            if os.path.isfile(file_path):
-                # Check if the file is an image
-                try:
-                    width, height = getImageDimensions(file_path)
-                    if width is not None and height is not None:
-                        placeholders = {
-                            ".png": "",
-                        }
-                        for target, replacement in placeholders.items():
-                            file_name = file_name.replace(target, replacement)
-                            match = re.search(r"[\[【].+?[\]】](.*)", file_name)
-                            if match:
-                                text = match.group(1)
-                            else:
-                                text = file_name
-                        imageList[0].append(text)
-                        imageList[1].append([width, height])
-                        imageList[2].append(file_name)
-                except Exception as e:
-                    print(f"Error processing {file_name}: {e}")
-
-        if ".txt" in file_name:
-            try:
-                with open(f"{directory_path}/{file_name}", "r", encoding="utf8") as file:
-                    for line in file:
-                        line = line.strip()
-                        line = line.replace(":", "：")
-                        line = line.replace("/", "／")
-                        line = line.replace("?", "？")
-                        imageList[0].append(line)  # Using strip() to remove any extra newlines or spaces
-                        imageList[1].append([104, 15])
-            except FileNotFoundError:
-                print(f"The file at {file_path} was not found.")
-            except IOError:
-                print(f"An error occurred while reading the file at {file_path}.")
-    return imageList
-
-
-def translateImages(imageList):
-    totalTokens = [0, 0]
-
-    # Translate GPT
-    response = translateAI(imageList[0], "Keep the Translation as brief as possible")
-    translatedList = response[0]
-    totalTokens[0] += response[1][0]
-    totalTokens[1] += response[1][1]
-
-    return [translatedList, totalTokens, None]
-
-# Save some money and enter the character before translation
-def getSpeaker(speaker):
-    match speaker:
-        case "ファイン":
-            return ["Fine", [0, 0]]
-        case "":
-            return ["", [0, 0]]
-        case _:
-            # Find Speaker
-            for i in range(len(NAMESLIST)):
-                if speaker == NAMESLIST[i][0]:
-                    return [NAMESLIST[i][1], [0, 0]]
-
-            # Translate and Store Speaker
-            response = translateAI(
-                f"{speaker}",
-                "Reply with the " + LANGUAGE + " translation of the NPC name.",
-                False,
-            )
-            response[0] = response[0].title()
-            response[0] = response[0].replace("'S", "'s")
-            response[0] = response[0].replace("Speaker: ", "")
-
-            # Retry if name doesn't translate for some reason
-            if re.search(r"([a-zA-Z？?])", response[0]) == None:
-                response = translateAI(
-                    f"{speaker}",
-                    "Reply with the " + LANGUAGE + " translation of the NPC name.",
-                    False,
-                )
-                response[0] = response[0].title()
-                response[0] = response[0].replace("'S", "'s")
-
-            speakerList = [speaker, response[0]]
-            NAMESLIST.append(speakerList)
-            return response
-    return [speaker, [0, 0]]
-def translateAI(text, history, history_ctx=None):
-    """
-    Legacy wrapper function for the new shared translation utility.
-    This maintains compatibility with existing code while using the new shared implementation.
-    """
-    global PBAR, MISMATCH, FILENAME
-    
-    # Update config estimate mode based on global ESTIMATE
-    TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
-    
-    # Call the new shared translation function
-    return sharedtranslateAI(
-        text=text,
-        history=history,
-        config=TRANSLATION_CONFIG,
-        filename=FILENAME,
-        pbar=PBAR,
-        lock=LOCK,
-        mismatchList=MISMATCH
-    )
+# Libraries
+from PIL import Image, ImageDraw, ImageFont
+import json
+import os
+import re
+import threading
+import time
+import traceback
+import tiktoken
+from pathlib import Path
+from colorama import Fore
+from dotenv import load_dotenv
+from retry import retry
+from tqdm import tqdm
+from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
+
+# Globals
+MODEL = os.getenv("model")
+TIMEOUT = int(os.getenv("timeout"))
+LANGUAGE = os.getenv("language").capitalize()
+from util.paths import PROMPT_PATH, VOCAB_PATH
+
+PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
+VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
+LOCK = threading.Lock()
+PBAR = None
+WIDTH = int(os.getenv("width"))
+LISTWIDTH = int(os.getenv("listWidth"))
+NOTEWIDTH = int(os.getenv("noteWidth"))
+MAXHISTORY = 10
+ESTIMATE = ""
+TOKENS = [0, 0]
+NAMESLIST = []
+MISMATCH = []  # Lists files that throw a mismatch error (Length of GPT list response is wrong)
+FILENAME = None
+
+# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
+LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
+
+# Get pricing configuration based on the model
+PRICING_CONFIG = getPricingConfig(MODEL)
+INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
+OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
+BATCHSIZE = PRICING_CONFIG["batchSize"]
+FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
+
+# Initialize Translation Config
+TRANSLATION_CONFIG = TranslationConfig(
+    model=MODEL,
+    language=LANGUAGE,
+    prompt=PROMPT,
+    vocab=VOCAB,
+    langRegex=LANGREGEX,
+    batchSize=BATCHSIZE,
+    maxHistory=MAXHISTORY,
+    estimateMode=False  # Will be set dynamically based on ESTIMATE
+)
+LEAVE = False
+
+# tqdm Globals
+BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
+POSITION = 0
+LEAVE = False
+
+
+def handleImages(folderName, estimate):
+    global ESTIMATE, TOKENS, FILENAME
+    ESTIMATE = estimate
+    FILENAME = folderName
+    start = time.time()
+
+    # Translate Strings
+    translatedData = openFiles(f"files/{folderName}")
+
+    # Custom Names
+    # customList = [[], []]
+    # customList = processImagesDir("Custom", customList)
+
+    # Write TL To Images
+    try:
+        translatedList, originalList, dimensionsList = translatedData[0]
+        for i in range(len(translatedList)):
+            try:
+                # Create image from string
+                image = stringToImageOutline(translatedList[i], dimensionsList[i][0], dimensionsList[i][1])
+                # Save image using the corresponding original filename
+                image.save(rf"translated/{folderName}/{originalList[i]}.png", quality=100)
+            except Exception as e:
+                # Log error if image saving fails
+                PBAR.write(f"Error processing {translatedList[i]}: {str(e)}")
+    except IndexError:
+        PBAR.write("Translated data is incomplete. Please check your input.")
+
+    # Print File
+    end = time.time()
+    tqdm.write(getResultString(translatedData, end - start, folderName))
+    with LOCK:
+        TOKENS[0] += translatedData[1][0]
+        TOKENS[1] += translatedData[1][1]
+
+    # Print Total
+    totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
+
+    # Print any errors on maps
+    if len(MISMATCH) > 0:
+        return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
+    else:
+        return totalString
+
+
+def openFiles(folderName):
+    global PBAR
+
+    if os.path.isdir(folderName):
+        imageList = [[], [], []]
+        imageList = processImagesDir(folderName, imageList)
+
+        # Start Translation
+        with tqdm(
+            bar_format=BAR_FORMAT,
+            position=POSITION,
+            leave=LEAVE,
+            desc=folderName,
+            total=len(imageList[0]),
+        ) as PBAR:
+            translatedData = translateImages(imageList)
+            translatedData = [
+                [translatedData[0], imageList[2], imageList[1]],
+                translatedData[1],
+                translatedData[2],
+            ]
+
+        return translatedData
+    else:
+        print("The provided directory path does not exist.")
+
+
+def getResultString(translatedData, translationTime, filename):
+    # File Print String
+    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
+    totalTokenstring = (
+        Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
+        "[Output: "
+        + str(translatedData[1][1])
+        + "]" "[Cost: ${:,.4f}".format(cost)
+        + "]"
+    )
+    timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
+
+    if translatedData[2] is None:
+        # Success
+        return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
+    else:
+        # Fail
+        try:
+            raise translatedData[2]
+        except Exception as e:
+            traceback.print_exc()
+            errorString = str(e) + Fore.RED
+            return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
+
+
+def getFontSize(text, image_width, image_height, font_path):
+    # Start with a high font size and keep reducing it until the text fits within the image bounds
+    font_size = min(image_width, image_height)
+
+    while font_size > 0:
+        font = ImageFont.truetype(font_path, font_size)
+        text_bbox = ImageDraw.Draw(Image.new("RGB", (1, 1))).textbbox((0, 0), text, font=font)
+        text_width = text_bbox[2] - text_bbox[0]
+        text_height = text_bbox[3] - text_bbox[1]
+
+        if text_width <= image_width and text_height <= image_height:
+            return font_size
+        font_size -= 1
+
+    return font_size
+
+
+def stringToImage(text, width, height, font_path="fonts/TsunagiGothic.ttf", scale_factor=4):
+    # Increase the resolution
+    scaled_width = int(width * scale_factor)
+    scaled_height = int(height * scale_factor)
+
+    # Find the appropriate font size for the scaled up image
+    font_size = getFontSize(text, scaled_width, scaled_height, font_path)
+    if font_size == 0:
+        raise ValueError("Text is too long to fit in the supplied dimensions.")
+
+    # Create a new image with the scaled width and height and a transparent background
+    image = Image.new("RGBA", (scaled_width, scaled_height), (255, 255, 255, 0))
+
+    # Create a drawing context
+    draw = ImageDraw.Draw(image)
+
+    # Load the appropriate font
+    font = ImageFont.truetype(font_path, font_size)
+
+    # Calculate the size of the text to center it
+    text_bbox = draw.textbbox((0, 0), text, font=font)
+    text_width = text_bbox[2] - text_bbox[0]
+    text_height = text_bbox[3] - text_bbox[1] + 20
+    x = 0
+
+    x = (scaled_width - text_width) // 2
+    y = (scaled_height - text_height) // 2
+
+    # Draw the text on the image
+    draw.text((x, y), text, font=font, fill=(255, 255, 255, 255))
+
+    # Resize back to the original dimensions to get a clearer text rendering
+    image = image.resize(
+        (width, height),
+        Image.LANCZOS,
+    )
+
+    return image
+
+
+from PIL import Image, ImageDraw, ImageFont
+
+
+def stringToImageOutline(text, width, height, font_path="fonts/TsunagiGothic.ttf", scale_factor=4):
+    # Outline
+    outline_color = (255, 255, 255, 255)
+    text_color = (0, 0, 0, 255)
+    outline_thickness = 4
+
+    # Increase the resolution
+    scaled_width = int(width * scale_factor)
+    scaled_height = int(height * scale_factor)
+
+    # Find the appropriate font size for the scaled up image
+    font_size = getFontSize(text, scaled_width, scaled_height, font_path)
+    if font_size == 0:
+        raise ValueError("Text is too long to fit in the supplied dimensions.")
+
+    # Create a new image with the scaled width and height and a transparent background
+    image = Image.new("RGBA", (scaled_width, scaled_height), (255, 255, 255, 0))
+
+    # Create a drawing context
+    draw = ImageDraw.Draw(image)
+
+    # Load the appropriate font
+    font = ImageFont.truetype(font_path, font_size)
+
+    # Calculate the size of the text to center it
+    text_bbox = draw.textbbox((0, 0), text, font=font)
+    text_width = text_bbox[2] - text_bbox[0]
+    text_height = text_bbox[3] - text_bbox[1] + 20
+    x = (scaled_width - text_width) // 2
+    y = (scaled_height - text_height) // 2
+
+    # Draw the text outline by applying the text multiple times with small offsets
+    for dx in range(-outline_thickness, outline_thickness + 1):
+        for dy in range(-outline_thickness, outline_thickness + 1):
+            if dx != 0 or dy != 0:
+                draw.text((x + dx, y + dy), text, font=font, fill=outline_color)
+
+    # Draw the main text
+    draw.text((x, y), text, font=font, fill=text_color)
+
+    # Resize back to the original dimensions to get a clearer text rendering
+    image = image.resize((width, height), Image.LANCZOS)
+
+    return image
+
+
+def stringToImageBox(text, width, height, font_path="fonts/TsunagiGothic.ttf", scale_factor=4):
+    # Increase the resolution
+    scaled_width = int(width * scale_factor)
+    scaled_height = int(height * scale_factor)
+
+    # Padding around the text
+    padding = 10
+
+    # Calculate the dimensions available for text placement
+    available_width = scaled_width - 2 * padding
+    available_height = scaled_height - 2 * padding
+
+    # Determine the best font size to fit within the available dimensions
+    font_size = getFontSize(text, available_width, available_height, font_path)
+    if font_size <= 0:
+        raise ValueError("Text is too long to fit in the supplied dimensions.")
+
+    # Create a new image with increased resolution
+    image = Image.new("RGBA", (scaled_width, scaled_height), (255, 255, 255, 0))
+    draw = ImageDraw.Draw(image)
+
+    # Load the calculated font
+    font = ImageFont.truetype(font_path, font_size)
+
+    # Calculate the size and bounding box of the text
+    text_bbox = draw.textbbox((0, 0), text, font=font)
+    text_width = text_bbox[2] - text_bbox[0]
+    text_height = text_bbox[3] - text_bbox[1] + 20
+
+    # Determine centered position for the text while considering padding
+    # Additional adjustment ensures text appears centrally aligned
+    x = (scaled_width - text_width) // 2
+    y = (scaled_height - text_height) // 2
+
+    # Draw a black box with a white outline that fits the image dimensions precisely
+    draw.rectangle([0, 0, scaled_width - 1, scaled_height - 1], outline=(255, 255, 255, 255), width=1)
+
+    # Fill the inside box with black color
+    draw.rectangle([1, 1, scaled_width - 2, scaled_height - 2], fill=(0, 0, 0, 255))
+
+    # Render the text within the image
+    draw.text((x, y), text, font=font, fill=(255, 255, 255, 255))
+
+    # Shrink the image back to original dimensions with high-quality interpolation
+    image = image.resize(
+        (width, height),
+        Image.LANCZOS,
+    )
+
+    return image
+
+
+def getImageDimensions(file_path):
+    try:
+        with Image.open(file_path) as img:
+            width, height = img.size
+            return width, height
+    except Exception as e:
+        print(f"Error reading {file_path}: {e}")
+        return None, None
+
+
+def processImagesDir(directory_path, imageList):
+    for file_name in os.listdir(directory_path):
+        # .png and Japanese
+        if ".png" in file_name:
+            file_path = os.path.join(directory_path, file_name)
+            if os.path.isfile(file_path):
+                # Check if the file is an image
+                try:
+                    width, height = getImageDimensions(file_path)
+                    if width is not None and height is not None:
+                        placeholders = {
+                            ".png": "",
+                        }
+                        for target, replacement in placeholders.items():
+                            file_name = file_name.replace(target, replacement)
+                            match = re.search(r"[\[【].+?[\]】](.*)", file_name)
+                            if match:
+                                text = match.group(1)
+                            else:
+                                text = file_name
+                        imageList[0].append(text)
+                        imageList[1].append([width, height])
+                        imageList[2].append(file_name)
+                except Exception as e:
+                    print(f"Error processing {file_name}: {e}")
+
+        if ".txt" in file_name:
+            try:
+                with open(f"{directory_path}/{file_name}", "r", encoding="utf8") as file:
+                    for line in file:
+                        line = line.strip()
+                        line = line.replace(":", "：")
+                        line = line.replace("/", "／")
+                        line = line.replace("?", "？")
+                        imageList[0].append(line)  # Using strip() to remove any extra newlines or spaces
+                        imageList[1].append([104, 15])
+            except FileNotFoundError:
+                print(f"The file at {file_path} was not found.")
+            except IOError:
+                print(f"An error occurred while reading the file at {file_path}.")
+    return imageList
+
+
+def translateImages(imageList):
+    totalTokens = [0, 0]
+
+    # Translate GPT
+    response = translateAI(imageList[0], "Keep the Translation as brief as possible")
+    translatedList = response[0]
+    totalTokens[0] += response[1][0]
+    totalTokens[1] += response[1][1]
+
+    return [translatedList, totalTokens, None]
+
+# Save some money and enter the character before translation
+def getSpeaker(speaker):
+    match speaker:
+        case "ファイン":
+            return ["Fine", [0, 0]]
+        case "":
+            return ["", [0, 0]]
+        case _:
+            # Find Speaker
+            for i in range(len(NAMESLIST)):
+                if speaker == NAMESLIST[i][0]:
+                    return [NAMESLIST[i][1], [0, 0]]
+
+            # Translate and Store Speaker
+            response = translateAI(
+                f"{speaker}",
+                "Reply with the " + LANGUAGE + " translation of the NPC name.",
+                False,
+            )
+            response[0] = response[0].title()
+            response[0] = response[0].replace("'S", "'s")
+            response[0] = response[0].replace("Speaker: ", "")
+
+            # Retry if name doesn't translate for some reason
+            if re.search(r"([a-zA-Z？?])", response[0]) == None:
+                response = translateAI(
+                    f"{speaker}",
+                    "Reply with the " + LANGUAGE + " translation of the NPC name.",
+                    False,
+                )
+                response[0] = response[0].title()
+                response[0] = response[0].replace("'S", "'s")
+
+            speakerList = [speaker, response[0]]
+            NAMESLIST.append(speakerList)
+            return response
+    return [speaker, [0, 0]]
+def translateAI(text, history, history_ctx=None):
+    """
+    Legacy wrapper function for the new shared translation utility.
+    This maintains compatibility with existing code while using the new shared implementation.
+    """
+    global PBAR, MISMATCH, FILENAME
+    
+    # Update config estimate mode based on global ESTIMATE
+    TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
+    
+    # Call the new shared translation function
+    return sharedtranslateAI(
+        text=text,
+        history=history,
+        config=TRANSLATION_CONFIG,
+        filename=FILENAME,
+        pbar=PBAR,
+        lock=LOCK,
+        mismatchList=MISMATCH
+    )

modules/lune.py

@@ -1,372 +1,374 @@
-# Libraries
-import json
-import os
-import re
-import util.dazedwrap as dazedwrap
-import threading
-import time
-import traceback
-import tiktoken
-from pathlib import Path
-from colorama import Fore
-from dotenv import load_dotenv
-from retry import retry
-from tqdm import tqdm
-from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
-import tempfile
-
-# Globals
-MODEL = os.getenv("model")
-TIMEOUT = int(os.getenv("timeout"))
-LANGUAGE = os.getenv("language").capitalize()
-PROMPT = Path("prompt.txt").read_text(encoding="utf-8")
-VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
-LOCK = threading.Lock()
-WIDTH = int(os.getenv("width"))
-LISTWIDTH = int(os.getenv("listWidth"))
-NOTEWIDTH = 70
-MAXHISTORY = 10
-ESTIMATE = ""
-TOKENS = [0, 0]
-NAMESLIST = []
-NAMES = False  # Output a list of all the character names found
-BRFLAG = False  # If the game uses <br> instead
-FIXTEXTWRAP = True  # Overwrites textwrap
-IGNORETLTEXT = False  # Ignores all translated text.
-MISMATCH = []  # Lists files that throw a mismatch error (Length of GPT list response is wrong)
-PBAR = None
-FILENAME = None
-
-# tqdm Globals
-BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
-POSITION = 0
-LEAVE = False
-
-# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
-LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
-
-# Get pricing configuration based on the model
-PRICING_CONFIG = getPricingConfig(MODEL)
-INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
-OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
-BATCHSIZE = PRICING_CONFIG["batchSize"]
-FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
-
-# Initialize Translation Config
-TRANSLATION_CONFIG = TranslationConfig(
-    model=MODEL,
-    language=LANGUAGE,
-    prompt=PROMPT,
-    vocab=VOCAB,
-    langRegex=LANGREGEX,
-    batchSize=BATCHSIZE,
-    maxHistory=MAXHISTORY,
-    estimateMode=False  # Will be set dynamically based on ESTIMATE
-)
-LEAVE = False
-
-def handleLune(filename, estimate):
-    global FILENAME, ESTIMATE, totalTokens
-    ESTIMATE = estimate
-    FILENAME = filename
-
-    if estimate:
-        start = time.time()
-        translatedData = openFiles(filename)
-
-        # Print Result
-        end = time.time()
-        tqdm.write(getResultString(translatedData, end - start, filename))
-        with LOCK:
-            TOKENS[0] += translatedData[1][0]
-            TOKENS[1] += translatedData[1][1]
-
-        return getResultString(["", TOKENS, None], end - start, "TOTAL")
-
-    else:
-        try:
-            with open("translated/" + filename, "w", encoding="utf-8", newline="\n") as outFile:
-                start = time.time()
-                translatedData = openFiles(filename)
-
-                # Print Result
-                end = time.time()
-                json.dump(translatedData[0], outFile, ensure_ascii=False, indent=4)
-                tqdm.write(getResultString(translatedData, end - start, filename))
-                with LOCK:
-                    TOKENS[0] += translatedData[1][0]
-                    TOKENS[1] += translatedData[1][1]
-        except Exception:
-            return "Fail"
-
-    return getResultString(["", TOKENS, None], end - start, "TOTAL")
-
-
-def openFiles(filename):
-    with open("files/" + filename, "r", encoding="UTF-8-sig") as f:
-        data = json.load(f)
-
-        # Map Files
-        if ".json" in filename:
-            translatedData = parseJSON(data, filename)
-
-        else:
-            raise NameError(filename + " Not Supported")
-
-    return translatedData
-
-
-def getResultString(translatedData, translationTime, filename):
-    # File Print String
-    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
-    totalTokenstring = (
-        Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
-        "[Output: "
-        + str(translatedData[1][1])
-        + "]" "[Cost: ${:,.4f}".format(cost)
-        + "]"
-    )
-    timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
-
-    if translatedData[2] == None:
-        # Success
-        return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
-
-    else:
-        # Fail
-        try:
-            raise translatedData[2]
-        except Exception as e:
-            traceback.print_exc()
-            errorString = str(e) + Fore.RED
-            return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
-
-
-def parseJSON(data, filename):
-    totalTokens = [0, 0]
-    totalLines = 0
-    totalLines = len(data)
-    global LOCK
-
-    with tqdm(bar_format=BAR_FORMAT, position=POSITION, total=totalLines, leave=LEAVE) as pbar:
-        pbar.desc = filename
-        pbar.total = totalLines
-        try:
-            result = translateJSON(data, pbar)
-            totalTokens[0] += result[0]
-            totalTokens[1] += result[1]
-        except Exception as e:
-            return [data, totalTokens, e]
-    return [data, totalTokens, None]
-
-
-def save_progress_json(data, filename):
-    """Atomically save current JSON translation progress."""
-    try:
-        if ESTIMATE:
-            return
-        os.makedirs("translated", exist_ok=True)
-        tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
-        try:
-            with os.fdopen(tmp_fd, "w", encoding="utf-8", newline="\n") as tmp_file:
-                json.dump(data, tmp_file, ensure_ascii=False, indent=4)
-            os.replace(tmp_path, os.path.join("translated", filename))
-        finally:
-            if os.path.exists(tmp_path):
-                try:
-                    os.remove(tmp_path)
-                except OSError:
-                    pass
-    except Exception:
-        traceback.print_exc()
-
-
-def translateJSON(data, pbar):
-    global PBAR
-    PBAR = pbar
-    textHistory = []
-    batch = []
-    maxHistory = MAXHISTORY
-    tokens = [0, 0]
-    speaker = "None"
-    insertBool = False
-    i = 0
-    batchStartIndex = 0
-
-    while i < len(data):
-        item = data[i]
-        # Speaker
-        if "name" in item:
-            if item["name"] not in [None, "-"]:
-                response = getSpeaker(item["name"])
-                speaker = response[0]
-                tokens[0] += response[1][0]
-                tokens[1] += response[1][1]
-                item["name"] = speaker
-                save_progress_json(data, FILENAME or "output.json")
-            else:
-                speaker = "None"
-
-        # Text
-        if "message" in item:
-            for text in [
-                "text",
-                "text2",
-                "help1",
-                "help2",
-                "help3",
-                "like",
-                "message",
-                "me",
-            ]:
-                if text in item:
-                    if item[text] != None:
-                        jaString = item[text]
-
-                        # Remove any textwrap
-                        if FIXTEXTWRAP == True:
-                            finalJAString = jaString.replace("\n", " ")
-
-                        # [Passthrough 1] Pulling From File
-                        if insertBool is False:
-                            # Append to List and Clear Values
-                            batch.append(finalJAString)
-                            speaker = ""
-
-                            # Translate Batch if Full
-                            if len(batch) == BATCHSIZE:
-                                # Translate
-                                response = translateAI(batch, textHistory)
-                                tokens[0] += response[1][0]
-                                tokens[1] += response[1][1]
-                                translatedBatch = response[0]
-                                textHistory = translatedBatch[-10:]
-
-                                # Set Values
-                                if len(batch) == len(translatedBatch):
-                                    i = batchStartIndex
-                                    insertBool = True
-
-                                # Mismatch
-                                else:
-                                    pbar.write(f"Mismatch: {batchStartIndex} - {i}")
-                                    MISMATCH.append(batch)
-                                    batchStartIndex = i
-                                    batch.clear()
-
-                            if insertBool is False:
-                                i += 1
-
-                            currentGroup = []
-
-                        # [Passthrough 2] Setting Data
-                        else:
-                            # Get Text
-                            translatedText = translatedBatch[0]
-
-                            # Remove added speaker
-                            translatedText = re.sub(r"^.+?:\s", "", translatedText)
-
-                            # Textwrap
-                            translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
-
-                            # Set Text
-                            item[text] = translatedText
-                            save_progress_json(data, FILENAME or "output.json")
-                            translatedBatch.pop(0)
-                            speaker = ""
-                            currentGroup = []
-                            i += 1
-
-                            # If Batch is empty. Move on.
-                            if len(translatedBatch) == 0:
-                                insertBool = False
-                                batchStartIndex = i
-                                batch.clear()
-        else:
-            i += 1
-
-        # Translate Batch if not empty and EOF
-        if len(batch) != 0 and i >= len(data):
-            # Translate
-            response = translateAI(batch, textHistory)
-            tokens[0] += response[1][0]
-            tokens[1] += response[1][1]
-            translatedBatch = response[0]
-            textHistory = translatedBatch[-10:]
-
-            # Set Values
-            if len(batch) == len(translatedBatch):
-                i = batchStartIndex
-                insertBool = True
-
-            # Mismatch
-            else:
-                pbar.write(f"Mismatch: {batchStartIndex} - {i}")
-                MISMATCH.append(batch)
-                batchStartIndex = i
-                batch.clear()
-
-            currentGroup = []
-            # After applying a batch, persist progress
-            save_progress_json(data, FILENAME or "output.json")
-    return tokens
-
-# Save some money and enter the character before translation
-def getSpeaker(speaker):
-    match speaker:
-        case "ファイン":
-            return ["Fine", [0, 0]]
-        case "":
-            return ["", [0, 0]]
-        case _:
-            # Find Speaker
-            for i in range(len(NAMESLIST)):
-                if speaker == NAMESLIST[i][0]:
-                    return [NAMESLIST[i][1], [0, 0]]
-
-            # Translate and Store Speaker
-            response = translateAI(
-                f"{speaker}",
-                "Reply with the " + LANGUAGE + " translation of the NPC name.",
-                False,
-            )
-            response[0] = response[0].title()
-            response[0] = response[0].replace("'S", "'s")
-            response[0] = response[0].replace("Speaker: ", "")
-
-            # Retry if name doesn't translate for some reason
-            if re.search(r"([a-zA-Z？?])", response[0]) == None:
-                response = translateAI(
-                    f"{speaker}",
-                    "Reply with the " + LANGUAGE + " translation of the NPC name.",
-                    False,
-                )
-                response[0] = response[0].title()
-                response[0] = response[0].replace("'S", "'s")
-
-            speakerList = [speaker, response[0]]
-            NAMESLIST.append(speakerList)
-            return response
-    return [speaker, [0, 0]]
-
-def translateAI(text, history, history_ctx=None):
-    """
-    Legacy wrapper function for the new shared translation utility.
-    This maintains compatibility with existing code while using the new shared implementation.
-    """
-    global PBAR, MISMATCH, FILENAME
-    
-    # Update config estimate mode based on global ESTIMATE
-    TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
-    
-    # Call the new shared translation function
-    return sharedtranslateAI(
-        text=text,
-        history=history,
-        config=TRANSLATION_CONFIG,
-        filename=FILENAME,
-        pbar=PBAR,
-        lock=LOCK,
-        mismatchList=MISMATCH
-    )
+# Libraries
+import json
+import os
+import re
+import util.dazedwrap as dazedwrap
+import threading
+import time
+import traceback
+import tiktoken
+from pathlib import Path
+from colorama import Fore
+from dotenv import load_dotenv
+from retry import retry
+from tqdm import tqdm
+from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
+import tempfile
+
+# Globals
+MODEL = os.getenv("model")
+TIMEOUT = int(os.getenv("timeout"))
+LANGUAGE = os.getenv("language").capitalize()
+from util.paths import PROMPT_PATH, VOCAB_PATH
+
+PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
+VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
+LOCK = threading.Lock()
+WIDTH = int(os.getenv("width"))
+LISTWIDTH = int(os.getenv("listWidth"))
+NOTEWIDTH = 70
+MAXHISTORY = 10
+ESTIMATE = ""
+TOKENS = [0, 0]
+NAMESLIST = []
+NAMES = False  # Output a list of all the character names found
+BRFLAG = False  # If the game uses <br> instead
+FIXTEXTWRAP = True  # Overwrites textwrap
+IGNORETLTEXT = False  # Ignores all translated text.
+MISMATCH = []  # Lists files that throw a mismatch error (Length of GPT list response is wrong)
+PBAR = None
+FILENAME = None
+
+# tqdm Globals
+BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
+POSITION = 0
+LEAVE = False
+
+# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
+LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
+
+# Get pricing configuration based on the model
+PRICING_CONFIG = getPricingConfig(MODEL)
+INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
+OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
+BATCHSIZE = PRICING_CONFIG["batchSize"]
+FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
+
+# Initialize Translation Config
+TRANSLATION_CONFIG = TranslationConfig(
+    model=MODEL,
+    language=LANGUAGE,
+    prompt=PROMPT,
+    vocab=VOCAB,
+    langRegex=LANGREGEX,
+    batchSize=BATCHSIZE,
+    maxHistory=MAXHISTORY,
+    estimateMode=False  # Will be set dynamically based on ESTIMATE
+)
+LEAVE = False
+
+def handleLune(filename, estimate):
+    global FILENAME, ESTIMATE, totalTokens
+    ESTIMATE = estimate
+    FILENAME = filename
+
+    if estimate:
+        start = time.time()
+        translatedData = openFiles(filename)
+
+        # Print Result
+        end = time.time()
+        tqdm.write(getResultString(translatedData, end - start, filename))
+        with LOCK:
+            TOKENS[0] += translatedData[1][0]
+            TOKENS[1] += translatedData[1][1]
+
+        return getResultString(["", TOKENS, None], end - start, "TOTAL")
+
+    else:
+        try:
+            with open("translated/" + filename, "w", encoding="utf-8", newline="\n") as outFile:
+                start = time.time()
+                translatedData = openFiles(filename)
+
+                # Print Result
+                end = time.time()
+                json.dump(translatedData[0], outFile, ensure_ascii=False, indent=4)
+                tqdm.write(getResultString(translatedData, end - start, filename))
+                with LOCK:
+                    TOKENS[0] += translatedData[1][0]
+                    TOKENS[1] += translatedData[1][1]
+        except Exception:
+            return "Fail"
+
+    return getResultString(["", TOKENS, None], end - start, "TOTAL")
+
+
+def openFiles(filename):
+    with open("files/" + filename, "r", encoding="UTF-8-sig") as f:
+        data = json.load(f)
+
+        # Map Files
+        if ".json" in filename:
+            translatedData = parseJSON(data, filename)
+
+        else:
+            raise NameError(filename + " Not Supported")
+
+    return translatedData
+
+
+def getResultString(translatedData, translationTime, filename):
+    # File Print String
+    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
+    totalTokenstring = (
+        Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
+        "[Output: "
+        + str(translatedData[1][1])
+        + "]" "[Cost: ${:,.4f}".format(cost)
+        + "]"
+    )
+    timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
+
+    if translatedData[2] == None:
+        # Success
+        return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
+
+    else:
+        # Fail
+        try:
+            raise translatedData[2]
+        except Exception as e:
+            traceback.print_exc()
+            errorString = str(e) + Fore.RED
+            return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
+
+
+def parseJSON(data, filename):
+    totalTokens = [0, 0]
+    totalLines = 0
+    totalLines = len(data)
+    global LOCK
+
+    with tqdm(bar_format=BAR_FORMAT, position=POSITION, total=totalLines, leave=LEAVE) as pbar:
+        pbar.desc = filename
+        pbar.total = totalLines
+        try:
+            result = translateJSON(data, pbar)
+            totalTokens[0] += result[0]
+            totalTokens[1] += result[1]
+        except Exception as e:
+            return [data, totalTokens, e]
+    return [data, totalTokens, None]
+
+
+def save_progress_json(data, filename):
+    """Atomically save current JSON translation progress."""
+    try:
+        if ESTIMATE:
+            return
+        os.makedirs("translated", exist_ok=True)
+        tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
+        try:
+            with os.fdopen(tmp_fd, "w", encoding="utf-8", newline="\n") as tmp_file:
+                json.dump(data, tmp_file, ensure_ascii=False, indent=4)
+            os.replace(tmp_path, os.path.join("translated", filename))
+        finally:
+            if os.path.exists(tmp_path):
+                try:
+                    os.remove(tmp_path)
+                except OSError:
+                    pass
+    except Exception:
+        traceback.print_exc()
+
+
+def translateJSON(data, pbar):
+    global PBAR
+    PBAR = pbar
+    textHistory = []
+    batch = []
+    maxHistory = MAXHISTORY
+    tokens = [0, 0]
+    speaker = "None"
+    insertBool = False
+    i = 0
+    batchStartIndex = 0
+
+    while i < len(data):
+        item = data[i]
+        # Speaker
+        if "name" in item:
+            if item["name"] not in [None, "-"]:
+                response = getSpeaker(item["name"])
+                speaker = response[0]
+                tokens[0] += response[1][0]
+                tokens[1] += response[1][1]
+                item["name"] = speaker
+                save_progress_json(data, FILENAME or "output.json")
+            else:
+                speaker = "None"
+
+        # Text
+        if "message" in item:
+            for text in [
+                "text",
+                "text2",
+                "help1",
+                "help2",
+                "help3",
+                "like",
+                "message",
+                "me",
+            ]:
+                if text in item:
+                    if item[text] != None:
+                        jaString = item[text]
+
+                        # Remove any textwrap
+                        if FIXTEXTWRAP == True:
+                            finalJAString = jaString.replace("\n", " ")
+
+                        # [Passthrough 1] Pulling From File
+                        if insertBool is False:
+                            # Append to List and Clear Values
+                            batch.append(finalJAString)
+                            speaker = ""
+
+                            # Translate Batch if Full
+                            if len(batch) == BATCHSIZE:
+                                # Translate
+                                response = translateAI(batch, textHistory)
+                                tokens[0] += response[1][0]
+                                tokens[1] += response[1][1]
+                                translatedBatch = response[0]
+                                textHistory = translatedBatch[-10:]
+
+                                # Set Values
+                                if len(batch) == len(translatedBatch):
+                                    i = batchStartIndex
+                                    insertBool = True
+
+                                # Mismatch
+                                else:
+                                    pbar.write(f"Mismatch: {batchStartIndex} - {i}")
+                                    MISMATCH.append(batch)
+                                    batchStartIndex = i
+                                    batch.clear()
+
+                            if insertBool is False:
+                                i += 1
+
+                            currentGroup = []
+
+                        # [Passthrough 2] Setting Data
+                        else:
+                            # Get Text
+                            translatedText = translatedBatch[0]
+
+                            # Remove added speaker
+                            translatedText = re.sub(r"^.+?:\s", "", translatedText)
+
+                            # Textwrap
+                            translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
+
+                            # Set Text
+                            item[text] = translatedText
+                            save_progress_json(data, FILENAME or "output.json")
+                            translatedBatch.pop(0)
+                            speaker = ""
+                            currentGroup = []
+                            i += 1
+
+                            # If Batch is empty. Move on.
+                            if len(translatedBatch) == 0:
+                                insertBool = False
+                                batchStartIndex = i
+                                batch.clear()
+        else:
+            i += 1
+
+        # Translate Batch if not empty and EOF
+        if len(batch) != 0 and i >= len(data):
+            # Translate
+            response = translateAI(batch, textHistory)
+            tokens[0] += response[1][0]
+            tokens[1] += response[1][1]
+            translatedBatch = response[0]
+            textHistory = translatedBatch[-10:]
+
+            # Set Values
+            if len(batch) == len(translatedBatch):
+                i = batchStartIndex
+                insertBool = True
+
+            # Mismatch
+            else:
+                pbar.write(f"Mismatch: {batchStartIndex} - {i}")
+                MISMATCH.append(batch)
+                batchStartIndex = i
+                batch.clear()
+
+            currentGroup = []
+            # After applying a batch, persist progress
+            save_progress_json(data, FILENAME or "output.json")
+    return tokens
+
+# Save some money and enter the character before translation
+def getSpeaker(speaker):
+    match speaker:
+        case "ファイン":
+            return ["Fine", [0, 0]]
+        case "":
+            return ["", [0, 0]]
+        case _:
+            # Find Speaker
+            for i in range(len(NAMESLIST)):
+                if speaker == NAMESLIST[i][0]:
+                    return [NAMESLIST[i][1], [0, 0]]
+
+            # Translate and Store Speaker
+            response = translateAI(
+                f"{speaker}",
+                "Reply with the " + LANGUAGE + " translation of the NPC name.",
+                False,
+            )
+            response[0] = response[0].title()
+            response[0] = response[0].replace("'S", "'s")
+            response[0] = response[0].replace("Speaker: ", "")
+
+            # Retry if name doesn't translate for some reason
+            if re.search(r"([a-zA-Z？?])", response[0]) == None:
+                response = translateAI(
+                    f"{speaker}",
+                    "Reply with the " + LANGUAGE + " translation of the NPC name.",
+                    False,
+                )
+                response[0] = response[0].title()
+                response[0] = response[0].replace("'S", "'s")
+
+            speakerList = [speaker, response[0]]
+            NAMESLIST.append(speakerList)
+            return response
+    return [speaker, [0, 0]]
+
+def translateAI(text, history, history_ctx=None):
+    """
+    Legacy wrapper function for the new shared translation utility.
+    This maintains compatibility with existing code while using the new shared implementation.
+    """
+    global PBAR, MISMATCH, FILENAME
+    
+    # Update config estimate mode based on global ESTIMATE
+    TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
+    
+    # Call the new shared translation function
+    return sharedtranslateAI(
+        text=text,
+        history=history,
+        config=TRANSLATION_CONFIG,
+        filename=FILENAME,
+        pbar=PBAR,
+        lock=LOCK,
+        mismatchList=MISMATCH
+    )

modules/main.py

@@ -1,381 +1,385 @@
-import sys
-import os
-import traceback
-import datetime
-from pathlib import Path
-from concurrent.futures import ThreadPoolExecutor, as_completed
-from colorama import Fore
-from tqdm import tqdm
-from dotenv import load_dotenv
-
-# This needs to be before the module imports as some of them currently try to read and use some of these values
-# upon import, in which case if they are unset the script will crash before we can output these messages.
-load_dotenv()
-_missing_envs = [
-    env for env in [
-        "api",
-        "key",
-        "model",
-        "language",
-        "timeout",
-        "fileThreads",
-        "threads",
-        "width",
-        "listWidth",
-    ]
-    if os.getenv(env) is None or str(os.getenv(env))[:1] == "<"
-]
-if _missing_envs:
-    names = ", ".join(_missing_envs)
-    tqdm.write(
-        Fore.RED
-        + f"Missing required environment variable(s): {names}. "
-        + "Set them in a .env file (see .env.example)."
-    )
-
-from modules.rpgmakermvmz import handleMVMZ, setSpeakerParseMode as setSpeakerParseMVMZ, finalizeSpeakerParse as finalizeSpeakerParseMVMZ
-from modules.csv import handleCSV
-from modules.tyrano import handleTyrano
-from modules.kirikiri import handleKirikiri
-from modules.json import handleJSON
-from modules.lune import handleLune
-from modules.yuris import handleYuris
-from modules.nscript import handleOnscripter
-from modules.wolf import handleWOLF
-from modules.wolf2 import handleWOLF2
-from modules.regex import handleRegex
-from modules.text import handleText
-from modules.renpy import handleRenpy
-from modules.unity import handleUnity
-from modules.images import handleImages
-from modules.rpgmakerplugin import handlePlugin
-from modules.srpg import handleSRPG
-from modules.aquedi4 import handleAquedi4
-
-# For GPT4 rate limit will be hit if you have more than 1 thread.
-# 1 Thread for each file. Controls how many files are worked on at once.
-THREADS = int(os.getenv("fileThreads"))
-
-# [Display name, file extension, handle function]
-MODULES = [
-    ["RPGMaker MV/MZ", ["json"], handleMVMZ],
-    ["RPGMaker Plugins", ["js", "rb"], handlePlugin],
-    ["CSV (From Translator++)", ["csv"], handleCSV],
-    ["Tyrano", ["ks"], handleTyrano],
-    ["Kirikiri", ["ks", "tjs", "ssd", "asd"], handleKirikiri],
-    ["JSON", ["json"], handleJSON],
-    ["Lune", ["json"], handleLune],
-    ["Yuris", ["json"], handleYuris],
-    ["NScript", ["txt"], handleOnscripter],
-    ["Wolf", ["json"], handleWOLF],
-    ["Wolf", ["txt"], handleWOLF2],
-    ["Regex", ["txt", "json", "script", "csv"], handleRegex],
-    ["Text", ["txt", "srt"], handleText],
-    ["Renpy", ["rpy"], handleRenpy],
-    ["Unity", ["txt"], handleUnity],
-    ["SRPG Studio", ["json"], handleSRPG],
-    ["Images", [""], handleImages],
-    ["Aquedi4 Prepared JSON", [".json"], handleAquedi4],
-]
-
-# Info Message
-tqdm.write(
-    Fore.CYAN
-    + "-Dazed MTL Tool -"
-    + Fore.RESET,
-    end="\n\n",
-)
-
-
-def main():
-    from util.translation import clear_cache
-
-    estimate = ""
-    batch_mode = False
-    speaker_parse = False  # Deferred until after engine select
-    while estimate == "":
-        estimate = input("Select Mode:\n\n 1. Translate\n 2. Estimate\n 3. Batch Translate (Anthropic Batches API, 50% off)\n")
-        match estimate:
-            case "1":
-                estimate = False
-            case "2":
-                estimate = True
-            case "3":
-                estimate = False
-                batch_mode = True
-            case _:
-                estimate = ""
-
-    resume_state = None
-    if batch_mode:
-        from util.translation import isClaudeNative, batchRunState
-        if not isClaudeNative(os.getenv("model", "")):
-            tqdm.write(
-                Fore.RED
-                + "Batch Translate requires a Claude model with the 'api' env var unset or pointing at anthropic.com."
-                + Fore.RESET
-            )
-            return
-        # An interrupted batch run can be resumed instead of re-collecting
-        # (a second submission would be billed again).
-        resume_state = batchRunState()
-        if resume_state:
-            confirm = ""
-            while confirm not in ("y", "n"):
-                confirm = input(f"A previous batch run was interrupted ({resume_state}). Resume it? (y/n)\n").strip().lower()
-            if confirm == "n":
-                resume_state = None
-
-    # Clear the translation cache at the start of the run. Kept when resuming a
-    # batch so names translated during collect stay consistent with the queued
-    # payloads in the consume pass.
-    if not resume_state:
-        clear_cache()
-
-    version = ""
-    while True:
-        tqdm.write("Select game engine:\n")
-        for position, module in enumerate(MODULES):
-            tqdm.write(f"{str(position + 1).rjust(2)}. {module[0]} (.{module[1]})")
-        version = input()
-        try:
-            version = int(version) - 1
-        except:
-            continue
-        if version in range(len(MODULES)):
-            break
-
-    totalCost = (
-        Fore.RED
-        + "Translation module didn't return the total cost. Make sure the \
-files to translate are in the /files folder and that you picked the right game engine."
-    )
-
-    # If translating RPGMaker MV/MZ, prompt for speaker parse mode
-    speaker_parse = False
-    if version == 0 and not estimate and not batch_mode:
-        sub = ""
-        while sub == "":
-            sub = input("RPGMaker MV/MZ options:\n\n 1. Standard Translate\n 2. Parse Speakers (collect speaker names only)\n")
-            match sub:
-                case "1":
-                    speaker_parse = False
-                case "2":
-                    speaker_parse = True
-                case _:
-                    sub = ""
-        if speaker_parse:
-            setSpeakerParseMVMZ(True)
-
-    # Open File (Threads) - recursively walk 'files' and preserve directory structure
-    # Prepare per-run log file so CLI runs also write to a run-specific history file
-    try:
-        hist_dir = Path("log") / "history"
-        hist_dir.mkdir(parents=True, exist_ok=True)
-        
-        # Clean up old log files, keeping only the 10 most recent
-        try:
-            log_files = sorted(hist_dir.glob("translationHistory_*.txt"), key=lambda p: p.stat().st_mtime, reverse=True)
-            # Keep only the 10 most recent, delete the rest
-            for old_log in log_files[10:]:
-                try:
-                    old_log.unlink()
-                except Exception:
-                    pass
-        except Exception:
-            pass
-        
-        fname = datetime.datetime.now().strftime("translationHistory_%Y%m%d_%H%M%S.txt")
-        run_log_path = hist_dir / fname
-        # Don't create the file yet - it will be created when first log is written
-        # Store the path in environment variable
-        try:
-            os.environ['TRANSLATION_RUN_LOG'] = str(run_log_path)
-        except Exception:
-            pass
-
-        # Try to create a hard link from legacy path to this run file for compatibility
-        # This will be created when the run_log_path file is first written to
-        legacy = Path("log") / "translationHistory.txt"
-        try:
-            if legacy.exists():
-                try:
-                    legacy.unlink()
-                except Exception:
-                    pass
-        except Exception:
-            pass
-    except Exception:
-        pass
-    
-    def runFiles(estimate):
-        runCost = totalCost
-        # Use single worker for estimate mode to prevent race conditions
-        max_workers = 1 if estimate else THREADS
-        with ThreadPoolExecutor(max_workers=max_workers) as executor:
-            futures = []
-            files_root = "files"
-
-            # Special-case: Images engine expects a folder, not a file; schedule per directory containing assets
-            if MODULES[version][0] == "Images":
-                for root, dirs, filenames in os.walk(files_root):
-                    # Skip hidden/system directories
-                    dirs[:] = [d for d in dirs if d not in {".git", "__pycache__"}]
-
-                    # Skip the root 'files' itself to avoid processing everything twice
-                    # We'll still allow scheduling for root if it contains assets
-
-                    # Only schedule directories that contain potential assets
-                    has_assets = any(fn.lower().endswith((".png", ".txt")) for fn in filenames)
-                    if not has_assets:
-                        continue
-
-                    # Compute relative directory path and ensure translated mirror exists
-                    rel_dir = os.path.relpath(root, files_root).replace(os.sep, "/")
-                    if rel_dir == ".":
-                        # Represent root as empty string so handler creates files under translated/ directly
-                        rel_dir = ""
-                    try:
-                        target_dir = os.path.join("translated", rel_dir.replace("/", os.sep)) if rel_dir else "translated"
-                        os.makedirs(target_dir, exist_ok=True)
-                    except Exception:
-                        pass
-
-                    futures.append(
-                        executor.submit(MODULES[version][2], rel_dir, estimate)
-                    )
-            else:
-                # Gather all candidate files recursively
-                for root, dirs, filenames in os.walk(files_root):
-                    # Skip hidden/system directories if any
-                    dirs[:] = [d for d in dirs if d not in {".git", "__pycache__"}]
-
-                    for fname in filenames:
-                        if fname == ".gitkeep":
-                            continue
-
-                        abs_path = os.path.join(root, fname)
-                        # Build relative path from 'files' root using POSIX-style separators so handlers can do 'files/' + rel
-                        rel_path = os.path.relpath(abs_path, files_root)
-                        rel_path_posix = rel_path.replace(os.sep, "/")
-
-                        # Check extension match for the selected module version
-                        for m in MODULES[version][1]:
-                            if rel_path_posix.endswith(m):
-                                # Ensure the corresponding directory exists under 'translated'
-                                rel_dir = os.path.dirname(rel_path_posix)
-                                if rel_dir:
-                                    try:
-                                        os.makedirs(os.path.join("translated", rel_dir.replace("/", os.sep)), exist_ok=True)
-                                    except Exception:
-                                        # Best-effort; handler may attempt write and fail if permissions are insufficient
-                                        pass
-
-                                futures.append(
-                                    executor.submit(MODULES[version][2], rel_path_posix, estimate)
-                                )
-                                break  # Avoid double-adding if multiple ext entries match
-
-            for future in as_completed(futures):
-                try:
-                    runCost = future.result()
-                except Exception as e:
-                    tracebackLineNo = str(traceback.extract_tb(sys.exc_info()[2])[-1].lineno)
-                    tqdm.write(Fore.RED + str(e) + "|" + tracebackLineNo + Fore.RESET)
-        return runCost
-
-    if batch_mode:
-        from util.translation import (
-            set_batch_phase,
-            clearBatchFiles,
-            pendingBatchRequests,
-            estimateBatchCost,
-            runTranslationBatches,
-        )
-        poll = int(os.getenv("batchPollInterval", "60") or 60)
-        run_consume = True
-
-        if resume_state is None:
-            clearBatchFiles()
-
-            # Pass 1 — queue dialogue for the batch; speaker/variable strings still hit live API.
-            tqdm.write(Fore.CYAN + "[BATCH] Pass 1/2: collecting requests..." + Fore.RESET)
-            tqdm.write(
-                Fore.YELLOW
-                + "[BATCH] Note: speaker names and similar short strings translate at live "
-                "API rates during collect (dialogue is batched after you confirm)."
-                + Fore.RESET
-            )
-            set_batch_phase("collect")
-            try:
-                totalCost = runFiles(False)
-            finally:
-                set_batch_phase(None)
-
-            if pendingBatchRequests() == 0:
-                tqdm.write("[BATCH] No requests queued — nothing needed the API.")
-                run_consume = False
-            else:
-                estimateBatchCost()
-                confirm = ""
-                while confirm not in ("y", "n"):
-                    confirm = input("Submit batch? (y/n)\n").strip().lower()
-                if confirm == "n":
-                    tqdm.write("[BATCH] Not submitted. The queue is kept in log/batch_requests.json.")
-                    return
-                runTranslationBatches(poll)
-        elif resume_state == "submitted":
-            tqdm.write(Fore.CYAN + "[BATCH] Resuming the submitted batch..." + Fore.RESET)
-            runTranslationBatches(poll)
-        else:  # "fetched" — results already downloaded, just write the files
-            tqdm.write(Fore.CYAN + "[BATCH] Resuming from fetched results..." + Fore.RESET)
-
-        if run_consume:
-            # Pass 2 — write the translated files from the fetched results.
-            # Anything the batch missed falls back to the live API.
-            tqdm.write(Fore.CYAN + "[BATCH] Pass 2/2: writing translated files..." + Fore.RESET)
-            set_batch_phase("consume")
-            try:
-                totalCost = runFiles(False)
-            finally:
-                set_batch_phase(None)
-    else:
-        totalCost = runFiles(estimate)
-
-    # Finalize speaker parse mode by writing collected speakers to vocab
-    if speaker_parse:
-        finalizeSpeakerParseMVMZ()
-
-    # Delete Tmp Files
-    if os.path.isfile("csv.tmp"):
-        os.remove("csv.tmp")
-
-    # Sweep any leftover temp files in translated/
-    try:
-        translated_dir = os.path.join("translated")
-        if os.path.isdir(translated_dir):
-            for fname in os.listdir(translated_dir):
-                if fname.endswith(".tmp"):
-                    fpath = os.path.join(translated_dir, fname)
-                    try:
-                        os.remove(fpath)
-                    except Exception:
-                        # Best-effort cleanup; ignore files locked by other processes
-                        pass
-    except Exception:
-        pass
-
-    # Finish
-    if totalCost != "Fail":
-        # if estimate is False:
-        # This is to encourage people to grab what's in /translated instead
-        # deleteFolderFiles("files")
-
-        tqdm.write(str(totalCost))
-
-
-def deleteFolderFiles(folderPath):
-    for filename in os.listdir(folderPath):
-        file_path = os.path.join(folderPath, filename)
-        if file_path.endswith((".json", ".ks")):
+import sys
+import os
+import traceback
+import datetime
+from pathlib import Path
+from concurrent.futures import ThreadPoolExecutor, as_completed
+from colorama import Fore
+from tqdm import tqdm
+from dotenv import load_dotenv
+
+# This needs to be before the module imports as some of them currently try to read and use some of these values
+# upon import, in which case if they are unset the script will crash before we can output these messages.
+load_dotenv()
+from util.paths import migrate_root_data_files, ensure_vocab_file
+
+migrate_root_data_files()
+ensure_vocab_file()
+_missing_envs = [
+    env for env in [
+        "api",
+        "key",
+        "model",
+        "language",
+        "timeout",
+        "fileThreads",
+        "threads",
+        "width",
+        "listWidth",
+    ]
+    if os.getenv(env) is None or str(os.getenv(env))[:1] == "<"
+]
+if _missing_envs:
+    names = ", ".join(_missing_envs)
+    tqdm.write(
+        Fore.RED
+        + f"Missing required environment variable(s): {names}. "
+        + "Set them in a .env file (see .env.example)."
+    )
+
+from modules.rpgmakermvmz import handleMVMZ, setSpeakerParseMode as setSpeakerParseMVMZ, finalizeSpeakerParse as finalizeSpeakerParseMVMZ
+from modules.csv import handleCSV
+from modules.tyrano import handleTyrano
+from modules.kirikiri import handleKirikiri
+from modules.json import handleJSON
+from modules.lune import handleLune
+from modules.yuris import handleYuris
+from modules.nscript import handleOnscripter
+from modules.wolf import handleWOLF
+from modules.wolf2 import handleWOLF2
+from modules.regex import handleRegex
+from modules.text import handleText
+from modules.renpy import handleRenpy
+from modules.unity import handleUnity
+from modules.images import handleImages
+from modules.rpgmakerplugin import handlePlugin
+from modules.srpg import handleSRPG
+from modules.aquedi4 import handleAquedi4
+
+# For GPT4 rate limit will be hit if you have more than 1 thread.
+# 1 Thread for each file. Controls how many files are worked on at once.
+THREADS = int(os.getenv("fileThreads"))
+
+# [Display name, file extension, handle function]
+MODULES = [
+    ["RPGMaker MV/MZ", ["json"], handleMVMZ],
+    ["RPGMaker Plugins", ["js", "rb"], handlePlugin],
+    ["CSV (From Translator++)", ["csv"], handleCSV],
+    ["Tyrano", ["ks"], handleTyrano],
+    ["Kirikiri", ["ks", "tjs", "ssd", "asd"], handleKirikiri],
+    ["JSON", ["json"], handleJSON],
+    ["Lune", ["json"], handleLune],
+    ["Yuris", ["json"], handleYuris],
+    ["NScript", ["txt"], handleOnscripter],
+    ["Wolf", ["json"], handleWOLF],
+    ["Wolf", ["txt"], handleWOLF2],
+    ["Regex", ["txt", "json", "script", "csv"], handleRegex],
+    ["Text", ["txt", "srt"], handleText],
+    ["Renpy", ["rpy"], handleRenpy],
+    ["Unity", ["txt"], handleUnity],
+    ["SRPG Studio", ["json"], handleSRPG],
+    ["Images", [""], handleImages],
+    ["Aquedi4 Prepared JSON", [".json"], handleAquedi4],
+]
+
+# Info Message
+tqdm.write(
+    Fore.CYAN
+    + "-Dazed MTL Tool -"
+    + Fore.RESET,
+    end="\n\n",
+)
+
+
+def main():
+    from util.translation import clear_cache
+
+    estimate = ""
+    batch_mode = False
+    speaker_parse = False  # Deferred until after engine select
+    while estimate == "":
+        estimate = input("Select Mode:\n\n 1. Translate\n 2. Estimate\n 3. Batch Translate (Anthropic Batches API, 50% off)\n")
+        match estimate:
+            case "1":
+                estimate = False
+            case "2":
+                estimate = True
+            case "3":
+                estimate = False
+                batch_mode = True
+            case _:
+                estimate = ""
+
+    resume_state = None
+    if batch_mode:
+        from util.translation import isClaudeNative, batchRunState
+        if not isClaudeNative(os.getenv("model", "")):
+            tqdm.write(
+                Fore.RED
+                + "Batch Translate requires a Claude model with the 'api' env var unset or pointing at anthropic.com."
+                + Fore.RESET
+            )
+            return
+        # An interrupted batch run can be resumed instead of re-collecting
+        # (a second submission would be billed again).
+        resume_state = batchRunState()
+        if resume_state:
+            confirm = ""
+            while confirm not in ("y", "n"):
+                confirm = input(f"A previous batch run was interrupted ({resume_state}). Resume it? (y/n)\n").strip().lower()
+            if confirm == "n":
+                resume_state = None
+
+    # Clear the translation cache at the start of the run. Kept when resuming a
+    # batch so names translated during collect stay consistent with the queued
+    # payloads in the consume pass.
+    if not resume_state:
+        clear_cache()
+
+    version = ""
+    while True:
+        tqdm.write("Select game engine:\n")
+        for position, module in enumerate(MODULES):
+            tqdm.write(f"{str(position + 1).rjust(2)}. {module[0]} (.{module[1]})")
+        version = input()
+        try:
+            version = int(version) - 1
+        except:
+            continue
+        if version in range(len(MODULES)):
+            break
+
+    totalCost = (
+        Fore.RED
+        + "Translation module didn't return the total cost. Make sure the \
+files to translate are in the /files folder and that you picked the right game engine."
+    )
+
+    # If translating RPGMaker MV/MZ, prompt for speaker parse mode
+    speaker_parse = False
+    if version == 0 and not estimate and not batch_mode:
+        sub = ""
+        while sub == "":
+            sub = input("RPGMaker MV/MZ options:\n\n 1. Standard Translate\n 2. Parse Speakers (collect speaker names only)\n")
+            match sub:
+                case "1":
+                    speaker_parse = False
+                case "2":
+                    speaker_parse = True
+                case _:
+                    sub = ""
+        if speaker_parse:
+            setSpeakerParseMVMZ(True)
+
+    # Open File (Threads) - recursively walk 'files' and preserve directory structure
+    # Prepare per-run log file so CLI runs also write to a run-specific history file
+    try:
+        hist_dir = Path("log") / "history"
+        hist_dir.mkdir(parents=True, exist_ok=True)
+        
+        # Clean up old log files, keeping only the 10 most recent
+        try:
+            log_files = sorted(hist_dir.glob("translationHistory_*.txt"), key=lambda p: p.stat().st_mtime, reverse=True)
+            # Keep only the 10 most recent, delete the rest
+            for old_log in log_files[10:]:
+                try:
+                    old_log.unlink()
+                except Exception:
+                    pass
+        except Exception:
+            pass
+        
+        fname = datetime.datetime.now().strftime("translationHistory_%Y%m%d_%H%M%S.txt")
+        run_log_path = hist_dir / fname
+        # Don't create the file yet - it will be created when first log is written
+        # Store the path in environment variable
+        try:
+            os.environ['TRANSLATION_RUN_LOG'] = str(run_log_path)
+        except Exception:
+            pass
+
+        # Try to create a hard link from legacy path to this run file for compatibility
+        # This will be created when the run_log_path file is first written to
+        legacy = Path("log") / "translationHistory.txt"
+        try:
+            if legacy.exists():
+                try:
+                    legacy.unlink()
+                except Exception:
+                    pass
+        except Exception:
+            pass
+    except Exception:
+        pass
+    
+    def runFiles(estimate):
+        runCost = totalCost
+        # Use single worker for estimate mode to prevent race conditions
+        max_workers = 1 if estimate else THREADS
+        with ThreadPoolExecutor(max_workers=max_workers) as executor:
+            futures = []
+            files_root = "files"
+
+            # Special-case: Images engine expects a folder, not a file; schedule per directory containing assets
+            if MODULES[version][0] == "Images":
+                for root, dirs, filenames in os.walk(files_root):
+                    # Skip hidden/system directories
+                    dirs[:] = [d for d in dirs if d not in {".git", "__pycache__"}]
+
+                    # Skip the root 'files' itself to avoid processing everything twice
+                    # We'll still allow scheduling for root if it contains assets
+
+                    # Only schedule directories that contain potential assets
+                    has_assets = any(fn.lower().endswith((".png", ".txt")) for fn in filenames)
+                    if not has_assets:
+                        continue
+
+                    # Compute relative directory path and ensure translated mirror exists
+                    rel_dir = os.path.relpath(root, files_root).replace(os.sep, "/")
+                    if rel_dir == ".":
+                        # Represent root as empty string so handler creates files under translated/ directly
+                        rel_dir = ""
+                    try:
+                        target_dir = os.path.join("translated", rel_dir.replace("/", os.sep)) if rel_dir else "translated"
+                        os.makedirs(target_dir, exist_ok=True)
+                    except Exception:
+                        pass
+
+                    futures.append(
+                        executor.submit(MODULES[version][2], rel_dir, estimate)
+                    )
+            else:
+                # Gather all candidate files recursively
+                for root, dirs, filenames in os.walk(files_root):
+                    # Skip hidden/system directories if any
+                    dirs[:] = [d for d in dirs if d not in {".git", "__pycache__"}]
+
+                    for fname in filenames:
+                        if fname == ".gitkeep":
+                            continue
+
+                        abs_path = os.path.join(root, fname)
+                        # Build relative path from 'files' root using POSIX-style separators so handlers can do 'files/' + rel
+                        rel_path = os.path.relpath(abs_path, files_root)
+                        rel_path_posix = rel_path.replace(os.sep, "/")
+
+                        # Check extension match for the selected module version
+                        for m in MODULES[version][1]:
+                            if rel_path_posix.endswith(m):
+                                # Ensure the corresponding directory exists under 'translated'
+                                rel_dir = os.path.dirname(rel_path_posix)
+                                if rel_dir:
+                                    try:
+                                        os.makedirs(os.path.join("translated", rel_dir.replace("/", os.sep)), exist_ok=True)
+                                    except Exception:
+                                        # Best-effort; handler may attempt write and fail if permissions are insufficient
+                                        pass
+
+                                futures.append(
+                                    executor.submit(MODULES[version][2], rel_path_posix, estimate)
+                                )
+                                break  # Avoid double-adding if multiple ext entries match
+
+            for future in as_completed(futures):
+                try:
+                    runCost = future.result()
+                except Exception as e:
+                    tracebackLineNo = str(traceback.extract_tb(sys.exc_info()[2])[-1].lineno)
+                    tqdm.write(Fore.RED + str(e) + "|" + tracebackLineNo + Fore.RESET)
+        return runCost
+
+    if batch_mode:
+        from util.translation import (
+            set_batch_phase,
+            clearBatchFiles,
+            pendingBatchRequests,
+            estimateBatchCost,
+            runTranslationBatches,
+        )
+        poll = int(os.getenv("batchPollInterval", "60") or 60)
+        run_consume = True
+
+        if resume_state is None:
+            clearBatchFiles()
+
+            # Pass 1 — queue dialogue for the batch; speaker/variable strings still hit live API.
+            tqdm.write(Fore.CYAN + "[BATCH] Pass 1/2: collecting requests..." + Fore.RESET)
+            tqdm.write(
+                Fore.YELLOW
+                + "[BATCH] Note: speaker names and similar short strings translate at live "
+                "API rates during collect (dialogue is batched after you confirm)."
+                + Fore.RESET
+            )
+            set_batch_phase("collect")
+            try:
+                totalCost = runFiles(False)
+            finally:
+                set_batch_phase(None)
+
+            if pendingBatchRequests() == 0:
+                tqdm.write("[BATCH] No requests queued — nothing needed the API.")
+                run_consume = False
+            else:
+                estimateBatchCost()
+                confirm = ""
+                while confirm not in ("y", "n"):
+                    confirm = input("Submit batch? (y/n)\n").strip().lower()
+                if confirm == "n":
+                    tqdm.write("[BATCH] Not submitted. The queue is kept in log/batch_requests.json.")
+                    return
+                runTranslationBatches(poll)
+        elif resume_state == "submitted":
+            tqdm.write(Fore.CYAN + "[BATCH] Resuming the submitted batch..." + Fore.RESET)
+            runTranslationBatches(poll)
+        else:  # "fetched" — results already downloaded, just write the files
+            tqdm.write(Fore.CYAN + "[BATCH] Resuming from fetched results..." + Fore.RESET)
+
+        if run_consume:
+            # Pass 2 — write the translated files from the fetched results.
+            # Anything the batch missed falls back to the live API.
+            tqdm.write(Fore.CYAN + "[BATCH] Pass 2/2: writing translated files..." + Fore.RESET)
+            set_batch_phase("consume")
+            try:
+                totalCost = runFiles(False)
+            finally:
+                set_batch_phase(None)
+    else:
+        totalCost = runFiles(estimate)
+
+    # Finalize speaker parse mode by writing collected speakers to vocab
+    if speaker_parse:
+        finalizeSpeakerParseMVMZ()
+
+    # Delete Tmp Files
+    if os.path.isfile("csv.tmp"):
+        os.remove("csv.tmp")
+
+    # Sweep any leftover temp files in translated/
+    try:
+        translated_dir = os.path.join("translated")
+        if os.path.isdir(translated_dir):
+            for fname in os.listdir(translated_dir):
+                if fname.endswith(".tmp"):
+                    fpath = os.path.join(translated_dir, fname)
+                    try:
+                        os.remove(fpath)
+                    except Exception:
+                        # Best-effort cleanup; ignore files locked by other processes
+                        pass
+    except Exception:
+        pass
+
+    # Finish
+    if totalCost != "Fail":
+        # if estimate is False:
+        # This is to encourage people to grab what's in /translated instead
+        # deleteFolderFiles("files")
+
+        tqdm.write(str(totalCost))
+
+
+def deleteFolderFiles(folderPath):
+    for filename in os.listdir(folderPath):
+        file_path = os.path.join(folderPath, filename)
+        if file_path.endswith((".json", ".ks")):
             os.remove(file_path)

modules/nscript.py

@@ -1,426 +1,428 @@
-# Libraries
-import json
-import os
-import re
-import util.dazedwrap as dazedwrap
-import threading
-import time
-import traceback
-import tiktoken
-from pathlib import Path
-from colorama import Fore
-from dotenv import load_dotenv
-from retry import retry
-from tqdm import tqdm
-from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
-import tempfile
-
-# Globals
-MODEL = os.getenv("model")
-TIMEOUT = int(os.getenv("timeout"))
-LANGUAGE = os.getenv("language").capitalize()
-PROMPT = Path("prompt.txt").read_text(encoding="utf-8")
-VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
-LOCK = threading.Lock()
-WIDTH = int(os.getenv("width"))
-LISTWIDTH = int(os.getenv("listWidth"))
-NOTEWIDTH = 70
-MAXHISTORY = 10
-ESTIMATE = ""
-TOKENS = [0, 0]
-NAMESLIST = []
-NAMES = False  # Output a list of all the character names found
-BRFLAG = False  # If the game uses <br> instead
-FIXTEXTWRAP = True  # Overwrites textwrap
-IGNORETLTEXT = False  # Ignores all translated text.
-MISMATCH = []  # Lists files that throw a mismatch error (Length of GPT list response is wrong)
-CONVERTTOWIDE = False # Default (False)
-
-# tqdm Globals
-BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
-POSITION = 0
-LEAVE = False
-PBAR = None
-FILENAME = None
-
-# Full Width
-ascii_to_wide = dict((i, chr(i + 0xFEE0)) for i in range(0x21, 0x7F))
-ascii_to_wide.update({0x20: "\u3000", 0x2D: "\u2212"})  # space and minus
-wide_to_ascii = dict((i, chr(i - 0xFEE0)) for i in range(0xFF01, 0xFF5F))
-wide_to_ascii.update({0x3000: " ", 0x2212: "-"})  # space and minus
-
-# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
-LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
-
-# Get pricing configuration based on the model
-PRICING_CONFIG = getPricingConfig(MODEL)
-INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
-OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
-BATCHSIZE = PRICING_CONFIG["batchSize"]
-FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
-
-# Initialize Translation Config
-TRANSLATION_CONFIG = TranslationConfig(
-    model=MODEL,
-    language=LANGUAGE,
-    prompt=PROMPT,
-    vocab=VOCAB,
-    langRegex=LANGREGEX,
-    batchSize=BATCHSIZE,
-    maxHistory=MAXHISTORY,
-    estimateMode=False  # Will be set dynamically based on ESTIMATE
-)
-LEAVE = False
-
-def handleOnscripter(filename, estimate):
-    global ESTIMATE, FILENAME
-    ESTIMATE = estimate
-    FILENAME = filename
-
-    if ESTIMATE:
-        start = time.time()
-        translatedData = openFiles(filename)
-
-        # Print Result
-        end = time.time()
-        tqdm.write(getResultString(translatedData, end - start, filename))
-        with LOCK:
-            TOKENS[0] += translatedData[1][0]
-            TOKENS[1] += translatedData[1][1]
-
-        # Print Total
-        totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
-
-        # Print any errors on maps
-        if len(MISMATCH) > 0:
-            return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
-        else:
-            return totalString
-
-    else:
-        try:
-            with open("translated/" + filename, "w", encoding="cp932", errors="ignore") as outFile:
-                start = time.time()
-                translatedData = openFiles(filename)
-
-                # Print Result
-                end = time.time()
-                outFile.writelines(translatedData[0])
-                tqdm.write(getResultString(translatedData, end - start, filename))
-                with LOCK:
-                    TOKENS[0] += translatedData[1][0]
-                    TOKENS[1] += translatedData[1][1]
-        except Exception:
-            traceback.print_exc()
-            return "Fail"
-
-    return getResultString(["", TOKENS, None], end - start, "TOTAL")
-
-
-def getResultString(translatedData, translationTime, filename):
-    # File Print String
-    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
-    totalTokenstring = (
-        Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
-        "[Output: "
-        + str(translatedData[1][1])
-        + "]" "[Cost: ${:,.4f}".format(cost)
-        + "]"
-    )
-    timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
-
-    if translatedData[2] == None:
-        # Success
-        return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
-
-    else:
-        # Fail
-        try:
-            raise translatedData[2]
-        except Exception as e:
-            traceback.print_exc()
-            errorString = str(e) + Fore.RED
-            return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
-
-
-def openFiles(filename):
-    with open("files/" + filename, "r", encoding="cp932") as readFile:
-        translatedData = parseOnscripter(readFile, filename)
-
-        # Delete lines marked for deletion
-        finalData = []
-        for line in translatedData[0]:
-            if line != "\\d\n":
-                finalData.append(line)
-        translatedData[0] = finalData
-
-    return translatedData
-
-
-def parseOnscripter(readFile, filename):
-    totalTokens = [0, 0]
-
-    # Read File into data
-    data = readFile.readlines()
-
-    # Create Progress Bar
-    with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
-        pbar.desc = filename
-
-        try:
-            result = translateOnscripter(data, pbar, filename, [])
-            totalTokens[0] += result[0]
-            totalTokens[1] += result[1]
-        except Exception as e:
-            traceback.print_exc()
-            return [data, totalTokens, e]
-    return [data, totalTokens, None]
-
-
-def save_progress_lines(lines, filename, encoding="cp932"):
-    """Atomically save current line-based translation progress."""
-    try:
-        if ESTIMATE:
-            return
-        os.makedirs("translated", exist_ok=True)
-        tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
-        try:
-            with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file:
-                tmp_file.writelines(lines)
-            os.replace(tmp_path, os.path.join("translated", filename))
-        finally:
-            if os.path.exists(tmp_path):
-                try:
-                    os.remove(tmp_path)
-                except OSError:
-                    pass
-    except Exception:
-        traceback.print_exc()
-
-
-def translateOnscripter(data, pbar, filename, translatedList):
-    stringList = []
-    currentGroup = []
-    tokens = [0, 0]
-    speaker = ""
-    voice = False
-    global LOCK, ESTIMATE, PBAR
-    PBAR = pbar
-    i = 0
-
-    # Dialogue
-    while i < len(data):
-        # Lines
-        regex = r"^([\u3000「（【[][^\n]+)"
-        match = re.search(regex, data[i])
-        if match != None and match.group(1) != "":
-            originalString = match.group(1)
-            # Pass 1
-            if translatedList == []:
-                # Grab Consecutive Strings
-                jaString = match.group(1)
-                while len(data) > i + 1 and re.match(regex, data[i + 1]):
-                    data[i] = ""
-                    i += 1
-                    jaString = f"{jaString} {data[i]}"
-
-                # Convert from Wide
-                jaString = jaString.translate(wide_to_ascii)
-
-                # Remove any textwrap and \u3000 and \
-                jaString = jaString.replace("\n", "")
-                jaString = jaString.replace("\u3000", "")
-                jaString = jaString.replace("\\", "")
-                jaString = jaString.replace("　＞", ")")
-                jaString = jaString.replace("＜　", "(")
-
-                # Remove Furigana
-                furiMatch = re.findall(r"({(.+?)\/(.+?)})", jaString)
-                if furiMatch:
-                    for match in furiMatch:
-                        jaString = jaString.replace(match[0], match[2])
-
-                # Add String
-                stringList.append(jaString.strip())
-
-            # Pass 2
-            else:
-                # Get Text
-                if translatedList:
-                    # Grab and Pop
-                    translatedText = translatedList[0]
-                    translatedList.pop(0)
-
-                    # Set to None if empty list
-                    if len(translatedList) <= 0:
-                        translatedList = None
-
-                    # Textwrap & Other Text
-                    translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
-                    translatedText = translatedText.replace("\n", "\n\u3000")
-
-                    # Split the string into lines
-                    lines = translatedText.split("\n")
-
-                    # Add a backslash after every 3rd line
-                    j = 0
-                    while j < len(lines):
-                        if j == 4:
-                            lines[j - 1] = f"{lines[j-1]}\\"
-                            lines[j] = f"\n{lines[j]}"
-                        j += 1
-
-                    # Join the lines back into a single string
-                    translatedText = "\n".join(lines)
-
-                    # Remove Double Spaces
-                    translatedText = translatedText.replace("  ", " ")
-
-                    # Convert to Wide
-                    if CONVERTTOWIDE:
-                        translatedText = translatedText.translate(ascii_to_wide)
-
-                    # Fix Formatting
-                    translatedText = fixText(translatedText)
-
-                    # Set Data
-                    data[i] = data[i].replace(originalString, f"{translatedText}")
-                    save_progress_lines(data, filename)
-            i += 1
-
-        # Choices
-        elif "csel" in data[i] and translatedList != []:
-            choiceList = []
-            jaString = data[i]
-
-            choiceList = re.findall(r"\"(.*?)\"", jaString)
-            if len(choiceList) > 0:
-                # Translate
-                response = translateAI(choiceList, "This will be a dialogue option")
-                translatedTextList = response[0]
-                tokens[0] += response[1][0]
-                tokens[1] += response[1][1]
-
-                # Set Data
-                for j in range(len(translatedTextList)):
-                    # Convert to Wide
-                    if CONVERTTOWIDE:
-                        translatedText = translatedTextList[j].translate(ascii_to_wide)
-
-                    # Set
-                    data[i] = data[i].replace(choiceList[j], translatedText)
-                save_progress_lines(data, filename)
-            i += 1
-
-        # Nothing relevant. Skip Line.
-        else:
-            i += 1
-
-    # EOF
-    if len(stringList) > 0:
-        # Set Progress
-        pbar.total = len(stringList)
-        pbar.refresh()
-
-        # Translate
-        response = translateAI(stringList, "")
-        tokens[0] += response[1][0]
-        tokens[1] += response[1][1]
-        translatedList = response[0]
-
-        # Set Strings
-        if len(stringList) == len(translatedList):
-            translateOnscripter(data, pbar, filename, translatedList)
-
-        # Mismatch
-        else:
-            with LOCK:
-                if filename not in MISMATCH:
-                    MISMATCH.append(filename)
-    return tokens
-
-
-def fixText(translatedText):
-    # Add Break
-    translatedText = translatedText.replace('"', "'")
-    translatedText = f"\u3000{translatedText}\\"
-
-    # Unconvert Codes
-    matchList = re.findall(r"([＄].+?)[^\w]", translatedText)
-    if matchList:
-        for match in matchList:
-            translatedText = translatedText.replace(match, match.translate(wide_to_ascii))
-
-    # Unconvert Color Codes
-    matchList = re.findall(r"([＃][\w\d]{6})", translatedText)
-    if matchList:
-        for match in matchList:
-            translatedText = translatedText.replace(match, match.translate(wide_to_ascii))
-
-    # Unconvert Variables
-    matchList = re.findall(r"([％]\w.+?)[^\w＿]", translatedText)
-    if matchList:
-        for match in matchList:
-            translatedText = translatedText.replace(match, match.translate(wide_to_ascii))
-
-    # Unconvert Backslashes
-    matchList = re.findall(r"＼", translatedText)
-    if matchList:
-        for match in matchList:
-            translatedText = translatedText.replace(match, match.translate(wide_to_ascii))
-
-    return translatedText
-
-# Save some money and enter the character before translation
-def getSpeaker(speaker):
-    match speaker:
-        case "ファイン":
-            return ["Fine", [0, 0]]
-        case "":
-            return ["", [0, 0]]
-        case _:
-            # Find Speaker
-            for i in range(len(NAMESLIST)):
-                if speaker == NAMESLIST[i][0]:
-                    return [NAMESLIST[i][1], [0, 0]]
-
-            # Translate and Store Speaker
-            response = translateAI(
-                f"{speaker}",
-                "Reply with the " + LANGUAGE + " translation of the NPC name.",
-                False,
-            )
-            response[0] = response[0].title()
-            response[0] = response[0].replace("'S", "'s")
-            response[0] = response[0].replace("Speaker: ", "")
-
-            # Retry if name doesn't translate for some reason
-            if re.search(r"([a-zA-Z？?])", response[0]) == None:
-                response = translateAI(
-                    f"{speaker}",
-                    "Reply with the " + LANGUAGE + " translation of the NPC name.",
-                    False,
-                )
-                response[0] = response[0].title()
-                response[0] = response[0].replace("'S", "'s")
-
-            speakerList = [speaker, response[0]]
-            NAMESLIST.append(speakerList)
-            return response
-    return [speaker, [0, 0]]
-
-
-def translateAI(text, history, history_ctx=None):
-    """
-    Translate text using the shared translation utility.
-    This function maintains compatibility with existing code while using the new shared implementation.
-    """
-    return sharedtranslateAI(
-        text=text,
-        history=history,
-        config=TRANSLATION_CONFIG,
-        filename=FILENAME,
-        pbar=PBAR,
-        lock=LOCK,
-        mismatchList=MISMATCH
-    )
+# Libraries
+import json
+import os
+import re
+import util.dazedwrap as dazedwrap
+import threading
+import time
+import traceback
+import tiktoken
+from pathlib import Path
+from colorama import Fore
+from dotenv import load_dotenv
+from retry import retry
+from tqdm import tqdm
+from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
+import tempfile
+
+# Globals
+MODEL = os.getenv("model")
+TIMEOUT = int(os.getenv("timeout"))
+LANGUAGE = os.getenv("language").capitalize()
+from util.paths import PROMPT_PATH, VOCAB_PATH
+
+PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
+VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
+LOCK = threading.Lock()
+WIDTH = int(os.getenv("width"))
+LISTWIDTH = int(os.getenv("listWidth"))
+NOTEWIDTH = 70
+MAXHISTORY = 10
+ESTIMATE = ""
+TOKENS = [0, 0]
+NAMESLIST = []
+NAMES = False  # Output a list of all the character names found
+BRFLAG = False  # If the game uses <br> instead
+FIXTEXTWRAP = True  # Overwrites textwrap
+IGNORETLTEXT = False  # Ignores all translated text.
+MISMATCH = []  # Lists files that throw a mismatch error (Length of GPT list response is wrong)
+CONVERTTOWIDE = False # Default (False)
+
+# tqdm Globals
+BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
+POSITION = 0
+LEAVE = False
+PBAR = None
+FILENAME = None
+
+# Full Width
+ascii_to_wide = dict((i, chr(i + 0xFEE0)) for i in range(0x21, 0x7F))
+ascii_to_wide.update({0x20: "\u3000", 0x2D: "\u2212"})  # space and minus
+wide_to_ascii = dict((i, chr(i - 0xFEE0)) for i in range(0xFF01, 0xFF5F))
+wide_to_ascii.update({0x3000: " ", 0x2212: "-"})  # space and minus
+
+# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
+LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
+
+# Get pricing configuration based on the model
+PRICING_CONFIG = getPricingConfig(MODEL)
+INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
+OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
+BATCHSIZE = PRICING_CONFIG["batchSize"]
+FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
+
+# Initialize Translation Config
+TRANSLATION_CONFIG = TranslationConfig(
+    model=MODEL,
+    language=LANGUAGE,
+    prompt=PROMPT,
+    vocab=VOCAB,
+    langRegex=LANGREGEX,
+    batchSize=BATCHSIZE,
+    maxHistory=MAXHISTORY,
+    estimateMode=False  # Will be set dynamically based on ESTIMATE
+)
+LEAVE = False
+
+def handleOnscripter(filename, estimate):
+    global ESTIMATE, FILENAME
+    ESTIMATE = estimate
+    FILENAME = filename
+
+    if ESTIMATE:
+        start = time.time()
+        translatedData = openFiles(filename)
+
+        # Print Result
+        end = time.time()
+        tqdm.write(getResultString(translatedData, end - start, filename))
+        with LOCK:
+            TOKENS[0] += translatedData[1][0]
+            TOKENS[1] += translatedData[1][1]
+
+        # Print Total
+        totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
+
+        # Print any errors on maps
+        if len(MISMATCH) > 0:
+            return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
+        else:
+            return totalString
+
+    else:
+        try:
+            with open("translated/" + filename, "w", encoding="cp932", errors="ignore") as outFile:
+                start = time.time()
+                translatedData = openFiles(filename)
+
+                # Print Result
+                end = time.time()
+                outFile.writelines(translatedData[0])
+                tqdm.write(getResultString(translatedData, end - start, filename))
+                with LOCK:
+                    TOKENS[0] += translatedData[1][0]
+                    TOKENS[1] += translatedData[1][1]
+        except Exception:
+            traceback.print_exc()
+            return "Fail"
+
+    return getResultString(["", TOKENS, None], end - start, "TOTAL")
+
+
+def getResultString(translatedData, translationTime, filename):
+    # File Print String
+    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
+    totalTokenstring = (
+        Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
+        "[Output: "
+        + str(translatedData[1][1])
+        + "]" "[Cost: ${:,.4f}".format(cost)
+        + "]"
+    )
+    timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
+
+    if translatedData[2] == None:
+        # Success
+        return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
+
+    else:
+        # Fail
+        try:
+            raise translatedData[2]
+        except Exception as e:
+            traceback.print_exc()
+            errorString = str(e) + Fore.RED
+            return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
+
+
+def openFiles(filename):
+    with open("files/" + filename, "r", encoding="cp932") as readFile:
+        translatedData = parseOnscripter(readFile, filename)
+
+        # Delete lines marked for deletion
+        finalData = []
+        for line in translatedData[0]:
+            if line != "\\d\n":
+                finalData.append(line)
+        translatedData[0] = finalData
+
+    return translatedData
+
+
+def parseOnscripter(readFile, filename):
+    totalTokens = [0, 0]
+
+    # Read File into data
+    data = readFile.readlines()
+
+    # Create Progress Bar
+    with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
+        pbar.desc = filename
+
+        try:
+            result = translateOnscripter(data, pbar, filename, [])
+            totalTokens[0] += result[0]
+            totalTokens[1] += result[1]
+        except Exception as e:
+            traceback.print_exc()
+            return [data, totalTokens, e]
+    return [data, totalTokens, None]
+
+
+def save_progress_lines(lines, filename, encoding="cp932"):
+    """Atomically save current line-based translation progress."""
+    try:
+        if ESTIMATE:
+            return
+        os.makedirs("translated", exist_ok=True)
+        tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
+        try:
+            with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file:
+                tmp_file.writelines(lines)
+            os.replace(tmp_path, os.path.join("translated", filename))
+        finally:
+            if os.path.exists(tmp_path):
+                try:
+                    os.remove(tmp_path)
+                except OSError:
+                    pass
+    except Exception:
+        traceback.print_exc()
+
+
+def translateOnscripter(data, pbar, filename, translatedList):
+    stringList = []
+    currentGroup = []
+    tokens = [0, 0]
+    speaker = ""
+    voice = False
+    global LOCK, ESTIMATE, PBAR
+    PBAR = pbar
+    i = 0
+
+    # Dialogue
+    while i < len(data):
+        # Lines
+        regex = r"^([\u3000「（【[][^\n]+)"
+        match = re.search(regex, data[i])
+        if match != None and match.group(1) != "":
+            originalString = match.group(1)
+            # Pass 1
+            if translatedList == []:
+                # Grab Consecutive Strings
+                jaString = match.group(1)
+                while len(data) > i + 1 and re.match(regex, data[i + 1]):
+                    data[i] = ""
+                    i += 1
+                    jaString = f"{jaString} {data[i]}"
+
+                # Convert from Wide
+                jaString = jaString.translate(wide_to_ascii)
+
+                # Remove any textwrap and \u3000 and \
+                jaString = jaString.replace("\n", "")
+                jaString = jaString.replace("\u3000", "")
+                jaString = jaString.replace("\\", "")
+                jaString = jaString.replace("　＞", ")")
+                jaString = jaString.replace("＜　", "(")
+
+                # Remove Furigana
+                furiMatch = re.findall(r"({(.+?)\/(.+?)})", jaString)
+                if furiMatch:
+                    for match in furiMatch:
+                        jaString = jaString.replace(match[0], match[2])
+
+                # Add String
+                stringList.append(jaString.strip())
+
+            # Pass 2
+            else:
+                # Get Text
+                if translatedList:
+                    # Grab and Pop
+                    translatedText = translatedList[0]
+                    translatedList.pop(0)
+
+                    # Set to None if empty list
+                    if len(translatedList) <= 0:
+                        translatedList = None
+
+                    # Textwrap & Other Text
+                    translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
+                    translatedText = translatedText.replace("\n", "\n\u3000")
+
+                    # Split the string into lines
+                    lines = translatedText.split("\n")
+
+                    # Add a backslash after every 3rd line
+                    j = 0
+                    while j < len(lines):
+                        if j == 4:
+                            lines[j - 1] = f"{lines[j-1]}\\"
+                            lines[j] = f"\n{lines[j]}"
+                        j += 1
+
+                    # Join the lines back into a single string
+                    translatedText = "\n".join(lines)
+
+                    # Remove Double Spaces
+                    translatedText = translatedText.replace("  ", " ")
+
+                    # Convert to Wide
+                    if CONVERTTOWIDE:
+                        translatedText = translatedText.translate(ascii_to_wide)
+
+                    # Fix Formatting
+                    translatedText = fixText(translatedText)
+
+                    # Set Data
+                    data[i] = data[i].replace(originalString, f"{translatedText}")
+                    save_progress_lines(data, filename)
+            i += 1
+
+        # Choices
+        elif "csel" in data[i] and translatedList != []:
+            choiceList = []
+            jaString = data[i]
+
+            choiceList = re.findall(r"\"(.*?)\"", jaString)
+            if len(choiceList) > 0:
+                # Translate
+                response = translateAI(choiceList, "This will be a dialogue option")
+                translatedTextList = response[0]
+                tokens[0] += response[1][0]
+                tokens[1] += response[1][1]
+
+                # Set Data
+                for j in range(len(translatedTextList)):
+                    # Convert to Wide
+                    if CONVERTTOWIDE:
+                        translatedText = translatedTextList[j].translate(ascii_to_wide)
+
+                    # Set
+                    data[i] = data[i].replace(choiceList[j], translatedText)
+                save_progress_lines(data, filename)
+            i += 1
+
+        # Nothing relevant. Skip Line.
+        else:
+            i += 1
+
+    # EOF
+    if len(stringList) > 0:
+        # Set Progress
+        pbar.total = len(stringList)
+        pbar.refresh()
+
+        # Translate
+        response = translateAI(stringList, "")
+        tokens[0] += response[1][0]
+        tokens[1] += response[1][1]
+        translatedList = response[0]
+
+        # Set Strings
+        if len(stringList) == len(translatedList):
+            translateOnscripter(data, pbar, filename, translatedList)
+
+        # Mismatch
+        else:
+            with LOCK:
+                if filename not in MISMATCH:
+                    MISMATCH.append(filename)
+    return tokens
+
+
+def fixText(translatedText):
+    # Add Break
+    translatedText = translatedText.replace('"', "'")
+    translatedText = f"\u3000{translatedText}\\"
+
+    # Unconvert Codes
+    matchList = re.findall(r"([＄].+?)[^\w]", translatedText)
+    if matchList:
+        for match in matchList:
+            translatedText = translatedText.replace(match, match.translate(wide_to_ascii))
+
+    # Unconvert Color Codes
+    matchList = re.findall(r"([＃][\w\d]{6})", translatedText)
+    if matchList:
+        for match in matchList:
+            translatedText = translatedText.replace(match, match.translate(wide_to_ascii))
+
+    # Unconvert Variables
+    matchList = re.findall(r"([％]\w.+?)[^\w＿]", translatedText)
+    if matchList:
+        for match in matchList:
+            translatedText = translatedText.replace(match, match.translate(wide_to_ascii))
+
+    # Unconvert Backslashes
+    matchList = re.findall(r"＼", translatedText)
+    if matchList:
+        for match in matchList:
+            translatedText = translatedText.replace(match, match.translate(wide_to_ascii))
+
+    return translatedText
+
+# Save some money and enter the character before translation
+def getSpeaker(speaker):
+    match speaker:
+        case "ファイン":
+            return ["Fine", [0, 0]]
+        case "":
+            return ["", [0, 0]]
+        case _:
+            # Find Speaker
+            for i in range(len(NAMESLIST)):
+                if speaker == NAMESLIST[i][0]:
+                    return [NAMESLIST[i][1], [0, 0]]
+
+            # Translate and Store Speaker
+            response = translateAI(
+                f"{speaker}",
+                "Reply with the " + LANGUAGE + " translation of the NPC name.",
+                False,
+            )
+            response[0] = response[0].title()
+            response[0] = response[0].replace("'S", "'s")
+            response[0] = response[0].replace("Speaker: ", "")
+
+            # Retry if name doesn't translate for some reason
+            if re.search(r"([a-zA-Z？?])", response[0]) == None:
+                response = translateAI(
+                    f"{speaker}",
+                    "Reply with the " + LANGUAGE + " translation of the NPC name.",
+                    False,
+                )
+                response[0] = response[0].title()
+                response[0] = response[0].replace("'S", "'s")
+
+            speakerList = [speaker, response[0]]
+            NAMESLIST.append(speakerList)
+            return response
+    return [speaker, [0, 0]]
+
+
+def translateAI(text, history, history_ctx=None):
+    """
+    Translate text using the shared translation utility.
+    This function maintains compatibility with existing code while using the new shared implementation.
+    """
+    return sharedtranslateAI(
+        text=text,
+        history=history,
+        config=TRANSLATION_CONFIG,
+        filename=FILENAME,
+        pbar=PBAR,
+        lock=LOCK,
+        mismatchList=MISMATCH
+    )

modules/renpy.py

@@ -1,351 +1,353 @@
-# Libraries
-import json
-import os
-import re
-import util.dazedwrap as dazedwrap
-import threading
-import time
-import traceback
-import tiktoken
-from pathlib import Path
-from colorama import Fore
-from dotenv import load_dotenv
-from retry import retry
-from tqdm import tqdm
-from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost, getPricingConfig, calculateCost
-import tempfile
-
-# Globals
-MODEL = os.getenv("model")
-TIMEOUT = int(os.getenv("timeout"))
-LANGUAGE = os.getenv("language").capitalize()
-PROMPT = Path("prompt.txt").read_text(encoding="utf-8")
-VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
-LOCK = threading.Lock()
-WIDTH = int(os.getenv("width"))
-LISTWIDTH = int(os.getenv("listWidth"))
-NOTEWIDTH = 70
-MAXHISTORY = 10
-ESTIMATE = ""
-TOKENS = [0, 0]
-NAMESLIST = []
-NAMES = False  # Output a list of all the character names found
-BRFLAG = False  # If the game uses <br> instead
-FIXTEXTWRAP = True  # Overwrites textwrap
-IGNORETLTEXT = False  # Ignores all translated text.
-MISMATCH = []  # Lists files that throw a mismatch error (Length of GPT list response is wrong)
-FILENAME = None
-
-# tqdm Globals
-BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
-POSITION = 0
-LEAVE = False
-PBAR = None
-
-# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
-LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
-
-# Get pricing configuration based on the model
-PRICING_CONFIG = getPricingConfig(MODEL)
-INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
-OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
-BATCHSIZE = PRICING_CONFIG["batchSize"]
-FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
-
-# Initialize Translation Config
-TRANSLATION_CONFIG = TranslationConfig(
-    model=MODEL,
-    language=LANGUAGE,
-    prompt=PROMPT,
-    vocab=VOCAB,
-    langRegex=LANGREGEX,
-    batchSize=BATCHSIZE,
-    maxHistory=MAXHISTORY,
-    estimateMode=False  # Will be set dynamically based on ESTIMATE
-)
-LEAVE = False
-
-def handleRenpy(filename, estimate):
-    global ESTIMATE
-    global FILENAME
-    FILENAME = filename
-    ESTIMATE = estimate
-
-    if ESTIMATE:
-        start = time.time()
-        translatedData = openFiles(filename)
-
-        # Print Result
-        end = time.time()
-        tqdm.write(getResultString(translatedData, end - start, filename))
-        with LOCK:
-            TOKENS[0] += translatedData[1][0]
-            TOKENS[1] += translatedData[1][1]
-
-        # Print Total
-        totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
-
-        # Print any errors on maps
-        if len(MISMATCH) > 0:
-            return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
-        else:
-            return totalString
-
-    else:
-        try:
-            with open("translated/" + filename, "w", encoding="utf8", errors="ignore") as outFile:
-                start = time.time()
-                translatedData = openFiles(filename)
-
-                # Print Result
-                end = time.time()
-                outFile.writelines(translatedData[0])
-                tqdm.write(getResultString(translatedData, end - start, filename))
-                with LOCK:
-                    TOKENS[0] += translatedData[1][0]
-                    TOKENS[1] += translatedData[1][1]
-        except Exception:
-            traceback.print_exc()
-            return "Fail"
-
-    return getResultString(["", TOKENS, None], end - start, "TOTAL")
-
-
-def getResultString(translatedData, translationTime, filename):
-    # File Print String
-    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
-    totalTokenstring = (
-        Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
-        "[Output: "
-        + str(translatedData[1][1])
-        + "]" "[Cost: ${:,.4f}".format(cost)
-        + "]"
-    )
-    timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
-
-    if translatedData[2] == None:
-        # Success
-        return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
-
-    else:
-        # Fail
-        try:
-            raise translatedData[2]
-        except Exception as e:
-            traceback.print_exc()
-            errorString = str(e) + Fore.RED
-            return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
-
-
-def openFiles(filename):
-    with open("files/" + filename, "r", encoding="utf8") as readFile:
-        translatedData = parseRenpy(readFile, filename)
-
-        # Delete lines marked for deletion
-        finalData = []
-        for line in translatedData[0]:
-            if line != "\\d\n":
-                finalData.append(line)
-        translatedData[0] = finalData
-
-    return translatedData
-
-
-def parseRenpy(readFile, filename):
-    global PBAR
-    totalTokens = [0, 0]
-
-    # Read File into data
-    data = readFile.readlines()
-
-    # Create Progress Bar
-    with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
-        pbar.desc = filename
-        PBAR = pbar
-
-        try:
-            result = translateRenpy(data, filename, [])
-            totalTokens[0] += result[0]
-            totalTokens[1] += result[1]
-        except Exception as e:
-            traceback.print_exc()
-            return [data, totalTokens, e]
-    return [data, totalTokens, None]
-
-
-def save_progress_lines(lines, filename, encoding="utf-8"):
-    """Atomically save current line-based translation progress."""
-    try:
-        if ESTIMATE:
-            return
-        os.makedirs("translated", exist_ok=True)
-        tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
-        try:
-            with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file:
-                tmp_file.writelines(lines)
-            os.replace(tmp_path, os.path.join("translated", filename))
-        finally:
-            if os.path.exists(tmp_path):
-                try:
-                    os.remove(tmp_path)
-                except OSError:
-                    pass
-    except Exception:
-        traceback.print_exc()
-
-
-def translateRenpy(data, filename, translatedList):
-    stringList = []
-    currentGroup = []
-    tokens = [0, 0]
-    speaker = ""
-    voice = False
-    global LOCK, ESTIMATE, FILENAME, PBAR
-    i = 0
-
-    while i < len(data):
-        voice = False
-        speaker = ""
-        lineRegexNoSpeaker = r'^\s\s\s\s"(.*)"'
-        lineRegexSpeaker = r'^\s\s\s\s(.+?)\s"(.*)"'
-
-        # Grab Line
-        match = re.search(lineRegexSpeaker, data[i])
-        if match:
-            response = getSpeaker(match.group(1))
-            jaString = match.group(2)
-            speaker = response[0]
-            tokens[0] += response[1][0]
-            tokens[1] += response[1][1]
-        else:
-            match = re.search(lineRegexNoSpeaker, data[i])
-            if match:
-                jaString = match.group(1)
-
-        # Valid Line
-        if match and "voice" not in data[i]:
-            originalString = jaString
-            # Pass 1
-            if translatedList == []:
-                # Remove any textwrap
-                jaString = jaString.replace("\\n", " ")
-
-                # Add String
-                if speaker:
-                    stringList.append(f"[{speaker}]: {jaString.strip()}")
-                else:
-                    stringList.append(jaString.strip())
-
-            # Pass 2
-            else:
-                # Get Text
-                if translatedList:
-                    # Grab and Pop
-                    translatedText = translatedList[0]
-                    translatedList.pop(0)
-
-                    # Set to None if empty list
-                    if len(translatedList) <= 0:
-                        translatedList = None
-
-                    # Remove speaker
-                    if speaker != "":
-                        matchSpeakerList = re.findall(r"^\[?(.+?)\]?\s?[|:]\s?", translatedText)
-                        translatedText = re.sub(r"^\[?(.+?)\]?\s?[|:]\s?", "", translatedText)
-
-                    # Escape Quotes
-                    translatedText = re.sub(r'[\\]*(")', '\\"', translatedText)
-                    translatedText = re.sub(r"[\\]*(')", "\\'", translatedText)
-
-                    # Textwrap
-                    translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
-                    translatedText = translatedText.replace("\n", "\\n")
-
-                    # Set Data
-                    data[i] = data[i].replace(originalString, translatedText)
-                    save_progress_lines(data, filename)
-            i += 1
-        else:
-            i += 1
-
-    # EOF
-    if len(stringList) > 0:
-        # Set Progress
-        PBAR.total = len(stringList)
-        PBAR.refresh()
-
-        # Translate
-        response = translateAI(stringList, "Reply with the English TL of the NPC Name")
-        tokens[0] += response[1][0]
-        tokens[1] += response[1][1]
-        translatedList = response[0]
-
-        # Set Strings
-        if len(stringList) == len(translatedList):
-            translateRenpy(data, filename, translatedList)
-
-        # Mismatch
-        else:
-            with LOCK:
-                if FILENAME not in MISMATCH:
-                    MISMATCH.append(FILENAME)
-    return tokens
-
-# Save some money and enter the character before translation
-def getSpeaker(speaker):
-    match speaker:
-        case "ファイン":
-            return ["Fine", [0, 0]]
-        case "":
-            return ["", [0, 0]]
-        case _:
-            # Find Speaker
-            for i in range(len(NAMESLIST)):
-                if speaker == NAMESLIST[i][0]:
-                    return [NAMESLIST[i][1], [0, 0]]
-
-            # Translate and Store Speaker
-            response = translateAI(
-                f"{speaker}",
-                "Reply with the " + LANGUAGE + " translation of the NPC name.",
-                False,
-            )
-            response[0] = response[0].title()
-            response[0] = response[0].replace("'S", "'s")
-            response[0] = response[0].replace("Speaker: ", "")
-
-            # Retry if name doesn't translate for some reason
-            if re.search(r"([a-zA-Z？?])", response[0]) == None:
-                response = translateAI(
-                    f"{speaker}",
-                    "Reply with the " + LANGUAGE + " translation of the NPC name.",
-                    False,
-                )
-                response[0] = response[0].title()
-                response[0] = response[0].replace("'S", "'s")
-
-            speakerList = [speaker, response[0]]
-            NAMESLIST.append(speakerList)
-            return response
-    return [speaker, [0, 0]]
-
-def translateAI(text, history, history_ctx=None):
-    """
-    Legacy wrapper function for the new shared translation utility.
-    This maintains compatibility with existing code while using the new shared implementation.
-    """
-    global PBAR, MISMATCH, FILENAME
-    
-    # Update config estimate mode based on global ESTIMATE
-    TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
-    
-    # Call the new shared translation function
-    return sharedtranslateAI(
-        text=text,
-        history=history,
-        config=TRANSLATION_CONFIG,
-        filename=FILENAME,
-        pbar=PBAR,
-        lock=LOCK,
-        mismatchList=MISMATCH
-    )
+# Libraries
+import json
+import os
+import re
+import util.dazedwrap as dazedwrap
+import threading
+import time
+import traceback
+import tiktoken
+from pathlib import Path
+from colorama import Fore
+from dotenv import load_dotenv
+from retry import retry
+from tqdm import tqdm
+from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost, getPricingConfig, calculateCost
+import tempfile
+
+# Globals
+MODEL = os.getenv("model")
+TIMEOUT = int(os.getenv("timeout"))
+LANGUAGE = os.getenv("language").capitalize()
+from util.paths import PROMPT_PATH, VOCAB_PATH
+
+PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
+VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
+LOCK = threading.Lock()
+WIDTH = int(os.getenv("width"))
+LISTWIDTH = int(os.getenv("listWidth"))
+NOTEWIDTH = 70
+MAXHISTORY = 10
+ESTIMATE = ""
+TOKENS = [0, 0]
+NAMESLIST = []
+NAMES = False  # Output a list of all the character names found
+BRFLAG = False  # If the game uses <br> instead
+FIXTEXTWRAP = True  # Overwrites textwrap
+IGNORETLTEXT = False  # Ignores all translated text.
+MISMATCH = []  # Lists files that throw a mismatch error (Length of GPT list response is wrong)
+FILENAME = None
+
+# tqdm Globals
+BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
+POSITION = 0
+LEAVE = False
+PBAR = None
+
+# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
+LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
+
+# Get pricing configuration based on the model
+PRICING_CONFIG = getPricingConfig(MODEL)
+INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
+OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
+BATCHSIZE = PRICING_CONFIG["batchSize"]
+FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
+
+# Initialize Translation Config
+TRANSLATION_CONFIG = TranslationConfig(
+    model=MODEL,
+    language=LANGUAGE,
+    prompt=PROMPT,
+    vocab=VOCAB,
+    langRegex=LANGREGEX,
+    batchSize=BATCHSIZE,
+    maxHistory=MAXHISTORY,
+    estimateMode=False  # Will be set dynamically based on ESTIMATE
+)
+LEAVE = False
+
+def handleRenpy(filename, estimate):
+    global ESTIMATE
+    global FILENAME
+    FILENAME = filename
+    ESTIMATE = estimate
+
+    if ESTIMATE:
+        start = time.time()
+        translatedData = openFiles(filename)
+
+        # Print Result
+        end = time.time()
+        tqdm.write(getResultString(translatedData, end - start, filename))
+        with LOCK:
+            TOKENS[0] += translatedData[1][0]
+            TOKENS[1] += translatedData[1][1]
+
+        # Print Total
+        totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
+
+        # Print any errors on maps
+        if len(MISMATCH) > 0:
+            return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
+        else:
+            return totalString
+
+    else:
+        try:
+            with open("translated/" + filename, "w", encoding="utf8", errors="ignore") as outFile:
+                start = time.time()
+                translatedData = openFiles(filename)
+
+                # Print Result
+                end = time.time()
+                outFile.writelines(translatedData[0])
+                tqdm.write(getResultString(translatedData, end - start, filename))
+                with LOCK:
+                    TOKENS[0] += translatedData[1][0]
+                    TOKENS[1] += translatedData[1][1]
+        except Exception:
+            traceback.print_exc()
+            return "Fail"
+
+    return getResultString(["", TOKENS, None], end - start, "TOTAL")
+
+
+def getResultString(translatedData, translationTime, filename):
+    # File Print String
+    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
+    totalTokenstring = (
+        Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
+        "[Output: "
+        + str(translatedData[1][1])
+        + "]" "[Cost: ${:,.4f}".format(cost)
+        + "]"
+    )
+    timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
+
+    if translatedData[2] == None:
+        # Success
+        return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
+
+    else:
+        # Fail
+        try:
+            raise translatedData[2]
+        except Exception as e:
+            traceback.print_exc()
+            errorString = str(e) + Fore.RED
+            return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
+
+
+def openFiles(filename):
+    with open("files/" + filename, "r", encoding="utf8") as readFile:
+        translatedData = parseRenpy(readFile, filename)
+
+        # Delete lines marked for deletion
+        finalData = []
+        for line in translatedData[0]:
+            if line != "\\d\n":
+                finalData.append(line)
+        translatedData[0] = finalData
+
+    return translatedData
+
+
+def parseRenpy(readFile, filename):
+    global PBAR
+    totalTokens = [0, 0]
+
+    # Read File into data
+    data = readFile.readlines()
+
+    # Create Progress Bar
+    with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
+        pbar.desc = filename
+        PBAR = pbar
+
+        try:
+            result = translateRenpy(data, filename, [])
+            totalTokens[0] += result[0]
+            totalTokens[1] += result[1]
+        except Exception as e:
+            traceback.print_exc()
+            return [data, totalTokens, e]
+    return [data, totalTokens, None]
+
+
+def save_progress_lines(lines, filename, encoding="utf-8"):
+    """Atomically save current line-based translation progress."""
+    try:
+        if ESTIMATE:
+            return
+        os.makedirs("translated", exist_ok=True)
+        tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
+        try:
+            with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file:
+                tmp_file.writelines(lines)
+            os.replace(tmp_path, os.path.join("translated", filename))
+        finally:
+            if os.path.exists(tmp_path):
+                try:
+                    os.remove(tmp_path)
+                except OSError:
+                    pass
+    except Exception:
+        traceback.print_exc()
+
+
+def translateRenpy(data, filename, translatedList):
+    stringList = []
+    currentGroup = []
+    tokens = [0, 0]
+    speaker = ""
+    voice = False
+    global LOCK, ESTIMATE, FILENAME, PBAR
+    i = 0
+
+    while i < len(data):
+        voice = False
+        speaker = ""
+        lineRegexNoSpeaker = r'^\s\s\s\s"(.*)"'
+        lineRegexSpeaker = r'^\s\s\s\s(.+?)\s"(.*)"'
+
+        # Grab Line
+        match = re.search(lineRegexSpeaker, data[i])
+        if match:
+            response = getSpeaker(match.group(1))
+            jaString = match.group(2)
+            speaker = response[0]
+            tokens[0] += response[1][0]
+            tokens[1] += response[1][1]
+        else:
+            match = re.search(lineRegexNoSpeaker, data[i])
+            if match:
+                jaString = match.group(1)
+
+        # Valid Line
+        if match and "voice" not in data[i]:
+            originalString = jaString
+            # Pass 1
+            if translatedList == []:
+                # Remove any textwrap
+                jaString = jaString.replace("\\n", " ")
+
+                # Add String
+                if speaker:
+                    stringList.append(f"[{speaker}]: {jaString.strip()}")
+                else:
+                    stringList.append(jaString.strip())
+
+            # Pass 2
+            else:
+                # Get Text
+                if translatedList:
+                    # Grab and Pop
+                    translatedText = translatedList[0]
+                    translatedList.pop(0)
+
+                    # Set to None if empty list
+                    if len(translatedList) <= 0:
+                        translatedList = None
+
+                    # Remove speaker
+                    if speaker != "":
+                        matchSpeakerList = re.findall(r"^\[?(.+?)\]?\s?[|:]\s?", translatedText)
+                        translatedText = re.sub(r"^\[?(.+?)\]?\s?[|:]\s?", "", translatedText)
+
+                    # Escape Quotes
+                    translatedText = re.sub(r'[\\]*(")', '\\"', translatedText)
+                    translatedText = re.sub(r"[\\]*(')", "\\'", translatedText)
+
+                    # Textwrap
+                    translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
+                    translatedText = translatedText.replace("\n", "\\n")
+
+                    # Set Data
+                    data[i] = data[i].replace(originalString, translatedText)
+                    save_progress_lines(data, filename)
+            i += 1
+        else:
+            i += 1
+
+    # EOF
+    if len(stringList) > 0:
+        # Set Progress
+        PBAR.total = len(stringList)
+        PBAR.refresh()
+
+        # Translate
+        response = translateAI(stringList, "Reply with the English TL of the NPC Name")
+        tokens[0] += response[1][0]
+        tokens[1] += response[1][1]
+        translatedList = response[0]
+
+        # Set Strings
+        if len(stringList) == len(translatedList):
+            translateRenpy(data, filename, translatedList)
+
+        # Mismatch
+        else:
+            with LOCK:
+                if FILENAME not in MISMATCH:
+                    MISMATCH.append(FILENAME)
+    return tokens
+
+# Save some money and enter the character before translation
+def getSpeaker(speaker):
+    match speaker:
+        case "ファイン":
+            return ["Fine", [0, 0]]
+        case "":
+            return ["", [0, 0]]
+        case _:
+            # Find Speaker
+            for i in range(len(NAMESLIST)):
+                if speaker == NAMESLIST[i][0]:
+                    return [NAMESLIST[i][1], [0, 0]]
+
+            # Translate and Store Speaker
+            response = translateAI(
+                f"{speaker}",
+                "Reply with the " + LANGUAGE + " translation of the NPC name.",
+                False,
+            )
+            response[0] = response[0].title()
+            response[0] = response[0].replace("'S", "'s")
+            response[0] = response[0].replace("Speaker: ", "")
+
+            # Retry if name doesn't translate for some reason
+            if re.search(r"([a-zA-Z？?])", response[0]) == None:
+                response = translateAI(
+                    f"{speaker}",
+                    "Reply with the " + LANGUAGE + " translation of the NPC name.",
+                    False,
+                )
+                response[0] = response[0].title()
+                response[0] = response[0].replace("'S", "'s")
+
+            speakerList = [speaker, response[0]]
+            NAMESLIST.append(speakerList)
+            return response
+    return [speaker, [0, 0]]
+
+def translateAI(text, history, history_ctx=None):
+    """
+    Legacy wrapper function for the new shared translation utility.
+    This maintains compatibility with existing code while using the new shared implementation.
+    """
+    global PBAR, MISMATCH, FILENAME
+    
+    # Update config estimate mode based on global ESTIMATE
+    TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
+    
+    # Call the new shared translation function
+    return sharedtranslateAI(
+        text=text,
+        history=history,
+        config=TRANSLATION_CONFIG,
+        filename=FILENAME,
+        pbar=PBAR,
+        lock=LOCK,
+        mismatchList=MISMATCH
+    )

modules/rpgmakermvmz.py

@@ -19,8 +19,10 @@ from util.translation import TranslationConfig, translateAI as sharedtranslateAI
 MODEL = os.getenv("model")
 TIMEOUT = int(os.getenv("timeout"))
 LANGUAGE = os.getenv("language").capitalize()
-PROMPT = Path("prompt.txt").read_text(encoding="utf-8")
-VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
+from util.paths import PROMPT_PATH, VOCAB_PATH
+
+PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
+VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
 LOCK = threading.Lock()
 THREAD_CTX = threading.local()
 WIDTH = int(os.getenv("width"))
@@ -803,7 +805,7 @@ def update_vocab_section(category: str, pairs: list[tuple[str, str]]):
     The existing section is replaced entirely; other sections are preserved.
     """
     try:
-        vocab_path = Path("vocab.txt")
+        vocab_path = VOCAB_PATH
 
         # Helper: normalized comparison to detect no-op translations
         def _norm(s: str) -> str:
@@ -5071,7 +5073,7 @@ def finalizeSpeakerParse():
                         _speakerCache[orig] = norm
                         NAMESLIST.append([orig, norm])
 
-        vocab_path = Path("vocab.txt")
+        vocab_path = VOCAB_PATH
         if not vocab_path.exists():
             return
         content = vocab_path.read_text(encoding="utf-8")

modules/text.py

@@ -1,363 +1,365 @@
-# Libraries
-import json
-import os
-import re
-import util.dazedwrap as dazedwrap
-import threading
-import time
-import traceback
-import tiktoken
-from pathlib import Path
-from colorama import Fore
-from dotenv import load_dotenv
-from retry import retry
-from tqdm import tqdm
-from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost, getPricingConfig, calculateCost
-import tempfile
-
-# Globals
-MODEL = os.getenv("model")
-TIMEOUT = int(os.getenv("timeout"))
-LANGUAGE = os.getenv("language").capitalize()
-PROMPT = Path("prompt.txt").read_text(encoding="utf-8")
-VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
-LOCK = threading.Lock()
-WIDTH = int(os.getenv("width"))
-LISTWIDTH = int(os.getenv("listWidth"))
-NOTEWIDTH = 70
-MAXHISTORY = 10
-ESTIMATE = ""
-TOKENS = [0, 0]
-NAMESLIST = []
-NAMES = False  # Output a list of all the character names found
-BRFLAG = False  # If the game uses <br> instead
-FIXTEXTWRAP = True  # Overwrites textwrap
-IGNORETLTEXT = False  # Ignores all translated text.
-MISMATCH = []  # Lists files that throw a mismatch error (Length of GPT list response is wrong)
-FILENAME = None
-
-# tqdm Globals
-BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
-POSITION = 0
-LEAVE = False
-PBAR = None
-
-# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
-LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
-
-# Get pricing configuration based on the model
-PRICING_CONFIG = getPricingConfig(MODEL)
-INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
-OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
-BATCHSIZE = PRICING_CONFIG["batchSize"]
-FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
-
-# Initialize Translation Config
-TRANSLATION_CONFIG = TranslationConfig(
-    model=MODEL,
-    language=LANGUAGE,
-    prompt=PROMPT,
-    vocab=VOCAB,
-    langRegex=LANGREGEX,
-    batchSize=BATCHSIZE,
-    maxHistory=MAXHISTORY,
-    estimateMode=False  # Will be set dynamically based on ESTIMATE
-)
-LEAVE = False
-
-def handleText(filename, estimate):
-    global ESTIMATE, TOKENS, FILENAME
-    ESTIMATE = estimate
-    FILENAME = filename
-
-    # Translate
-    start = time.time()
-    translatedData = openFiles(filename)
-
-    # Translate
-    if not estimate:
-        try:
-            with open("translated/" + filename, "w", encoding="utf-8", newline="\n") as outFile:
-                outFile.writelines(translatedData[0])
-        except Exception:
-            traceback.print_exc()
-            return "Fail"
-
-    # Print File
-    end = time.time()
-    tqdm.write(getResultString(translatedData, end - start, filename))
-    with LOCK:
-        TOKENS[0] += translatedData[1][0]
-        TOKENS[1] += translatedData[1][1]
-
-    # Print Total
-    totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
-
-    # Print any errors on maps
-    if len(MISMATCH) > 0:
-        return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
-    else:
-        return totalString
-
-
-def getResultString(translatedData, translationTime, filename):
-    # File Print String
-    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
-    totalTokenstring = (
-        Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
-        "[Output: "
-        + str(translatedData[1][1])
-        + "]" "[Cost: ${:,.4f}".format(cost)
-        + "]"
-    )
-    timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
-
-    if translatedData[2] == None:
-        # Success
-        return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
-
-    else:
-        # Fail
-        try:
-            raise translatedData[2]
-        except Exception as e:
-            traceback.print_exc()
-            errorString = str(e) + Fore.RED
-            return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
-
-
-def openFiles(filename):
-    with open("files/" + filename, "r", encoding="utf8") as readFile:
-        translatedData = parseText(readFile, filename)
-
-        # Delete lines marked for deletion
-        finalData = []
-        for line in translatedData[0]:
-            if line != "\\d\n":
-                finalData.append(line)
-        translatedData[0] = finalData
-
-    return translatedData
-
-
-def parseText(readFile, filename):
-    global PBAR
-    totalTokens = [0, 0]
-
-    # Read File into data
-    data = readFile.readlines()
-
-    # Create Progress Bar
-    with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
-        pbar.desc = filename
-        PBAR = pbar
-
-        try:
-            result = translateTxt(data, filename, [])
-            totalTokens[0] += result[0]
-            totalTokens[1] += result[1]
-        except Exception as e:
-            traceback.print_exc()
-            return [data, totalTokens, e]
-    return [data, totalTokens, None]
-
-
-def save_progress_lines(lines, filename, encoding="utf-8"):
-    """Atomically save current line-based translation progress."""
-    try:
-        if ESTIMATE:
-            return
-        os.makedirs("translated", exist_ok=True)
-        tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
-        try:
-            with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file:
-                tmp_file.writelines(lines)
-            os.replace(tmp_path, os.path.join("translated", filename))
-        finally:
-            if os.path.exists(tmp_path):
-                try:
-                    os.remove(tmp_path)
-                except OSError:
-                    pass
-    except Exception:
-        traceback.print_exc()
-
-
-def saveCheckLines(lines, filename, tokens=None, encoding="utf-8"):
-    """Save progress only when tokens indicate work or when explicitly called after a mutation."""
-    try:
-        if tokens is not None:
-            if not (isinstance(tokens, (list, tuple)) and len(tokens) >= 2 and (tokens[0] or tokens[1])):
-                return
-        save_progress_lines(lines, filename, encoding=encoding)
-    except Exception:
-        traceback.print_exc()
-
-
-def translateTxt(data, filename, translatedList):
-    if translatedList:
-        stringList = translatedList[0]
-        choiceList = translatedList[1]
-    else:
-        stringList = []
-        choiceList = []
-    tokens = [0, 0]
-    speaker = ""
-    global LOCK, ESTIMATE, FILENAME, PBAR, MISMATCH
-    i = 0
-
-    # Regex
-    lineTextRegex = r"(?:^\[.+?\]:)?(.+)"
-    speakerTextRegex = r"^\[(.+?)\]"
-
-    while i < len(data):
-        # Speaker
-        match = re.search(speakerTextRegex, data[i])
-        if match:
-            # Get Speaker
-            speakerData = getSpeaker(match.group(1))
-            speaker = speakerData[0]
-            tokens[0] += speakerData[1][0]
-            tokens[1] += speakerData[1][1]
-            data[i] = data[i].replace(match.group(1), speaker)
-
-        # Dialogue
-        match = re.search(lineTextRegex, data[i])
-        jaString = None
-        if match:
-            # Set String
-            jaString = match.group(1)
-
-            # Pass 1
-            if not translatedList:
-                # Strip Spaces
-                jaString = jaString.strip()
-
-                if jaString:
-                    if speaker:
-                        stringList.append(f"[{speaker}]: {jaString}")
-                    else:
-                        stringList.append(jaString)
-
-            # Pass 2
-            else:
-                # Get Text
-                if stringList:
-                    # Grab and Pop
-                    translatedText = stringList[0]
-                    stringList.pop(0)
-
-                    # Set to None if empty list
-                    if len(stringList) <= 0:
-                        stringList = None
-
-                    # Remove speaker
-                    translatedText = re.sub(r"(^\[.+?\]\s?[|:]\s?)", "", translatedText)
-
-                    # # Textwrap
-                    # translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
-                    # translatedText = translatedText.replace("\n", "\\n")
-
-                    # Set Data
-                    data[i] = f"{translatedText}\n"
-                    saveCheckLines(data, filename)
-
-            i += 1
-        else:
-            i += 1
-
-    # EOF
-    if not translatedList:
-        stringListTL = []
-        choiceListTL = []
-
-        # String List
-        if stringList:
-            PBAR.total = len(stringList)
-            PBAR.refresh()
-            response = translateAI(stringList, "Reply with the English Translation")
-            tokens[0] += response[1][0]
-            tokens[1] += response[1][1]
-            stringListTL = response[0]
-
-            if len(stringList) != len(stringListTL):
-                # Mismatch
-                with LOCK:
-                    if FILENAME not in MISMATCH:
-                        MISMATCH.append(FILENAME)
-
-        # Choice List
-        if choiceList:
-            response = translateAI(choiceList, "Reply with the English TL of the Dialogue Choice")
-            tokens[0] += response[1][0]
-            tokens[1] += response[1][1]
-            choiceListTL = response[0]
-
-            if len(choiceList) != len(choiceListTL):
-                # Mismatch
-                with LOCK:
-                    if FILENAME not in MISMATCH:
-                        MISMATCH.append(FILENAME)
-
-        # Set Strings
-        translateTxt(data, filename, [stringListTL, choiceListTL])
-    return tokens
-
-# Save some money and enter the character before translation
-def getSpeaker(speaker):
-    match speaker:
-        case "ファイン":
-            return ["Fine", [0, 0]]
-        case "":
-            return ["", [0, 0]]
-        case _:
-            # Find Speaker
-            for i in range(len(NAMESLIST)):
-                if speaker == NAMESLIST[i][0]:
-                    return [NAMESLIST[i][1], [0, 0]]
-
-            # Translate and Store Speaker
-            response = translateAI(
-                f"{speaker}",
-                "Reply with the " + LANGUAGE + " translation of the NPC name.",
-                False,
-            )
-            response[0] = response[0].title()
-            response[0] = response[0].replace("'S", "'s")
-            response[0] = response[0].replace("Speaker: ", "")
-
-            # Retry if name doesn't translate for some reason
-            if re.search(r"([a-zA-Z？?])", response[0]) == None:
-                response = translateAI(
-                    f"{speaker}",
-                    "Reply with the " + LANGUAGE + " translation of the NPC name.",
-                    False,
-                )
-                response[0] = response[0].title()
-                response[0] = response[0].replace("'S", "'s")
-
-            speakerList = [speaker, response[0]]
-            NAMESLIST.append(speakerList)
-            return response
-    return [speaker, [0, 0]]
-
-def translateAI(text, history, history_ctx=None):
-    """
-    Legacy wrapper function for the new shared translation utility.
-    This maintains compatibility with existing code while using the new shared implementation.
-    """
-    global PBAR, MISMATCH, FILENAME
-    
-    # Update config estimate mode based on global ESTIMATE
-    TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
-    
-    # Call the new shared translation function
-    return sharedtranslateAI(
-        text=text,
-        history=history,
-        config=TRANSLATION_CONFIG,
-        filename=FILENAME,
-        pbar=PBAR,
-        lock=LOCK,
-        mismatchList=MISMATCH
+# Libraries
+import json
+import os
+import re
+import util.dazedwrap as dazedwrap
+import threading
+import time
+import traceback
+import tiktoken
+from pathlib import Path
+from colorama import Fore
+from dotenv import load_dotenv
+from retry import retry
+from tqdm import tqdm
+from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost, getPricingConfig, calculateCost
+import tempfile
+
+# Globals
+MODEL = os.getenv("model")
+TIMEOUT = int(os.getenv("timeout"))
+LANGUAGE = os.getenv("language").capitalize()
+from util.paths import PROMPT_PATH, VOCAB_PATH
+
+PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
+VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
+LOCK = threading.Lock()
+WIDTH = int(os.getenv("width"))
+LISTWIDTH = int(os.getenv("listWidth"))
+NOTEWIDTH = 70
+MAXHISTORY = 10
+ESTIMATE = ""
+TOKENS = [0, 0]
+NAMESLIST = []
+NAMES = False  # Output a list of all the character names found
+BRFLAG = False  # If the game uses <br> instead
+FIXTEXTWRAP = True  # Overwrites textwrap
+IGNORETLTEXT = False  # Ignores all translated text.
+MISMATCH = []  # Lists files that throw a mismatch error (Length of GPT list response is wrong)
+FILENAME = None
+
+# tqdm Globals
+BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
+POSITION = 0
+LEAVE = False
+PBAR = None
+
+# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
+LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
+
+# Get pricing configuration based on the model
+PRICING_CONFIG = getPricingConfig(MODEL)
+INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
+OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
+BATCHSIZE = PRICING_CONFIG["batchSize"]
+FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
+
+# Initialize Translation Config
+TRANSLATION_CONFIG = TranslationConfig(
+    model=MODEL,
+    language=LANGUAGE,
+    prompt=PROMPT,
+    vocab=VOCAB,
+    langRegex=LANGREGEX,
+    batchSize=BATCHSIZE,
+    maxHistory=MAXHISTORY,
+    estimateMode=False  # Will be set dynamically based on ESTIMATE
+)
+LEAVE = False
+
+def handleText(filename, estimate):
+    global ESTIMATE, TOKENS, FILENAME
+    ESTIMATE = estimate
+    FILENAME = filename
+
+    # Translate
+    start = time.time()
+    translatedData = openFiles(filename)
+
+    # Translate
+    if not estimate:
+        try:
+            with open("translated/" + filename, "w", encoding="utf-8", newline="\n") as outFile:
+                outFile.writelines(translatedData[0])
+        except Exception:
+            traceback.print_exc()
+            return "Fail"
+
+    # Print File
+    end = time.time()
+    tqdm.write(getResultString(translatedData, end - start, filename))
+    with LOCK:
+        TOKENS[0] += translatedData[1][0]
+        TOKENS[1] += translatedData[1][1]
+
+    # Print Total
+    totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
+
+    # Print any errors on maps
+    if len(MISMATCH) > 0:
+        return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
+    else:
+        return totalString
+
+
+def getResultString(translatedData, translationTime, filename):
+    # File Print String
+    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
+    totalTokenstring = (
+        Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
+        "[Output: "
+        + str(translatedData[1][1])
+        + "]" "[Cost: ${:,.4f}".format(cost)
+        + "]"
+    )
+    timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
+
+    if translatedData[2] == None:
+        # Success
+        return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
+
+    else:
+        # Fail
+        try:
+            raise translatedData[2]
+        except Exception as e:
+            traceback.print_exc()
+            errorString = str(e) + Fore.RED
+            return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
+
+
+def openFiles(filename):
+    with open("files/" + filename, "r", encoding="utf8") as readFile:
+        translatedData = parseText(readFile, filename)
+
+        # Delete lines marked for deletion
+        finalData = []
+        for line in translatedData[0]:
+            if line != "\\d\n":
+                finalData.append(line)
+        translatedData[0] = finalData
+
+    return translatedData
+
+
+def parseText(readFile, filename):
+    global PBAR
+    totalTokens = [0, 0]
+
+    # Read File into data
+    data = readFile.readlines()
+
+    # Create Progress Bar
+    with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
+        pbar.desc = filename
+        PBAR = pbar
+
+        try:
+            result = translateTxt(data, filename, [])
+            totalTokens[0] += result[0]
+            totalTokens[1] += result[1]
+        except Exception as e:
+            traceback.print_exc()
+            return [data, totalTokens, e]
+    return [data, totalTokens, None]
+
+
+def save_progress_lines(lines, filename, encoding="utf-8"):
+    """Atomically save current line-based translation progress."""
+    try:
+        if ESTIMATE:
+            return
+        os.makedirs("translated", exist_ok=True)
+        tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
+        try:
+            with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file:
+                tmp_file.writelines(lines)
+            os.replace(tmp_path, os.path.join("translated", filename))
+        finally:
+            if os.path.exists(tmp_path):
+                try:
+                    os.remove(tmp_path)
+                except OSError:
+                    pass
+    except Exception:
+        traceback.print_exc()
+
+
+def saveCheckLines(lines, filename, tokens=None, encoding="utf-8"):
+    """Save progress only when tokens indicate work or when explicitly called after a mutation."""
+    try:
+        if tokens is not None:
+            if not (isinstance(tokens, (list, tuple)) and len(tokens) >= 2 and (tokens[0] or tokens[1])):
+                return
+        save_progress_lines(lines, filename, encoding=encoding)
+    except Exception:
+        traceback.print_exc()
+
+
+def translateTxt(data, filename, translatedList):
+    if translatedList:
+        stringList = translatedList[0]
+        choiceList = translatedList[1]
+    else:
+        stringList = []
+        choiceList = []
+    tokens = [0, 0]
+    speaker = ""
+    global LOCK, ESTIMATE, FILENAME, PBAR, MISMATCH
+    i = 0
+
+    # Regex
+    lineTextRegex = r"(?:^\[.+?\]:)?(.+)"
+    speakerTextRegex = r"^\[(.+?)\]"
+
+    while i < len(data):
+        # Speaker
+        match = re.search(speakerTextRegex, data[i])
+        if match:
+            # Get Speaker
+            speakerData = getSpeaker(match.group(1))
+            speaker = speakerData[0]
+            tokens[0] += speakerData[1][0]
+            tokens[1] += speakerData[1][1]
+            data[i] = data[i].replace(match.group(1), speaker)
+
+        # Dialogue
+        match = re.search(lineTextRegex, data[i])
+        jaString = None
+        if match:
+            # Set String
+            jaString = match.group(1)
+
+            # Pass 1
+            if not translatedList:
+                # Strip Spaces
+                jaString = jaString.strip()
+
+                if jaString:
+                    if speaker:
+                        stringList.append(f"[{speaker}]: {jaString}")
+                    else:
+                        stringList.append(jaString)
+
+            # Pass 2
+            else:
+                # Get Text
+                if stringList:
+                    # Grab and Pop
+                    translatedText = stringList[0]
+                    stringList.pop(0)
+
+                    # Set to None if empty list
+                    if len(stringList) <= 0:
+                        stringList = None
+
+                    # Remove speaker
+                    translatedText = re.sub(r"(^\[.+?\]\s?[|:]\s?)", "", translatedText)
+
+                    # # Textwrap
+                    # translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
+                    # translatedText = translatedText.replace("\n", "\\n")
+
+                    # Set Data
+                    data[i] = f"{translatedText}\n"
+                    saveCheckLines(data, filename)
+
+            i += 1
+        else:
+            i += 1
+
+    # EOF
+    if not translatedList:
+        stringListTL = []
+        choiceListTL = []
+
+        # String List
+        if stringList:
+            PBAR.total = len(stringList)
+            PBAR.refresh()
+            response = translateAI(stringList, "Reply with the English Translation")
+            tokens[0] += response[1][0]
+            tokens[1] += response[1][1]
+            stringListTL = response[0]
+
+            if len(stringList) != len(stringListTL):
+                # Mismatch
+                with LOCK:
+                    if FILENAME not in MISMATCH:
+                        MISMATCH.append(FILENAME)
+
+        # Choice List
+        if choiceList:
+            response = translateAI(choiceList, "Reply with the English TL of the Dialogue Choice")
+            tokens[0] += response[1][0]
+            tokens[1] += response[1][1]
+            choiceListTL = response[0]
+
+            if len(choiceList) != len(choiceListTL):
+                # Mismatch
+                with LOCK:
+                    if FILENAME not in MISMATCH:
+                        MISMATCH.append(FILENAME)
+
+        # Set Strings
+        translateTxt(data, filename, [stringListTL, choiceListTL])
+    return tokens
+
+# Save some money and enter the character before translation
+def getSpeaker(speaker):
+    match speaker:
+        case "ファイン":
+            return ["Fine", [0, 0]]
+        case "":
+            return ["", [0, 0]]
+        case _:
+            # Find Speaker
+            for i in range(len(NAMESLIST)):
+                if speaker == NAMESLIST[i][0]:
+                    return [NAMESLIST[i][1], [0, 0]]
+
+            # Translate and Store Speaker
+            response = translateAI(
+                f"{speaker}",
+                "Reply with the " + LANGUAGE + " translation of the NPC name.",
+                False,
+            )
+            response[0] = response[0].title()
+            response[0] = response[0].replace("'S", "'s")
+            response[0] = response[0].replace("Speaker: ", "")
+
+            # Retry if name doesn't translate for some reason
+            if re.search(r"([a-zA-Z？?])", response[0]) == None:
+                response = translateAI(
+                    f"{speaker}",
+                    "Reply with the " + LANGUAGE + " translation of the NPC name.",
+                    False,
+                )
+                response[0] = response[0].title()
+                response[0] = response[0].replace("'S", "'s")
+
+            speakerList = [speaker, response[0]]
+            NAMESLIST.append(speakerList)
+            return response
+    return [speaker, [0, 0]]
+
+def translateAI(text, history, history_ctx=None):
+    """
+    Legacy wrapper function for the new shared translation utility.
+    This maintains compatibility with existing code while using the new shared implementation.
+    """
+    global PBAR, MISMATCH, FILENAME
+    
+    # Update config estimate mode based on global ESTIMATE
+    TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
+    
+    # Call the new shared translation function
+    return sharedtranslateAI(
+        text=text,
+        history=history,
+        config=TRANSLATION_CONFIG,
+        filename=FILENAME,
+        pbar=PBAR,
+        lock=LOCK,
+        mismatchList=MISMATCH
     )

modules/tyrano.py

@@ -1,457 +1,459 @@
-# Libraries
-import json
-import os
-import re
-import util.dazedwrap as dazedwrap
-import threading
-import time
-import traceback
-import tiktoken
-from pathlib import Path
-from colorama import Fore
-from dotenv import load_dotenv
-from retry import retry
-from tqdm import tqdm
-from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
-import tempfile
-
-# OpenAI initialization centralized in util/translation.py
-
-# Globals
-MODEL = os.getenv("model")
-TIMEOUT = int(os.getenv("timeout"))
-LANGUAGE = os.getenv("language").capitalize()
-PROMPT = Path("prompt.txt").read_text(encoding="utf-8")
-VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
-LOCK = threading.Lock()
-WIDTH = int(os.getenv("width"))
-LISTWIDTH = int(os.getenv("listWidth"))
-NOTEWIDTH = 70
-MAXHISTORY = 10
-ESTIMATE = ""
-TOKENS = [0, 0]
-NAMESLIST = []
-NAMES = False  # Output a list of all the character names found
-BRFLAG = False  # If the game uses <br> instead
-FIXTEXTWRAP = True  # Overwrites textwrap
-IGNORETLTEXT = False  # Ignores all translated text.
-MISMATCH = []  # Lists files that throw a mismatch error (Length of GPT list response is wrong)
-FILENAME = None
-
-# tqdm Globals
-BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
-POSITION = 0
-LEAVE = False
-PBAR = None
-
-# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
-LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
-
-# Get pricing configuration based on the model
-PRICING_CONFIG = getPricingConfig(MODEL)
-INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
-OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
-BATCHSIZE = PRICING_CONFIG["batchSize"]
-FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
-
-# Initialize Translation Config
-TRANSLATION_CONFIG = TranslationConfig(
-    model=MODEL,
-    language=LANGUAGE,
-    prompt=PROMPT,
-    vocab=VOCAB,
-    langRegex=LANGREGEX,
-    batchSize=BATCHSIZE,
-    maxHistory=MAXHISTORY,
-    estimateMode=False  # Will be set dynamically based on ESTIMATE
-)
-LEAVE = False
-
-def handleTyrano(filename, estimate):
-    global ESTIMATE, TOKENS, FILENAME
-    ESTIMATE = estimate
-    FILENAME = filename
-
-    # Translate
-    start = time.time()
-    translatedData = openFiles(filename)
-
-    # Translate
-    if not estimate:
-        try:
-            with open("translated/" + filename, "w", encoding="utf-8", newline="\n") as outFile:
-                outFile.writelines(translatedData[0])
-        except Exception:
-            traceback.print_exc()
-            return "Fail"
-
-    # Print File
-    end = time.time()
-    tqdm.write(getResultString(translatedData, end - start, filename))
-    with LOCK:
-        TOKENS[0] += translatedData[1][0]
-        TOKENS[1] += translatedData[1][1]
-
-    # Print Total
-    totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
-
-    # Print any errors on maps
-    if len(MISMATCH) > 0:
-        return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
-    else:
-        return totalString
-
-
-def getResultString(translatedData, translationTime, filename):
-    # File Print String
-    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
-    totalTokenstring = (
-        Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
-        "[Output: "
-        + str(translatedData[1][1])
-        + "]" "[Cost: ${:,.4f}".format(cost)
-        + "]"
-    )
-    timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
-
-    if translatedData[2] == None:
-        # Success
-        return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
-
-    else:
-        # Fail
-        try:
-            raise translatedData[2]
-        except Exception as e:
-            traceback.print_exc()
-            errorString = str(e) + Fore.RED
-            return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
-
-
-def openFiles(filename):
-    with open("files/" + filename, "r", encoding="utf8") as readFile:
-        translatedData = parseTyrano(readFile, filename)
-
-        # Delete lines marked for deletion
-        finalData = []
-        for line in translatedData[0]:
-            if line != "\\d\n":
-                finalData.append(line)
-        translatedData[0] = finalData
-
-    return translatedData
-
-
-def parseTyrano(readFile, filename):
-    global PBAR
-    totalTokens = [0, 0]
-
-    # Read File into data
-    data = readFile.readlines()
-
-    # Create Progress Bar
-    with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
-        pbar.desc = filename
-        PBAR = pbar
-
-        try:
-            result = translateTyrano(data, filename, [])
-            totalTokens[0] += result[0]
-            totalTokens[1] += result[1]
-        except Exception as e:
-            traceback.print_exc()
-            return [data, totalTokens, e]
-    return [data, totalTokens, None]
-
-
-def save_progress_lines(lines, filename, encoding="utf-8"):
-    """Atomically save current line-based translation progress."""
-    try:
-        if ESTIMATE:
-            return
-        os.makedirs("translated", exist_ok=True)
-        tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
-        try:
-            with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file:
-                tmp_file.writelines(lines)
-            os.replace(tmp_path, os.path.join("translated", filename))
-        finally:
-            if os.path.exists(tmp_path):
-                try:
-                    os.remove(tmp_path)
-                except OSError:
-                    pass
-    except Exception:
-        traceback.print_exc()
-
-
-def saveCheckLines(lines, filename, tokens=None, encoding="utf-8"):
-    """Save progress only when tokens indicate work or when tokens is None but we know a mutation occurred."""
-    try:
-        # If explicit tokens provided, gate on tokens; otherwise assume we were called on actual mutation
-        if tokens is not None:
-            if not (isinstance(tokens, (list, tuple)) and len(tokens) >= 2 and (tokens[0] or tokens[1])):
-                return
-        save_progress_lines(lines, filename, encoding=encoding)
-    except Exception:
-        traceback.print_exc()
-
-
-def translateTyrano(data, filename, translatedList):
-    if translatedList:
-        stringList = translatedList[0]
-        choiceList = translatedList[1]
-    else:
-        stringList = []
-        choiceList = []
-    tokens = [0, 0]
-    speaker = ""
-    stringListTL = []
-    choiceListTL = []
-    global LOCK, ESTIMATE, FILENAME, PBAR, MISMATCH
-    i = 0
-
-    while i < len(data):
-        voice = False
-        lineRegexNoSpeaker = r"^([^\[#;*@\n]+)\[p\]|^([^\[#;*@\n]+)\[l\]\[[rp]\]|^([^\[#;*@\n]+)\[[rpl]\]|^([^\[#;*@_\n]+)\n$"
-        lineRegexSpeaker = r"^#(.*)"
-        furiganaRegex = r"(\[ruby\stext=(.*?)\])"
-        choiceRegex = r'\[glink.+?text="(.*?)"'
-
-        # Speaker
-        match = re.search(lineRegexSpeaker, data[i])
-        if match:
-            if match.group(1):
-                response = getSpeaker(match.group(1))
-                speaker = response[0]
-                tokens[0] += response[1][0]
-                tokens[1] += response[1][1]
-                data[i] = data[i].replace(match.group(1), speaker)
-            else:
-                speaker = None
-
-        # Furigana
-        match = re.search(r"^\[ruby\stext", data[i])
-        furiganaList = []
-        if match:
-            # Check next line and combine
-            while match:
-                furiganaList.append(data[i].replace("\n", ""))
-                del data[i]
-                match = re.search(r"^\[ruby\stext", data[i])
-            jaString = "".join(furiganaList)
-
-            # Ruby Text
-            furiganaList = re.findall(furiganaRegex, jaString)
-            for furigana in furiganaList:
-                jaString = jaString.replace(furigana[0], furigana[1])
-
-            data.insert(i, f"{jaString}[r]")
-
-        # Dialogue
-        match = re.search(lineRegexNoSpeaker, data[i])
-        jaString = None
-        if match:
-            # Find which group matched
-            for group_num in range(1, 5):
-                try:
-                    if match.group(group_num):
-                        jaString = match.group(group_num)
-                        break
-                except IndexError:
-                    break
-            
-            # Skip if no valid string found
-            if not jaString:
-                i += 1
-                continue
-
-            # Combine w/ next line if necessary
-            repeatRegex = r"(.+?)\[[rpl]\]"
-            if i + 1 < len(data):
-                match = re.search(repeatRegex, data[i + 1])
-                while match and "[p]" not in data[i]:
-                    jaString = jaString + match.group(1)
-                    jaString = jaString.replace("_　", "")
-                    if "[p]" in data[i + 1]:
-                        data[i] = f"{jaString}[p]\n"
-                    else:
-                        data[i] = jaString
-                    del data[i + 1]
-                    if i + 1 < len(data):
-                        match = re.search(repeatRegex, data[i + 1])
-                    else:
-                        break
-
-            originalString = jaString
-
-            # Pass 1
-            if not translatedList:
-                # Remove any textwrap and commands
-                jaString = jaString.replace("[r]", " ")
-                jaString = jaString.replace("[l]", "")
-
-                # Ruby Text
-                furiganaList = re.findall(furiganaRegex, jaString)
-                for furigana in furiganaList:
-                    jaString = jaString.replace(furigana[0], furigana[1])
-
-                # Strip Spaces
-                jaString = jaString.strip()
-
-                if jaString:
-                    if speaker:
-                        stringList.append(f"[{speaker}]: {jaString}")
-                    else:
-                        stringList.append(jaString)
-
-            # Pass 2
-            else:
-                # Get Text
-                if stringList:
-                    # Grab and Pop
-                    translatedText = stringList[0]
-                    stringList.pop(0)
-
-                    # Set to None if empty list
-                    if len(stringList) <= 0:
-                        stringList = None
-
-                    # Remove speaker
-                    if speaker != "":
-                        matchSpeakerList = re.findall(r"^\[?(.+?)\]?\s?[|:]\s?", translatedText)
-                        translatedText = re.sub(r"^\[?(.+?)\]?\s?[|:]\s?", "", translatedText)
-
-                    # Avoid Crashes
-                    translatedText = translatedText.replace("[", "(")
-                    translatedText = translatedText.replace("]", ")")
-
-                    # Textwrap
-                    translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
-                    translatedText = translatedText.replace("\n", "[r]")
-
-                    # Set Data
-                    data[i] = data[i].replace(originalString, translatedText)
-                    # Save progress after each line change (we mutated, so tokens gate optional)
-                    saveCheckLines(data, filename)
-
-        # Choices
-        match = re.search(choiceRegex, data[i])
-        if match:
-            # Pass 1
-            if not translatedList:
-                choiceList.append(match.group(1))
-                match = re.search(choiceRegex, data[i + 1])
-
-            # Pass 2
-            else:
-                # Grab and Pop
-                translatedText = choiceList[0]
-                choiceList.pop(0)
-
-                # Replace Spaces
-                translatedText = translatedText.replace(" ", "\u3000")
-
-                # Set
-                data[i] = data[i].replace(match.group(1), translatedText)
-                saveCheckLines(data, filename)
-
-            i += 1
-        else:
-            i += 1
-
-    # EOF
-    if not translatedList:
-        # String List
-        if stringList:
-            PBAR.total = len(stringList)
-            PBAR.refresh()
-            response = translateAI(stringList, "Reply with the English Translation")
-            tokens[0] += response[1][0]
-            tokens[1] += response[1][1]
-            stringListTL = response[0]
-
-            if len(stringList) != len(stringListTL):
-                # Mismatch
-                with LOCK:
-                    if FILENAME not in MISMATCH:
-                        MISMATCH.append(FILENAME)
-
-        # Choice List
-        if choiceList:
-            response = translateAI(choiceList, "Reply with the English TL of the Dialogue Choice")
-            tokens[0] += response[1][0]
-            tokens[1] += response[1][1]
-            choiceListTL = response[0]
-
-            if len(choiceList) != len(choiceListTL):
-                # Mismatch
-                with LOCK:
-                    if FILENAME not in MISMATCH:
-                        MISMATCH.append(FILENAME)
-
-        # Recursive call for Pass 2 with translated strings
-        result = translateTyrano(data, filename, [stringListTL, choiceListTL])
-        tokens[0] += result[0]
-        tokens[1] += result[1]
-    
-    return tokens
-
-# Save some money and enter the character before translation
-def getSpeaker(speaker):
-    match speaker:
-        case "ファイン":
-            return ["Fine", [0, 0]]
-        case "":
-            return ["", [0, 0]]
-        case _:
-            # Find Speaker
-            for i in range(len(NAMESLIST)):
-                if speaker == NAMESLIST[i][0]:
-                    return [NAMESLIST[i][1], [0, 0]]
-
-            # Translate and Store Speaker
-            response = translateAI(
-                f"{speaker}",
-                "Reply with the " + LANGUAGE + " translation of the NPC name.",
-                False,
-            )
-            response[0] = response[0].title()
-            response[0] = response[0].replace("'S", "'s")
-            response[0] = response[0].replace("Speaker: ", "")
-
-            # Retry if name doesn't translate for some reason
-            if re.search(r"([a-zA-Z？?])", response[0]) == None:
-                response = translateAI(
-                    f"{speaker}",
-                    "Reply with the " + LANGUAGE + " translation of the NPC name.",
-                    False,
-                )
-                response[0] = response[0].title()
-                response[0] = response[0].replace("'S", "'s")
-
-            speakerList = [speaker, response[0]]
-            NAMESLIST.append(speakerList)
-            return response
-    return [speaker, [0, 0]]
-
-def translateAI(text, history, history_ctx=None):
-    """
-    Legacy wrapper function for the new shared translation utility.
-    This maintains compatibility with existing code while using the new shared implementation.
-    """
-    global PBAR, MISMATCH, FILENAME
-    
-    # Update config estimate mode based on global ESTIMATE
-    TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
-    
-    # Call the new shared translation function
-    return sharedtranslateAI(
-        text=text,
-        history=history,
-        config=TRANSLATION_CONFIG,
-        filename=FILENAME,
-        pbar=PBAR,
-        lock=LOCK,
-        mismatchList=MISMATCH
-    )
+# Libraries
+import json
+import os
+import re
+import util.dazedwrap as dazedwrap
+import threading
+import time
+import traceback
+import tiktoken
+from pathlib import Path
+from colorama import Fore
+from dotenv import load_dotenv
+from retry import retry
+from tqdm import tqdm
+from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
+import tempfile
+
+# OpenAI initialization centralized in util/translation.py
+
+# Globals
+MODEL = os.getenv("model")
+TIMEOUT = int(os.getenv("timeout"))
+LANGUAGE = os.getenv("language").capitalize()
+from util.paths import PROMPT_PATH, VOCAB_PATH
+
+PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
+VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
+LOCK = threading.Lock()
+WIDTH = int(os.getenv("width"))
+LISTWIDTH = int(os.getenv("listWidth"))
+NOTEWIDTH = 70
+MAXHISTORY = 10
+ESTIMATE = ""
+TOKENS = [0, 0]
+NAMESLIST = []
+NAMES = False  # Output a list of all the character names found
+BRFLAG = False  # If the game uses <br> instead
+FIXTEXTWRAP = True  # Overwrites textwrap
+IGNORETLTEXT = False  # Ignores all translated text.
+MISMATCH = []  # Lists files that throw a mismatch error (Length of GPT list response is wrong)
+FILENAME = None
+
+# tqdm Globals
+BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
+POSITION = 0
+LEAVE = False
+PBAR = None
+
+# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
+LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
+
+# Get pricing configuration based on the model
+PRICING_CONFIG = getPricingConfig(MODEL)
+INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
+OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
+BATCHSIZE = PRICING_CONFIG["batchSize"]
+FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
+
+# Initialize Translation Config
+TRANSLATION_CONFIG = TranslationConfig(
+    model=MODEL,
+    language=LANGUAGE,
+    prompt=PROMPT,
+    vocab=VOCAB,
+    langRegex=LANGREGEX,
+    batchSize=BATCHSIZE,
+    maxHistory=MAXHISTORY,
+    estimateMode=False  # Will be set dynamically based on ESTIMATE
+)
+LEAVE = False
+
+def handleTyrano(filename, estimate):
+    global ESTIMATE, TOKENS, FILENAME
+    ESTIMATE = estimate
+    FILENAME = filename
+
+    # Translate
+    start = time.time()
+    translatedData = openFiles(filename)
+
+    # Translate
+    if not estimate:
+        try:
+            with open("translated/" + filename, "w", encoding="utf-8", newline="\n") as outFile:
+                outFile.writelines(translatedData[0])
+        except Exception:
+            traceback.print_exc()
+            return "Fail"
+
+    # Print File
+    end = time.time()
+    tqdm.write(getResultString(translatedData, end - start, filename))
+    with LOCK:
+        TOKENS[0] += translatedData[1][0]
+        TOKENS[1] += translatedData[1][1]
+
+    # Print Total
+    totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
+
+    # Print any errors on maps
+    if len(MISMATCH) > 0:
+        return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
+    else:
+        return totalString
+
+
+def getResultString(translatedData, translationTime, filename):
+    # File Print String
+    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
+    totalTokenstring = (
+        Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
+        "[Output: "
+        + str(translatedData[1][1])
+        + "]" "[Cost: ${:,.4f}".format(cost)
+        + "]"
+    )
+    timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
+
+    if translatedData[2] == None:
+        # Success
+        return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
+
+    else:
+        # Fail
+        try:
+            raise translatedData[2]
+        except Exception as e:
+            traceback.print_exc()
+            errorString = str(e) + Fore.RED
+            return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
+
+
+def openFiles(filename):
+    with open("files/" + filename, "r", encoding="utf8") as readFile:
+        translatedData = parseTyrano(readFile, filename)
+
+        # Delete lines marked for deletion
+        finalData = []
+        for line in translatedData[0]:
+            if line != "\\d\n":
+                finalData.append(line)
+        translatedData[0] = finalData
+
+    return translatedData
+
+
+def parseTyrano(readFile, filename):
+    global PBAR
+    totalTokens = [0, 0]
+
+    # Read File into data
+    data = readFile.readlines()
+
+    # Create Progress Bar
+    with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
+        pbar.desc = filename
+        PBAR = pbar
+
+        try:
+            result = translateTyrano(data, filename, [])
+            totalTokens[0] += result[0]
+            totalTokens[1] += result[1]
+        except Exception as e:
+            traceback.print_exc()
+            return [data, totalTokens, e]
+    return [data, totalTokens, None]
+
+
+def save_progress_lines(lines, filename, encoding="utf-8"):
+    """Atomically save current line-based translation progress."""
+    try:
+        if ESTIMATE:
+            return
+        os.makedirs("translated", exist_ok=True)
+        tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
+        try:
+            with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file:
+                tmp_file.writelines(lines)
+            os.replace(tmp_path, os.path.join("translated", filename))
+        finally:
+            if os.path.exists(tmp_path):
+                try:
+                    os.remove(tmp_path)
+                except OSError:
+                    pass
+    except Exception:
+        traceback.print_exc()
+
+
+def saveCheckLines(lines, filename, tokens=None, encoding="utf-8"):
+    """Save progress only when tokens indicate work or when tokens is None but we know a mutation occurred."""
+    try:
+        # If explicit tokens provided, gate on tokens; otherwise assume we were called on actual mutation
+        if tokens is not None:
+            if not (isinstance(tokens, (list, tuple)) and len(tokens) >= 2 and (tokens[0] or tokens[1])):
+                return
+        save_progress_lines(lines, filename, encoding=encoding)
+    except Exception:
+        traceback.print_exc()
+
+
+def translateTyrano(data, filename, translatedList):
+    if translatedList:
+        stringList = translatedList[0]
+        choiceList = translatedList[1]
+    else:
+        stringList = []
+        choiceList = []
+    tokens = [0, 0]
+    speaker = ""
+    stringListTL = []
+    choiceListTL = []
+    global LOCK, ESTIMATE, FILENAME, PBAR, MISMATCH
+    i = 0
+
+    while i < len(data):
+        voice = False
+        lineRegexNoSpeaker = r"^([^\[#;*@\n]+)\[p\]|^([^\[#;*@\n]+)\[l\]\[[rp]\]|^([^\[#;*@\n]+)\[[rpl]\]|^([^\[#;*@_\n]+)\n$"
+        lineRegexSpeaker = r"^#(.*)"
+        furiganaRegex = r"(\[ruby\stext=(.*?)\])"
+        choiceRegex = r'\[glink.+?text="(.*?)"'
+
+        # Speaker
+        match = re.search(lineRegexSpeaker, data[i])
+        if match:
+            if match.group(1):
+                response = getSpeaker(match.group(1))
+                speaker = response[0]
+                tokens[0] += response[1][0]
+                tokens[1] += response[1][1]
+                data[i] = data[i].replace(match.group(1), speaker)
+            else:
+                speaker = None
+
+        # Furigana
+        match = re.search(r"^\[ruby\stext", data[i])
+        furiganaList = []
+        if match:
+            # Check next line and combine
+            while match:
+                furiganaList.append(data[i].replace("\n", ""))
+                del data[i]
+                match = re.search(r"^\[ruby\stext", data[i])
+            jaString = "".join(furiganaList)
+
+            # Ruby Text
+            furiganaList = re.findall(furiganaRegex, jaString)
+            for furigana in furiganaList:
+                jaString = jaString.replace(furigana[0], furigana[1])
+
+            data.insert(i, f"{jaString}[r]")
+
+        # Dialogue
+        match = re.search(lineRegexNoSpeaker, data[i])
+        jaString = None
+        if match:
+            # Find which group matched
+            for group_num in range(1, 5):
+                try:
+                    if match.group(group_num):
+                        jaString = match.group(group_num)
+                        break
+                except IndexError:
+                    break
+            
+            # Skip if no valid string found
+            if not jaString:
+                i += 1
+                continue
+
+            # Combine w/ next line if necessary
+            repeatRegex = r"(.+?)\[[rpl]\]"
+            if i + 1 < len(data):
+                match = re.search(repeatRegex, data[i + 1])
+                while match and "[p]" not in data[i]:
+                    jaString = jaString + match.group(1)
+                    jaString = jaString.replace("_　", "")
+                    if "[p]" in data[i + 1]:
+                        data[i] = f"{jaString}[p]\n"
+                    else:
+                        data[i] = jaString
+                    del data[i + 1]
+                    if i + 1 < len(data):
+                        match = re.search(repeatRegex, data[i + 1])
+                    else:
+                        break
+
+            originalString = jaString
+
+            # Pass 1
+            if not translatedList:
+                # Remove any textwrap and commands
+                jaString = jaString.replace("[r]", " ")
+                jaString = jaString.replace("[l]", "")
+
+                # Ruby Text
+                furiganaList = re.findall(furiganaRegex, jaString)
+                for furigana in furiganaList:
+                    jaString = jaString.replace(furigana[0], furigana[1])
+
+                # Strip Spaces
+                jaString = jaString.strip()
+
+                if jaString:
+                    if speaker:
+                        stringList.append(f"[{speaker}]: {jaString}")
+                    else:
+                        stringList.append(jaString)
+
+            # Pass 2
+            else:
+                # Get Text
+                if stringList:
+                    # Grab and Pop
+                    translatedText = stringList[0]
+                    stringList.pop(0)
+
+                    # Set to None if empty list
+                    if len(stringList) <= 0:
+                        stringList = None
+
+                    # Remove speaker
+                    if speaker != "":
+                        matchSpeakerList = re.findall(r"^\[?(.+?)\]?\s?[|:]\s?", translatedText)
+                        translatedText = re.sub(r"^\[?(.+?)\]?\s?[|:]\s?", "", translatedText)
+
+                    # Avoid Crashes
+                    translatedText = translatedText.replace("[", "(")
+                    translatedText = translatedText.replace("]", ")")
+
+                    # Textwrap
+                    translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
+                    translatedText = translatedText.replace("\n", "[r]")
+
+                    # Set Data
+                    data[i] = data[i].replace(originalString, translatedText)
+                    # Save progress after each line change (we mutated, so tokens gate optional)
+                    saveCheckLines(data, filename)
+
+        # Choices
+        match = re.search(choiceRegex, data[i])
+        if match:
+            # Pass 1
+            if not translatedList:
+                choiceList.append(match.group(1))
+                match = re.search(choiceRegex, data[i + 1])
+
+            # Pass 2
+            else:
+                # Grab and Pop
+                translatedText = choiceList[0]
+                choiceList.pop(0)
+
+                # Replace Spaces
+                translatedText = translatedText.replace(" ", "\u3000")
+
+                # Set
+                data[i] = data[i].replace(match.group(1), translatedText)
+                saveCheckLines(data, filename)
+
+            i += 1
+        else:
+            i += 1
+
+    # EOF
+    if not translatedList:
+        # String List
+        if stringList:
+            PBAR.total = len(stringList)
+            PBAR.refresh()
+            response = translateAI(stringList, "Reply with the English Translation")
+            tokens[0] += response[1][0]
+            tokens[1] += response[1][1]
+            stringListTL = response[0]
+
+            if len(stringList) != len(stringListTL):
+                # Mismatch
+                with LOCK:
+                    if FILENAME not in MISMATCH:
+                        MISMATCH.append(FILENAME)
+
+        # Choice List
+        if choiceList:
+            response = translateAI(choiceList, "Reply with the English TL of the Dialogue Choice")
+            tokens[0] += response[1][0]
+            tokens[1] += response[1][1]
+            choiceListTL = response[0]
+
+            if len(choiceList) != len(choiceListTL):
+                # Mismatch
+                with LOCK:
+                    if FILENAME not in MISMATCH:
+                        MISMATCH.append(FILENAME)
+
+        # Recursive call for Pass 2 with translated strings
+        result = translateTyrano(data, filename, [stringListTL, choiceListTL])
+        tokens[0] += result[0]
+        tokens[1] += result[1]
+    
+    return tokens
+
+# Save some money and enter the character before translation
+def getSpeaker(speaker):
+    match speaker:
+        case "ファイン":
+            return ["Fine", [0, 0]]
+        case "":
+            return ["", [0, 0]]
+        case _:
+            # Find Speaker
+            for i in range(len(NAMESLIST)):
+                if speaker == NAMESLIST[i][0]:
+                    return [NAMESLIST[i][1], [0, 0]]
+
+            # Translate and Store Speaker
+            response = translateAI(
+                f"{speaker}",
+                "Reply with the " + LANGUAGE + " translation of the NPC name.",
+                False,
+            )
+            response[0] = response[0].title()
+            response[0] = response[0].replace("'S", "'s")
+            response[0] = response[0].replace("Speaker: ", "")
+
+            # Retry if name doesn't translate for some reason
+            if re.search(r"([a-zA-Z？?])", response[0]) == None:
+                response = translateAI(
+                    f"{speaker}",
+                    "Reply with the " + LANGUAGE + " translation of the NPC name.",
+                    False,
+                )
+                response[0] = response[0].title()
+                response[0] = response[0].replace("'S", "'s")
+
+            speakerList = [speaker, response[0]]
+            NAMESLIST.append(speakerList)
+            return response
+    return [speaker, [0, 0]]
+
+def translateAI(text, history, history_ctx=None):
+    """
+    Legacy wrapper function for the new shared translation utility.
+    This maintains compatibility with existing code while using the new shared implementation.
+    """
+    global PBAR, MISMATCH, FILENAME
+    
+    # Update config estimate mode based on global ESTIMATE
+    TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
+    
+    # Call the new shared translation function
+    return sharedtranslateAI(
+        text=text,
+        history=history,
+        config=TRANSLATION_CONFIG,
+        filename=FILENAME,
+        pbar=PBAR,
+        lock=LOCK,
+        mismatchList=MISMATCH
+    )

modules/unity.py

@@ -1,353 +1,355 @@
-# Libraries
-import json
-import os
-import re
-import util.dazedwrap as dazedwrap
-import threading
-import time
-import traceback
-import tiktoken
-from pathlib import Path
-from colorama import Fore
-from dotenv import load_dotenv
-from retry import retry
-from tqdm import tqdm
-from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost, getPricingConfig, calculateCost
-import tempfile
-
-# OpenAI initialization centralized in util/translation.py
-
-# Globals
-MODEL = os.getenv("model")
-TIMEOUT = int(os.getenv("timeout"))
-LANGUAGE = os.getenv("language").capitalize()
-PROMPT = Path("prompt.txt").read_text(encoding="utf-8")
-VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
-LOCK = threading.Lock()
-WIDTH = int(os.getenv("width"))
-LISTWIDTH = int(os.getenv("listWidth"))
-NOTEWIDTH = 70
-MAXHISTORY = 10
-ESTIMATE = ""
-TOKENS = [0, 0]
-NAMESLIST = []
-NAMES = False  # Output a list of all the character names found
-BRFLAG = False  # If the game uses <br> instead
-FIXTEXTWRAP = True  # Overwrites textwrap
-IGNORETLTEXT = False  # Ignores all translated text.
-MISMATCH = []  # Lists files that throw a mismatch error (Length of GPT list response is wrong)
-
-# tqdm Globals
-BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
-POSITION = 0
-LEAVE = False
-PBAR = None
-FILENAME = None
-
-# Full Width
-ascii_to_wide = dict((i, chr(i + 0xFEE0)) for i in range(0x21, 0x7F))
-ascii_to_wide.update({0x20: "\u3000", 0x2D: "\u2212"})  # space and minus
-wide_to_ascii = dict((i, chr(i - 0xFEE0)) for i in range(0xFF01, 0xFF5F))
-wide_to_ascii.update({0x3000: " ", 0x2212: "-"})  # space and minus
-
-# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
-LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
-
-# Get pricing configuration based on the model
-PRICING_CONFIG = getPricingConfig(MODEL)
-INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
-OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
-BATCHSIZE = PRICING_CONFIG["batchSize"]
-FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
-
-# Initialize Translation Config
-TRANSLATION_CONFIG = TranslationConfig(
-    model=MODEL,
-    language=LANGUAGE,
-    prompt=PROMPT,
-    vocab=VOCAB,
-    langRegex=LANGREGEX,
-    batchSize=BATCHSIZE,
-    maxHistory=MAXHISTORY,
-    estimateMode=False  # Will be set dynamically based on ESTIMATE
-)
-LEAVE = False
-
-def handleUnity(filename, estimate):
-    global ESTIMATE, FILENAME
-    ESTIMATE = estimate
-    FILENAME = filename
-
-    if ESTIMATE:
-        start = time.time()
-        translatedData = openFiles(filename)
-
-        # Print Result
-        end = time.time()
-        tqdm.write(getResultString(translatedData, end - start, filename))
-        with LOCK:
-            TOKENS[0] += translatedData[1][0]
-            TOKENS[1] += translatedData[1][1]
-
-        # Print Total
-        totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
-
-        # Print any errors on maps
-        if len(MISMATCH) > 0:
-            return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
-        else:
-            return totalString
-
-    else:
-        try:
-            with open("translated/" + filename, "w", encoding="utf8", errors="ignore") as outFile:
-                start = time.time()
-                translatedData = openFiles(filename)
-
-                # Print Result
-                end = time.time()
-                outFile.writelines(translatedData[0])
-                tqdm.write(getResultString(translatedData, end - start, filename))
-                with LOCK:
-                    TOKENS[0] += translatedData[1][0]
-                    TOKENS[1] += translatedData[1][1]
-        except Exception:
-            traceback.print_exc()
-            return "Fail"
-
-    return getResultString(["", TOKENS, None], end - start, "TOTAL")
-
-
-def getResultString(translatedData, translationTime, filename):
-    # File Print String
-    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
-    totalTokenstring = (
-        Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
-        "[Output: "
-        + str(translatedData[1][1])
-        + "]" "[Cost: ${:,.4f}".format(cost)
-        + "]"
-    )
-    timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
-
-    if translatedData[2] == None:
-        # Success
-        return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
-
-    else:
-        # Fail
-        try:
-            raise translatedData[2]
-        except Exception as e:
-            traceback.print_exc()
-            errorString = str(e) + Fore.RED
-            return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
-
-
-def openFiles(filename):
-    with open("files/" + filename, "r", encoding="utf8") as readFile:
-        translatedData = parseUnity(readFile, filename)
-
-        # Delete lines marked for deletion
-        finalData = []
-        for line in translatedData[0]:
-            if line != "\\d\n":
-                finalData.append(line)
-        translatedData[0] = finalData
-
-    return translatedData
-
-
-def parseUnity(readFile, filename):
-    totalTokens = [0, 0]
-
-    # Read File into data
-    data = readFile.readlines()
-
-    # Create Progress Bar
-    with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
-        pbar.desc = filename
-
-        try:
-            result = translateUnity(data, pbar, filename, [])
-            totalTokens[0] += result[0]
-            totalTokens[1] += result[1]
-        except Exception as e:
-            traceback.print_exc()
-            return [data, totalTokens, e]
-    return [data, totalTokens, None]
-
-
-def save_progress_lines(lines, filename, encoding="utf-8"):
-    """Atomically save current line-based translation progress."""
-    try:
-        if ESTIMATE:
-            return
-        os.makedirs("translated", exist_ok=True)
-        tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
-        try:
-            with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file:
-                tmp_file.writelines(lines)
-            os.replace(tmp_path, os.path.join("translated", filename))
-        finally:
-            if os.path.exists(tmp_path):
-                try:
-                    os.remove(tmp_path)
-                except OSError:
-                    pass
-    except Exception:
-        traceback.print_exc()
-
-
-def saveCheckLines(lines, filename, tokens=None, encoding="utf-8"):
-    """Save progress only when tokens indicate work or when explicitly called after a mutation."""
-    try:
-        if tokens is not None:
-            if not (isinstance(tokens, (list, tuple)) and len(tokens) >= 2 and (tokens[0] or tokens[1])):
-                return
-        save_progress_lines(lines, filename, encoding=encoding)
-    except Exception:
-        traceback.print_exc()
-
-
-def translateUnity(data, pbar, filename, translatedList):
-    stringList = []
-    tokens = [0, 0]
-    global LOCK, ESTIMATE, PBAR
-    PBAR = pbar
-    i = 0
-
-    # Dialogue
-    while i < len(data):
-        # Lines
-        regex = r"(.*?)=(.*)"
-        match = re.search(regex, data[i])
-        if match:
-            leftString = match.group(1)
-            rightString = match.group(2)
-
-            # Validate Japanese Text
-            if not re.search(LANGREGEX, leftString) and IGNORETLTEXT:
-                i += 1
-                continue
-
-            # Pass 1
-            if translatedList == []:
-                jaString = leftString
-                
-                # Remove textwrap
-                # jaString = jaString.replace("\\n", " ")
-
-                # Add String
-                stringList.append(jaString.strip())
-
-            # Pass 2
-            else:
-                # Get Text
-                if translatedList:
-                    # Grab and Pop
-                    translatedText = translatedList[0]
-                    translatedList.pop(0)
-
-                    # Set to None if empty list
-                    if len(translatedList) <= 0:
-                        translatedList = None
-
-                    # Textwrap
-                    # translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
-                    # translatedText = translatedText.replace("\n", "\\n")
-
-                    # Remove Double Spaces and =
-                    translatedText = translatedText.replace("  ", " ")
-                    translatedText = translatedText.replace("=", "->")
-
-                    # Set Data
-                    data[i] = f"{leftString}={translatedText}\n"
-                    saveCheckLines(data, filename)
-            i += 1
-
-        # Nothing relevant. Skip Line.
-        else:
-            i += 1
-
-    # EOF
-    if len(stringList) > 0:
-        # Set Progress
-        pbar.total = len(stringList)
-        pbar.refresh()
-
-        # Translate
-        response = translateAI(stringList, "")
-        tokens[0] += response[1][0]
-        tokens[1] += response[1][1]
-        translatedList = response[0]
-
-        # Set Strings
-        if len(stringList) == len(translatedList):
-            translateUnity(data, pbar, filename, translatedList)
-
-        # Mismatch
-        else:
-            with LOCK:
-                if filename not in MISMATCH:
-                    MISMATCH.append(filename)
-    return tokens
-
-# Save some money and enter the character before translation
-def getSpeaker(speaker):
-    match speaker:
-        case "ファイン":
-            return ["Fine", [0, 0]]
-        case "":
-            return ["", [0, 0]]
-        case _:
-            # Find Speaker
-            for i in range(len(NAMESLIST)):
-                if speaker == NAMESLIST[i][0]:
-                    return [NAMESLIST[i][1], [0, 0]]
-
-            # Translate and Store Speaker
-            response = translateAI(
-                f"{speaker}",
-                "Reply with the " + LANGUAGE + " translation of the NPC name.",
-                False,
-            )
-            response[0] = response[0].title()
-            response[0] = response[0].replace("'S", "'s")
-            response[0] = response[0].replace("Speaker: ", "")
-
-            # Retry if name doesn't translate for some reason
-            if re.search(r"([a-zA-Z？?])", response[0]) == None:
-                response = translateAI(
-                    f"{speaker}",
-                    "Reply with the " + LANGUAGE + " translation of the NPC name.",
-                    False,
-                )
-                response[0] = response[0].title()
-                response[0] = response[0].replace("'S", "'s")
-
-            speakerList = [speaker, response[0]]
-            NAMESLIST.append(speakerList)
-            return response
-    return [speaker, [0, 0]]
-
-def translateAI(text, history, history_ctx=None):
-    """
-    Legacy wrapper function for the new shared translation utility.
-    This maintains compatibility with existing code while using the new shared implementation.
-    """
-    global PBAR, MISMATCH, FILENAME
-    
-    # Update config estimate mode based on global ESTIMATE
-    TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
-    
-    # Call the new shared translation function
-    return sharedtranslateAI(
-        text=text,
-        history=history,
-        config=TRANSLATION_CONFIG,
-        filename=FILENAME,
-        pbar=PBAR,
-        lock=LOCK,
-        mismatchList=MISMATCH
-    )
+# Libraries
+import json
+import os
+import re
+import util.dazedwrap as dazedwrap
+import threading
+import time
+import traceback
+import tiktoken
+from pathlib import Path
+from colorama import Fore
+from dotenv import load_dotenv
+from retry import retry
+from tqdm import tqdm
+from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost, getPricingConfig, calculateCost
+import tempfile
+
+# OpenAI initialization centralized in util/translation.py
+
+# Globals
+MODEL = os.getenv("model")
+TIMEOUT = int(os.getenv("timeout"))
+LANGUAGE = os.getenv("language").capitalize()
+from util.paths import PROMPT_PATH, VOCAB_PATH
+
+PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
+VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
+LOCK = threading.Lock()
+WIDTH = int(os.getenv("width"))
+LISTWIDTH = int(os.getenv("listWidth"))
+NOTEWIDTH = 70
+MAXHISTORY = 10
+ESTIMATE = ""
+TOKENS = [0, 0]
+NAMESLIST = []
+NAMES = False  # Output a list of all the character names found
+BRFLAG = False  # If the game uses <br> instead
+FIXTEXTWRAP = True  # Overwrites textwrap
+IGNORETLTEXT = False  # Ignores all translated text.
+MISMATCH = []  # Lists files that throw a mismatch error (Length of GPT list response is wrong)
+
+# tqdm Globals
+BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
+POSITION = 0
+LEAVE = False
+PBAR = None
+FILENAME = None
+
+# Full Width
+ascii_to_wide = dict((i, chr(i + 0xFEE0)) for i in range(0x21, 0x7F))
+ascii_to_wide.update({0x20: "\u3000", 0x2D: "\u2212"})  # space and minus
+wide_to_ascii = dict((i, chr(i - 0xFEE0)) for i in range(0xFF01, 0xFF5F))
+wide_to_ascii.update({0x3000: " ", 0x2212: "-"})  # space and minus
+
+# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
+LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
+
+# Get pricing configuration based on the model
+PRICING_CONFIG = getPricingConfig(MODEL)
+INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
+OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
+BATCHSIZE = PRICING_CONFIG["batchSize"]
+FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
+
+# Initialize Translation Config
+TRANSLATION_CONFIG = TranslationConfig(
+    model=MODEL,
+    language=LANGUAGE,
+    prompt=PROMPT,
+    vocab=VOCAB,
+    langRegex=LANGREGEX,
+    batchSize=BATCHSIZE,
+    maxHistory=MAXHISTORY,
+    estimateMode=False  # Will be set dynamically based on ESTIMATE
+)
+LEAVE = False
+
+def handleUnity(filename, estimate):
+    global ESTIMATE, FILENAME
+    ESTIMATE = estimate
+    FILENAME = filename
+
+    if ESTIMATE:
+        start = time.time()
+        translatedData = openFiles(filename)
+
+        # Print Result
+        end = time.time()
+        tqdm.write(getResultString(translatedData, end - start, filename))
+        with LOCK:
+            TOKENS[0] += translatedData[1][0]
+            TOKENS[1] += translatedData[1][1]
+
+        # Print Total
+        totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
+
+        # Print any errors on maps
+        if len(MISMATCH) > 0:
+            return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
+        else:
+            return totalString
+
+    else:
+        try:
+            with open("translated/" + filename, "w", encoding="utf8", errors="ignore") as outFile:
+                start = time.time()
+                translatedData = openFiles(filename)
+
+                # Print Result
+                end = time.time()
+                outFile.writelines(translatedData[0])
+                tqdm.write(getResultString(translatedData, end - start, filename))
+                with LOCK:
+                    TOKENS[0] += translatedData[1][0]
+                    TOKENS[1] += translatedData[1][1]
+        except Exception:
+            traceback.print_exc()
+            return "Fail"
+
+    return getResultString(["", TOKENS, None], end - start, "TOTAL")
+
+
+def getResultString(translatedData, translationTime, filename):
+    # File Print String
+    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
+    totalTokenstring = (
+        Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
+        "[Output: "
+        + str(translatedData[1][1])
+        + "]" "[Cost: ${:,.4f}".format(cost)
+        + "]"
+    )
+    timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
+
+    if translatedData[2] == None:
+        # Success
+        return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
+
+    else:
+        # Fail
+        try:
+            raise translatedData[2]
+        except Exception as e:
+            traceback.print_exc()
+            errorString = str(e) + Fore.RED
+            return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
+
+
+def openFiles(filename):
+    with open("files/" + filename, "r", encoding="utf8") as readFile:
+        translatedData = parseUnity(readFile, filename)
+
+        # Delete lines marked for deletion
+        finalData = []
+        for line in translatedData[0]:
+            if line != "\\d\n":
+                finalData.append(line)
+        translatedData[0] = finalData
+
+    return translatedData
+
+
+def parseUnity(readFile, filename):
+    totalTokens = [0, 0]
+
+    # Read File into data
+    data = readFile.readlines()
+
+    # Create Progress Bar
+    with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
+        pbar.desc = filename
+
+        try:
+            result = translateUnity(data, pbar, filename, [])
+            totalTokens[0] += result[0]
+            totalTokens[1] += result[1]
+        except Exception as e:
+            traceback.print_exc()
+            return [data, totalTokens, e]
+    return [data, totalTokens, None]
+
+
+def save_progress_lines(lines, filename, encoding="utf-8"):
+    """Atomically save current line-based translation progress."""
+    try:
+        if ESTIMATE:
+            return
+        os.makedirs("translated", exist_ok=True)
+        tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
+        try:
+            with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file:
+                tmp_file.writelines(lines)
+            os.replace(tmp_path, os.path.join("translated", filename))
+        finally:
+            if os.path.exists(tmp_path):
+                try:
+                    os.remove(tmp_path)
+                except OSError:
+                    pass
+    except Exception:
+        traceback.print_exc()
+
+
+def saveCheckLines(lines, filename, tokens=None, encoding="utf-8"):
+    """Save progress only when tokens indicate work or when explicitly called after a mutation."""
+    try:
+        if tokens is not None:
+            if not (isinstance(tokens, (list, tuple)) and len(tokens) >= 2 and (tokens[0] or tokens[1])):
+                return
+        save_progress_lines(lines, filename, encoding=encoding)
+    except Exception:
+        traceback.print_exc()
+
+
+def translateUnity(data, pbar, filename, translatedList):
+    stringList = []
+    tokens = [0, 0]
+    global LOCK, ESTIMATE, PBAR
+    PBAR = pbar
+    i = 0
+
+    # Dialogue
+    while i < len(data):
+        # Lines
+        regex = r"(.*?)=(.*)"
+        match = re.search(regex, data[i])
+        if match:
+            leftString = match.group(1)
+            rightString = match.group(2)
+
+            # Validate Japanese Text
+            if not re.search(LANGREGEX, leftString) and IGNORETLTEXT:
+                i += 1
+                continue
+
+            # Pass 1
+            if translatedList == []:
+                jaString = leftString
+                
+                # Remove textwrap
+                # jaString = jaString.replace("\\n", " ")
+
+                # Add String
+                stringList.append(jaString.strip())
+
+            # Pass 2
+            else:
+                # Get Text
+                if translatedList:
+                    # Grab and Pop
+                    translatedText = translatedList[0]
+                    translatedList.pop(0)
+
+                    # Set to None if empty list
+                    if len(translatedList) <= 0:
+                        translatedList = None
+
+                    # Textwrap
+                    # translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
+                    # translatedText = translatedText.replace("\n", "\\n")
+
+                    # Remove Double Spaces and =
+                    translatedText = translatedText.replace("  ", " ")
+                    translatedText = translatedText.replace("=", "->")
+
+                    # Set Data
+                    data[i] = f"{leftString}={translatedText}\n"
+                    saveCheckLines(data, filename)
+            i += 1
+
+        # Nothing relevant. Skip Line.
+        else:
+            i += 1
+
+    # EOF
+    if len(stringList) > 0:
+        # Set Progress
+        pbar.total = len(stringList)
+        pbar.refresh()
+
+        # Translate
+        response = translateAI(stringList, "")
+        tokens[0] += response[1][0]
+        tokens[1] += response[1][1]
+        translatedList = response[0]
+
+        # Set Strings
+        if len(stringList) == len(translatedList):
+            translateUnity(data, pbar, filename, translatedList)
+
+        # Mismatch
+        else:
+            with LOCK:
+                if filename not in MISMATCH:
+                    MISMATCH.append(filename)
+    return tokens
+
+# Save some money and enter the character before translation
+def getSpeaker(speaker):
+    match speaker:
+        case "ファイン":
+            return ["Fine", [0, 0]]
+        case "":
+            return ["", [0, 0]]
+        case _:
+            # Find Speaker
+            for i in range(len(NAMESLIST)):
+                if speaker == NAMESLIST[i][0]:
+                    return [NAMESLIST[i][1], [0, 0]]
+
+            # Translate and Store Speaker
+            response = translateAI(
+                f"{speaker}",
+                "Reply with the " + LANGUAGE + " translation of the NPC name.",
+                False,
+            )
+            response[0] = response[0].title()
+            response[0] = response[0].replace("'S", "'s")
+            response[0] = response[0].replace("Speaker: ", "")
+
+            # Retry if name doesn't translate for some reason
+            if re.search(r"([a-zA-Z？?])", response[0]) == None:
+                response = translateAI(
+                    f"{speaker}",
+                    "Reply with the " + LANGUAGE + " translation of the NPC name.",
+                    False,
+                )
+                response[0] = response[0].title()
+                response[0] = response[0].replace("'S", "'s")
+
+            speakerList = [speaker, response[0]]
+            NAMESLIST.append(speakerList)
+            return response
+    return [speaker, [0, 0]]
+
+def translateAI(text, history, history_ctx=None):
+    """
+    Legacy wrapper function for the new shared translation utility.
+    This maintains compatibility with existing code while using the new shared implementation.
+    """
+    global PBAR, MISMATCH, FILENAME
+    
+    # Update config estimate mode based on global ESTIMATE
+    TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
+    
+    # Call the new shared translation function
+    return sharedtranslateAI(
+        text=text,
+        history=history,
+        config=TRANSLATION_CONFIG,
+        filename=FILENAME,
+        pbar=PBAR,
+        lock=LOCK,
+        mismatchList=MISMATCH
+    )

modules/wolf2.py

@@ -1,420 +1,422 @@
-# Libraries
-import json
-import io
-import os
-import re
-import util.dazedwrap as dazedwrap
-import threading
-import time
-import traceback
-import tiktoken
-from pathlib import Path
-from colorama import Fore
-from dotenv import load_dotenv
-from retry import retry
-from tqdm import tqdm
-from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
-import tempfile
-
-# OpenAI initialization centralized in util/translation.py
-
-# Globals
-MODEL = os.getenv("model")
-TIMEOUT = int(os.getenv("timeout"))
-LANGUAGE = os.getenv("language").capitalize()
-PROMPT = Path("prompt.txt").read_text(encoding="utf-8")
-VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
-LOCK = threading.Lock()
-WIDTH = int(os.getenv("width"))
-LISTWIDTH = int(os.getenv("listWidth"))
-NOTEWIDTH = 70
-MAXHISTORY = 10
-ESTIMATE = ""
-TOKENS = [0, 0]
-NAMESLIST = []
-NAMES = False  # Output a list of all the character names found
-BRFLAG = False  # If the game uses <br> instead
-FIXTEXTWRAP = True  # Overwrites textwrap
-IGNORETLTEXT = False  # Ignores all translated text.
-MISMATCH = []  # Lists files that throw a mismatch error (Length of GPT list response is wrong)
-FILENAME = ""  # Current file being processed, used by translateAI wrapper
-
-# tqdm Globals
-BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
-POSITION = 0
-LEAVE = False
-
-# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
-LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
-
-# Get pricing configuration based on the model
-PRICING_CONFIG = getPricingConfig(MODEL)
-INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
-OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
-BATCHSIZE = PRICING_CONFIG["batchSize"]
-FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
-
-# Initialize Translation Config
-TRANSLATION_CONFIG = TranslationConfig(
-    model=MODEL,
-    language=LANGUAGE,
-    prompt=PROMPT,
-    vocab=VOCAB,
-    langRegex=LANGREGEX,
-    batchSize=BATCHSIZE,
-    maxHistory=MAXHISTORY,
-    estimateMode=False  # Will be set dynamically based on ESTIMATE
-)
-LEAVE = False
-
-def handleWOLF2(filename, estimate):
-    global ESTIMATE, FILENAME
-    ESTIMATE = estimate
-    FILENAME = filename
-
-    if ESTIMATE:
-        start = time.time()
-        translatedData = openFiles(filename)
-
-        # Print Result
-        end = time.time()
-        tqdm.write(getResultString(translatedData, end - start, filename))
-        with LOCK:
-            TOKENS[0] += translatedData[1][0]
-            TOKENS[1] += translatedData[1][1]
-
-        # Print Total
-        totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
-
-        # Print any errors on maps
-        if len(MISMATCH) > 0:
-            return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
-        else:
-            return totalString
-
-    else:
-        try:
-            with open("translated/" + filename, "w", encoding="shift_jis", errors="ignore") as outFile:
-                start = time.time()
-                translatedData = openFiles(filename)
-
-                # Print Result
-                end = time.time()
-                outFile.writelines(translatedData[0])
-                tqdm.write(getResultString(translatedData, end - start, filename))
-                with LOCK:
-                    TOKENS[0] += translatedData[1][0]
-                    TOKENS[1] += translatedData[1][1]
-        except Exception:
-            traceback.print_exc()
-            return "Fail"
-
-    return getResultString(["", TOKENS, None], end - start, "TOTAL")
-
-
-def getResultString(translatedData, translationTime, filename):
-    # File Print String
-    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
-    totalTokenstring = (
-        Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
-        "[Output: "
-        + str(translatedData[1][1])
-        + "]" "[Cost: ${:,.4f}".format(cost)
-        + "]"
-    )
-    timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
-
-    if translatedData[2] == None:
-        # Success
-        return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
-
-    else:
-        # Fail
-        try:
-            raise translatedData[2]
-        except Exception as e:
-            traceback.print_exc()
-            errorString = str(e) + Fore.RED
-            return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
-
-
-def openFiles(filename):
-    # Use a robust reader to handle cp932/Shift_JIS variants and occasional bad bytes
-    def read_file_lines_with_fallback(path: str):
-        encodings = ["cp932", "shift_jis", "utf-8-sig", "utf-8"]
-        for enc in encodings:
-            try:
-                with open(path, "r", encoding=enc) as f:
-                    return f.readlines()
-            except UnicodeDecodeError:
-                continue
-        # Last resort: ignore undecodable bytes under cp932
-        with open(path, "rb") as f:
-            raw = f.read()
-        try:
-            text = raw.decode("cp932", errors="ignore")
-        except Exception:
-            text = raw.decode("latin-1", errors="ignore")
-        return text.splitlines(keepends=True)
-
-    path = os.path.join("files", filename)
-    lines = read_file_lines_with_fallback(path)
-
-    # Keep parseWOLF API by wrapping lines in a file-like object
-    with io.StringIO("".join(lines)) as readFile:
-        translatedData = parseWOLF(readFile, filename)
-
-    # Delete lines marked for deletion
-    finalData = []
-    for line in translatedData[0]:
-        if line != "\\d\n":
-            finalData.append(line)
-    translatedData[0] = finalData
-
-    return translatedData
-
-
-def parseWOLF(readFile, filename):
-    totalTokens = [0, 0]
-
-    # Read File into data
-    data = readFile.readlines()
-
-    # Create Progress Bar
-    with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
-        pbar.desc = filename
-
-        try:
-            result = translateWOLF(data, [], pbar, filename)
-            totalTokens[0] += result[0]
-            totalTokens[1] += result[1]
-        except Exception as e:
-            traceback.print_exc()
-            return [data, totalTokens, e]
-    return [data, totalTokens, None]
-
-
-def save_progress_lines(lines, filename, encoding="shift_jis"):
-    """Atomically save current line-based translation progress."""
-    try:
-        if ESTIMATE:
-            return
-        os.makedirs("translated", exist_ok=True)
-        tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
-        try:
-            with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file:
-                tmp_file.writelines(lines)
-            os.replace(tmp_path, os.path.join("translated", filename))
-        finally:
-            if os.path.exists(tmp_path):
-                try:
-                    os.remove(tmp_path)
-                except OSError:
-                    pass
-    except Exception:
-        traceback.print_exc()
-
-
-def saveCheckLines(lines, filename, tokens=None, encoding="shift_jis"):
-    try:
-        if tokens is not None:
-            if not (isinstance(tokens, (list, tuple)) and len(tokens) >= 2 and (tokens[0] or tokens[1])):
-                return
-        save_progress_lines(lines, filename, encoding=encoding)
-    except Exception:
-        traceback.print_exc()
-
-
-def translateWOLF(data, translatedList, pbar, filename):
-    stringList = []
-    currentGroup = []
-    tokens = [0, 0]
-    speaker = ""
-    global LOCK, ESTIMATE, PBAR
-    PBAR = pbar
-    i = 0
-
-    while i < len(data):
-        # Speaker
-        matchList = re.findall(r"^([^/]*)：", data[i])
-        if len(matchList) != 0:
-            response = getSpeaker(matchList[0])
-            speaker = response[0]
-            tokens[0] += response[1][0]
-            tokens[1] += response[1][1]
-            data[i] = data[i].replace(matchList[0], f"{speaker}")
-            saveCheckLines(data, filename)
-            i += 1
-        else:
-            speaker = ""
-
-        # Options
-        if "//選択肢" in data[i]:
-            i += 1
-            choiceList = []
-            initialIndex = i
-            while "//" in data[i] and "の場合" not in data[i]:
-                choiceList.append(re.search(r"\/\/(.*)", data[i]).group(1))
-                i += 1
-
-            # Translate
-            response = translateAI(choiceList, "This will be a dialogue option")
-            tokens[0] += response[1][0]
-            tokens[1] += response[1][1]
-            choiceListTL = response[0]
-
-            # Set Data
-            if len(choiceList) == len(choiceListTL):
-                # Set Data
-                i = initialIndex
-                while "//" in data[i] and "の場合" not in data[i]:
-                    choiceListTL[0] = choiceListTL[0].replace(", ", "、")
-                    data[i] = f"//{choiceListTL[0]}\n"
-                    choiceListTL.pop(0)
-                    i += 1
-                    saveCheckLines(data, filename)
-
-            # Mismatch
-            else:
-                with LOCK:
-                    if filename not in MISMATCH:
-                        MISMATCH.append(filename)
-
-        # Lines
-        if r"/" not in data[i] and "@" not in data[i] and data[i] != "\n":
-            # Pass 1
-            if translatedList == []:
-                # Grab Consecutive Strings
-                currentGroup.append(data[i])
-                i += 1
-                while i < len(data) and r"/" not in data[i] and "@" not in data[i] and data[i] != "\n":
-                    currentGroup.append(data[i])
-                    i += 1
-
-                # Join up 401 groups for better translation.
-                if len(currentGroup) > 0:
-                    jaString = "".join(currentGroup)
-                    currentGroup = []
-
-                # Remove any textwrap
-                jaString = jaString.replace("\n", " ")
-
-                # Add Speaker (If there is one)
-                if speaker != "":
-                    jaString = f"[{speaker}]: {jaString}"
-
-                # Add String
-                stringList.append(jaString)
-                i += 1
-
-            # Pass 2
-            else:
-                # Insert Strings
-                while i < len(data) and r"/" not in data[i] and "@" not in data[i] and data[i] != "\n":
-                    data.pop(i)
-
-                # Get Text
-                translatedText = translatedList[0]
-                translatedList.pop(0)
-
-                if len(translatedList) <= 0:
-                    translatedList = None
-
-                # Remove speaker
-                matchSpeakerList = re.findall(r"^(\[.+?\]\s?[|:]\s?)\s?", translatedText)
-                if len(matchSpeakerList) > 0:
-                    translatedText = translatedText.replace(matchSpeakerList[0], "")
-
-                # Textwrap
-                translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
-
-                # Set Data
-                data.insert(i, f"{translatedText}\n")
-                saveCheckLines(data, filename)
-                i += 1
-
-        # Nothing relevant. Skip Line.
-        else:
-            i += 1
-
-    # EOF
-    if len(stringList) > 0:
-        # Set Progress
-        pbar.total = len(stringList)
-        pbar.refresh()
-
-        # Translate
-        response = translateAI(stringList, "")
-        tokens[0] += response[1][0]
-        tokens[1] += response[1][1]
-        translatedList = response[0]
-
-        # Set Strings
-        if len(stringList) == len(translatedList):
-            translateWOLF(data, translatedList, pbar, filename)
-
-        # Mismatch
-        else:
-            with LOCK:
-                if filename not in MISMATCH:
-                    MISMATCH.append(filename)
-    return tokens
-
-# Save some money and enter the character before translation
-def getSpeaker(speaker):
-    match speaker:
-        case "ファイン":
-            return ["Fine", [0, 0]]
-        case "":
-            return ["", [0, 0]]
-        case _:
-            # Find Speaker
-            for i in range(len(NAMESLIST)):
-                if speaker == NAMESLIST[i][0]:
-                    return [NAMESLIST[i][1], [0, 0]]
-
-            # Translate and Store Speaker
-            response = translateAI(
-                f"{speaker}",
-                "Reply with the " + LANGUAGE + " translation of the NPC name.",
-                False,
-            )
-            response[0] = response[0].title()
-            response[0] = response[0].replace("'S", "'s")
-            response[0] = response[0].replace("Speaker: ", "")
-
-            # Retry if name doesn't translate for some reason
-            if re.search(r"([a-zA-Z？?])", response[0]) == None:
-                response = translateAI(
-                    f"{speaker}",
-                    "Reply with the " + LANGUAGE + " translation of the NPC name.",
-                    False,
-                )
-                response[0] = response[0].title()
-                response[0] = response[0].replace("'S", "'s")
-
-            speakerList = [speaker, response[0]]
-            NAMESLIST.append(speakerList)
-            return response
-    return [speaker, [0, 0]]
-
-def translateAI(text, history, history_ctx=None):
-    """
-    Legacy wrapper function for the new shared translation utility.
-    This maintains compatibility with existing code while using the new shared implementation.
-    """
-    global PBAR, MISMATCH, FILENAME
-    
-    # Update config estimate mode based on global ESTIMATE
-    TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
-    
-    # Call the new shared translation function
-    return sharedtranslateAI(
-        text=text,
-        history=history,
-        config=TRANSLATION_CONFIG,
-        filename=FILENAME,
-        pbar=PBAR,
-        lock=LOCK,
-        mismatchList=MISMATCH
-    )
+# Libraries
+import json
+import io
+import os
+import re
+import util.dazedwrap as dazedwrap
+import threading
+import time
+import traceback
+import tiktoken
+from pathlib import Path
+from colorama import Fore
+from dotenv import load_dotenv
+from retry import retry
+from tqdm import tqdm
+from util.translation import TranslationConfig, translateAI as sharedtranslateAI, getPricingConfig, calculateCost
+import tempfile
+
+# OpenAI initialization centralized in util/translation.py
+
+# Globals
+MODEL = os.getenv("model")
+TIMEOUT = int(os.getenv("timeout"))
+LANGUAGE = os.getenv("language").capitalize()
+from util.paths import PROMPT_PATH, VOCAB_PATH
+
+PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
+VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
+LOCK = threading.Lock()
+WIDTH = int(os.getenv("width"))
+LISTWIDTH = int(os.getenv("listWidth"))
+NOTEWIDTH = 70
+MAXHISTORY = 10
+ESTIMATE = ""
+TOKENS = [0, 0]
+NAMESLIST = []
+NAMES = False  # Output a list of all the character names found
+BRFLAG = False  # If the game uses <br> instead
+FIXTEXTWRAP = True  # Overwrites textwrap
+IGNORETLTEXT = False  # Ignores all translated text.
+MISMATCH = []  # Lists files that throw a mismatch error (Length of GPT list response is wrong)
+FILENAME = ""  # Current file being processed, used by translateAI wrapper
+
+# tqdm Globals
+BAR_FORMAT = "{l_bar}{bar:10}{r_bar}{bar:-10b}"
+POSITION = 0
+LEAVE = False
+
+# Regex - Need to change this if you want to translate from/to other languages. Default is Japanese Regex
+LANGREGEX = r"[一-龠ぁ-ゔァ-ヴーａ-ｚＡ-Ｚ０-９\uFF61-\uFF9F]+"
+
+# Get pricing configuration based on the model
+PRICING_CONFIG = getPricingConfig(MODEL)
+INPUTAPICOST = PRICING_CONFIG["inputAPICost"]
+OUTPUTAPICOST = PRICING_CONFIG["outputAPICost"]
+BATCHSIZE = PRICING_CONFIG["batchSize"]
+FREQUENCY_PENALTY = PRICING_CONFIG["frequencyPenalty"]
+
+# Initialize Translation Config
+TRANSLATION_CONFIG = TranslationConfig(
+    model=MODEL,
+    language=LANGUAGE,
+    prompt=PROMPT,
+    vocab=VOCAB,
+    langRegex=LANGREGEX,
+    batchSize=BATCHSIZE,
+    maxHistory=MAXHISTORY,
+    estimateMode=False  # Will be set dynamically based on ESTIMATE
+)
+LEAVE = False
+
+def handleWOLF2(filename, estimate):
+    global ESTIMATE, FILENAME
+    ESTIMATE = estimate
+    FILENAME = filename
+
+    if ESTIMATE:
+        start = time.time()
+        translatedData = openFiles(filename)
+
+        # Print Result
+        end = time.time()
+        tqdm.write(getResultString(translatedData, end - start, filename))
+        with LOCK:
+            TOKENS[0] += translatedData[1][0]
+            TOKENS[1] += translatedData[1][1]
+
+        # Print Total
+        totalString = getResultString(["", TOKENS, None], end - start, "TOTAL")
+
+        # Print any errors on maps
+        if len(MISMATCH) > 0:
+            return totalString + Fore.RED + f"\nMismatch Errors: {MISMATCH}" + Fore.RESET
+        else:
+            return totalString
+
+    else:
+        try:
+            with open("translated/" + filename, "w", encoding="shift_jis", errors="ignore") as outFile:
+                start = time.time()
+                translatedData = openFiles(filename)
+
+                # Print Result
+                end = time.time()
+                outFile.writelines(translatedData[0])
+                tqdm.write(getResultString(translatedData, end - start, filename))
+                with LOCK:
+                    TOKENS[0] += translatedData[1][0]
+                    TOKENS[1] += translatedData[1][1]
+        except Exception:
+            traceback.print_exc()
+            return "Fail"
+
+    return getResultString(["", TOKENS, None], end - start, "TOTAL")
+
+
+def getResultString(translatedData, translationTime, filename):
+    # File Print String
+    cost = calculateCost(translatedData[1][0], translatedData[1][1], MODEL)
+    totalTokenstring = (
+        Fore.YELLOW + "[Input: " + str(translatedData[1][0]) + "]"
+        "[Output: "
+        + str(translatedData[1][1])
+        + "]" "[Cost: ${:,.4f}".format(cost)
+        + "]"
+    )
+    timeString = Fore.BLUE + "[" + str(round(translationTime, 1)) + "s]"
+
+    if translatedData[2] == None:
+        # Success
+        return filename + ": " + totalTokenstring + timeString + Fore.GREEN + " \u2713 " + Fore.RESET
+
+    else:
+        # Fail
+        try:
+            raise translatedData[2]
+        except Exception as e:
+            traceback.print_exc()
+            errorString = str(e) + Fore.RED
+            return filename + ": " + totalTokenstring + timeString + Fore.RED + " \u2717 " + errorString + Fore.RESET
+
+
+def openFiles(filename):
+    # Use a robust reader to handle cp932/Shift_JIS variants and occasional bad bytes
+    def read_file_lines_with_fallback(path: str):
+        encodings = ["cp932", "shift_jis", "utf-8-sig", "utf-8"]
+        for enc in encodings:
+            try:
+                with open(path, "r", encoding=enc) as f:
+                    return f.readlines()
+            except UnicodeDecodeError:
+                continue
+        # Last resort: ignore undecodable bytes under cp932
+        with open(path, "rb") as f:
+            raw = f.read()
+        try:
+            text = raw.decode("cp932", errors="ignore")
+        except Exception:
+            text = raw.decode("latin-1", errors="ignore")
+        return text.splitlines(keepends=True)
+
+    path = os.path.join("files", filename)
+    lines = read_file_lines_with_fallback(path)
+
+    # Keep parseWOLF API by wrapping lines in a file-like object
+    with io.StringIO("".join(lines)) as readFile:
+        translatedData = parseWOLF(readFile, filename)
+
+    # Delete lines marked for deletion
+    finalData = []
+    for line in translatedData[0]:
+        if line != "\\d\n":
+            finalData.append(line)
+    translatedData[0] = finalData
+
+    return translatedData
+
+
+def parseWOLF(readFile, filename):
+    totalTokens = [0, 0]
+
+    # Read File into data
+    data = readFile.readlines()
+
+    # Create Progress Bar
+    with tqdm(bar_format=BAR_FORMAT, position=POSITION, leave=LEAVE) as pbar:
+        pbar.desc = filename
+
+        try:
+            result = translateWOLF(data, [], pbar, filename)
+            totalTokens[0] += result[0]
+            totalTokens[1] += result[1]
+        except Exception as e:
+            traceback.print_exc()
+            return [data, totalTokens, e]
+    return [data, totalTokens, None]
+
+
+def save_progress_lines(lines, filename, encoding="shift_jis"):
+    """Atomically save current line-based translation progress."""
+    try:
+        if ESTIMATE:
+            return
+        os.makedirs("translated", exist_ok=True)
+        tmp_fd, tmp_path = tempfile.mkstemp(prefix=f"{filename}.", suffix=".tmp", dir="translated")
+        try:
+            with os.fdopen(tmp_fd, "w", encoding=encoding, newline="\n", errors="ignore") as tmp_file:
+                tmp_file.writelines(lines)
+            os.replace(tmp_path, os.path.join("translated", filename))
+        finally:
+            if os.path.exists(tmp_path):
+                try:
+                    os.remove(tmp_path)
+                except OSError:
+                    pass
+    except Exception:
+        traceback.print_exc()
+
+
+def saveCheckLines(lines, filename, tokens=None, encoding="shift_jis"):
+    try:
+        if tokens is not None:
+            if not (isinstance(tokens, (list, tuple)) and len(tokens) >= 2 and (tokens[0] or tokens[1])):
+                return
+        save_progress_lines(lines, filename, encoding=encoding)
+    except Exception:
+        traceback.print_exc()
+
+
+def translateWOLF(data, translatedList, pbar, filename):
+    stringList = []
+    currentGroup = []
+    tokens = [0, 0]
+    speaker = ""
+    global LOCK, ESTIMATE, PBAR
+    PBAR = pbar
+    i = 0
+
+    while i < len(data):
+        # Speaker
+        matchList = re.findall(r"^([^/]*)：", data[i])
+        if len(matchList) != 0:
+            response = getSpeaker(matchList[0])
+            speaker = response[0]
+            tokens[0] += response[1][0]
+            tokens[1] += response[1][1]
+            data[i] = data[i].replace(matchList[0], f"{speaker}")
+            saveCheckLines(data, filename)
+            i += 1
+        else:
+            speaker = ""
+
+        # Options
+        if "//選択肢" in data[i]:
+            i += 1
+            choiceList = []
+            initialIndex = i
+            while "//" in data[i] and "の場合" not in data[i]:
+                choiceList.append(re.search(r"\/\/(.*)", data[i]).group(1))
+                i += 1
+
+            # Translate
+            response = translateAI(choiceList, "This will be a dialogue option")
+            tokens[0] += response[1][0]
+            tokens[1] += response[1][1]
+            choiceListTL = response[0]
+
+            # Set Data
+            if len(choiceList) == len(choiceListTL):
+                # Set Data
+                i = initialIndex
+                while "//" in data[i] and "の場合" not in data[i]:
+                    choiceListTL[0] = choiceListTL[0].replace(", ", "、")
+                    data[i] = f"//{choiceListTL[0]}\n"
+                    choiceListTL.pop(0)
+                    i += 1
+                    saveCheckLines(data, filename)
+
+            # Mismatch
+            else:
+                with LOCK:
+                    if filename not in MISMATCH:
+                        MISMATCH.append(filename)
+
+        # Lines
+        if r"/" not in data[i] and "@" not in data[i] and data[i] != "\n":
+            # Pass 1
+            if translatedList == []:
+                # Grab Consecutive Strings
+                currentGroup.append(data[i])
+                i += 1
+                while i < len(data) and r"/" not in data[i] and "@" not in data[i] and data[i] != "\n":
+                    currentGroup.append(data[i])
+                    i += 1
+
+                # Join up 401 groups for better translation.
+                if len(currentGroup) > 0:
+                    jaString = "".join(currentGroup)
+                    currentGroup = []
+
+                # Remove any textwrap
+                jaString = jaString.replace("\n", " ")
+
+                # Add Speaker (If there is one)
+                if speaker != "":
+                    jaString = f"[{speaker}]: {jaString}"
+
+                # Add String
+                stringList.append(jaString)
+                i += 1
+
+            # Pass 2
+            else:
+                # Insert Strings
+                while i < len(data) and r"/" not in data[i] and "@" not in data[i] and data[i] != "\n":
+                    data.pop(i)
+
+                # Get Text
+                translatedText = translatedList[0]
+                translatedList.pop(0)
+
+                if len(translatedList) <= 0:
+                    translatedList = None
+
+                # Remove speaker
+                matchSpeakerList = re.findall(r"^(\[.+?\]\s?[|:]\s?)\s?", translatedText)
+                if len(matchSpeakerList) > 0:
+                    translatedText = translatedText.replace(matchSpeakerList[0], "")
+
+                # Textwrap
+                translatedText = dazedwrap.wrapText(translatedText, width=WIDTH)
+
+                # Set Data
+                data.insert(i, f"{translatedText}\n")
+                saveCheckLines(data, filename)
+                i += 1
+
+        # Nothing relevant. Skip Line.
+        else:
+            i += 1
+
+    # EOF
+    if len(stringList) > 0:
+        # Set Progress
+        pbar.total = len(stringList)
+        pbar.refresh()
+
+        # Translate
+        response = translateAI(stringList, "")
+        tokens[0] += response[1][0]
+        tokens[1] += response[1][1]
+        translatedList = response[0]
+
+        # Set Strings
+        if len(stringList) == len(translatedList):
+            translateWOLF(data, translatedList, pbar, filename)
+
+        # Mismatch
+        else:
+            with LOCK:
+                if filename not in MISMATCH:
+                    MISMATCH.append(filename)
+    return tokens
+
+# Save some money and enter the character before translation
+def getSpeaker(speaker):
+    match speaker:
+        case "ファイン":
+            return ["Fine", [0, 0]]
+        case "":
+            return ["", [0, 0]]
+        case _:
+            # Find Speaker
+            for i in range(len(NAMESLIST)):
+                if speaker == NAMESLIST[i][0]:
+                    return [NAMESLIST[i][1], [0, 0]]
+
+            # Translate and Store Speaker
+            response = translateAI(
+                f"{speaker}",
+                "Reply with the " + LANGUAGE + " translation of the NPC name.",
+                False,
+            )
+            response[0] = response[0].title()
+            response[0] = response[0].replace("'S", "'s")
+            response[0] = response[0].replace("Speaker: ", "")
+
+            # Retry if name doesn't translate for some reason
+            if re.search(r"([a-zA-Z？?])", response[0]) == None:
+                response = translateAI(
+                    f"{speaker}",
+                    "Reply with the " + LANGUAGE + " translation of the NPC name.",
+                    False,
+                )
+                response[0] = response[0].title()
+                response[0] = response[0].replace("'S", "'s")
+
+            speakerList = [speaker, response[0]]
+            NAMESLIST.append(speakerList)
+            return response
+    return [speaker, [0, 0]]
+
+def translateAI(text, history, history_ctx=None):
+    """
+    Legacy wrapper function for the new shared translation utility.
+    This maintains compatibility with existing code while using the new shared implementation.
+    """
+    global PBAR, MISMATCH, FILENAME
+    
+    # Update config estimate mode based on global ESTIMATE
+    TRANSLATION_CONFIG.estimateMode = bool(ESTIMATE)
+    
+    # Call the new shared translation function
+    return sharedtranslateAI(
+        text=text,
+        history=history,
+        config=TRANSLATION_CONFIG,
+        filename=FILENAME,
+        pbar=PBAR,
+        lock=LOCK,
+        mismatchList=MISMATCH
+    )

modules/yuris.py

@@ -21,8 +21,10 @@ from util.translation import (
 
 MODEL = os.getenv("model")
 LANGUAGE = os.getenv("language").capitalize()
-PROMPT = Path("prompt.txt").read_text(encoding="utf-8")
-VOCAB = Path("vocab.txt").read_text(encoding="utf-8")
+from util.paths import PROMPT_PATH, VOCAB_PATH
+
+PROMPT = PROMPT_PATH.read_text(encoding="utf-8")
+VOCAB = VOCAB_PATH.read_text(encoding="utf-8")
 LOCK = threading.Lock()
 WIDTH = int(os.getenv("width"))
 MAXHISTORY = 10

scripts/launch.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+# Launcher for DazedMTLTool.desktop (finds venv, then starts the GUI).
+ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$ROOT" || exit 1
+
+PYTHON="python3"
+if [[ -x "$ROOT/.venv/bin/python" ]]; then
+    PYTHON="$ROOT/.venv/bin/python"
+elif [[ -x "$ROOT/venv/bin/python" ]]; then
+    PYTHON="$ROOT/venv/bin/python"
+fi
+
+exec "$PYTHON" "$ROOT/scripts/start_gui.py"

scripts/start_gui.py

@@ -1,72 +1,74 @@
-#!/usr/bin/env python3
-"""
-Launch script for DazedMTLTool GUI
-"""
-
-import sys
-import os
-from pathlib import Path
-
-# Add the project root to Python path
-project_root = Path(__file__).parent
-sys.path.insert(0, str(project_root))
-
-def check_dependencies():
-    """Check if required dependencies are installed."""
-    missing_deps = []
-    
-    try:
-        import PyQt5
-    except ImportError:
-        missing_deps.append("PyQt5")
-        
-    try:
-        from dotenv import load_dotenv
-    except ImportError:
-        missing_deps.append("python-dotenv")
-        
-    if missing_deps:
-        print("Missing dependencies:")
-        for dep in missing_deps:
-            print(f"  - {dep}")
-        print("\nPlease install them using:")
-        print("  pip install -r requirements_gui.txt")
-        return False
-        
-    return True
-
-def main():
-    """Main entry point."""
-    print("DazedMTLTool GUI Launcher")
-    print("=" * 40)
-    
-    # Check dependencies
-    if not check_dependencies():
-        sys.exit(1)
-
-    try:
-        from util.ace.update_tools import seed_ace_tools
-        seed_ace_tools()
-    except Exception as exc:
-        print(f"Warning: Ace tool setup failed ({exc}). Ace features may be unavailable.")
-
-    try:
-        from util.forge.update_tools import seed_forge_plugins
-        seed_forge_plugins()
-    except Exception as exc:
-        print(f"Warning: Forge plugin setup failed ({exc}). Playtest Forge may be unavailable.")
-        
-    # Import and run GUI
-    try:
-        from gui.main import main as gui_main
-        gui_main()
-    except ImportError as e:
-        print(f"Error importing GUI modules: {e}")
-        print("Make sure all GUI files are in the 'gui' directory")
-        sys.exit(1)
-    except Exception as e:
-        print(f"Error starting GUI: {e}")
-        sys.exit(1)
-
-if __name__ == "__main__":
-    main()
+#!/usr/bin/env python3
+"""Launch script for DazedMTLTool GUI."""
+
+import sys
+from pathlib import Path
+
+PROJECT_ROOT = Path(__file__).resolve().parent.parent
+sys.path.insert(0, str(PROJECT_ROOT))
+
+
+def check_dependencies():
+    """Check if required dependencies are installed."""
+    missing_deps = []
+
+    try:
+        import PyQt5  # noqa: F401
+    except ImportError:
+        missing_deps.append("PyQt5")
+
+    try:
+        from dotenv import load_dotenv  # noqa: F401
+    except ImportError:
+        missing_deps.append("python-dotenv")
+
+    if missing_deps:
+        print("Missing dependencies:")
+        for dep in missing_deps:
+            print(f"  - {dep}")
+        print("\nPlease install them using:")
+        print("  pip install -r requirements.txt")
+        return False
+
+    return True
+
+
+def main():
+    """Main entry point."""
+    print("DazedMTLTool GUI Launcher")
+    print("=" * 40)
+
+    from util.paths import ensure_vocab_file, migrate_root_data_files
+
+    migrate_root_data_files()
+    ensure_vocab_file()
+
+    if not check_dependencies():
+        sys.exit(1)
+
+    try:
+        from util.ace.update_tools import seed_ace_tools
+        seed_ace_tools()
+    except Exception as exc:
+        print(f"Warning: Ace tool setup failed ({exc}). Ace features may be unavailable.")
+
+    try:
+        from util.forge.update_tools import seed_forge_plugins
+        seed_forge_plugins()
+    except Exception as exc:
+        print(f"Warning: Forge plugin setup failed ({exc}). Playtest Forge may be unavailable.")
+
+    try:
+        from gui.main import main as gui_main
+        gui_main()
+    except ImportError as e:
+        print(f"Error importing GUI modules: {e}")
+        print("Make sure all GUI files are in the 'gui' directory")
+        sys.exit(1)
+    except Exception as e:
+        print(f"Error starting GUI: {e}")
+        sys.exit(1)
+
+
+if __name__ == "__main__":
+    main()

util/defaults.py

@@ -1,7 +1,9 @@
 import re
 from copy import deepcopy
 
-# Canonical defaults used by the GUI and the `set_defaults` script.
+from util.paths import PROJECT_ROOT
+
+# Canonical defaults used by the GUI and the pre-commit hook.
 # Stored as booleans for easier consumption by the GUI.
 DEFAULTS = {
     'FIRSTLINESPEAKERS': False,
@@ -56,4 +58,4 @@ def set_defaults(file_path):
 
 
 if __name__ == "__main__":
-    set_defaults('modules/rpgmakermvmz.py')
+    set_defaults(PROJECT_ROOT / "modules/rpgmakermvmz.py")

util/linux_desktop.py

@@ -0,0 +1,72 @@
+"""Install/update the Linux .desktop entry for taskbar and window icons."""
+
+from __future__ import annotations
+
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+from util.paths import ICON_PATH, PROJECT_ROOT
+
+DESKTOP_ID = "DazedMTLTool"
+LAUNCH_SCRIPT = PROJECT_ROOT / "scripts" / "launch.sh"
+
+
+def installed_desktop_path() -> Path:
+    data_home = os.environ.get("XDG_DATA_HOME", "")
+    if data_home:
+        base = Path(data_home)
+    else:
+        base = Path.home() / ".local" / "share"
+    return base / "applications" / f"{DESKTOP_ID}.desktop"
+
+
+def _desktop_content(root: Path, icon: Path, launch: Path) -> str:
+    return (
+        "[Desktop Entry]\n"
+        "Type=Application\n"
+        f"Name={DESKTOP_ID}\n"
+        "GenericName=Translation Tool\n"
+        "Comment=AI translation tool for visual novels and RPG games\n"
+        f"Exec={launch}\n"
+        f"Icon={icon}\n"
+        f"Path={root}\n"
+        "Terminal=false\n"
+        "Categories=Development;Utility;\n"
+        f"StartupWMClass={DESKTOP_ID}\n"
+    )
+
+
+def ensure_linux_desktop_entry() -> None:
+    """Write ~/.local/share/applications/DazedMTLTool.desktop with absolute paths."""
+    if not sys.platform.startswith("linux"):
+        return
+    if not LAUNCH_SCRIPT.is_file() or not ICON_PATH.is_file():
+        return
+
+    desktop_path = installed_desktop_path()
+    content = _desktop_content(PROJECT_ROOT, ICON_PATH, LAUNCH_SCRIPT)
+
+    try:
+        if desktop_path.is_file() and desktop_path.read_text(encoding="utf-8") == content:
+            return
+        desktop_path.parent.mkdir(parents=True, exist_ok=True)
+        desktop_path.write_text(content, encoding="utf-8")
+        desktop_path.chmod(0o755)
+    except OSError as exc:
+        print(f"Warning: could not install Linux desktop entry ({exc})")
+        return
+
+    _refresh_desktop_database(desktop_path.parent)
+
+
+def _refresh_desktop_database(applications_dir: Path) -> None:
+    for cmd in (
+        ["update-desktop-database", str(applications_dir)],
+        ["kbuildsycoca5", "--noincremental"],
+    ):
+        try:
+            subprocess.run(cmd, check=False, stdout=subprocess.DEVNULL, stderr=subprocess.DEVNULL)
+        except OSError:
+            pass

util/paths.py

@@ -0,0 +1,45 @@
+"""Canonical project paths (repo root, data files, config)."""
+
+from __future__ import annotations
+
+from pathlib import Path
+
+PROJECT_ROOT = Path(__file__).resolve().parent.parent
+DATA_DIR = PROJECT_ROOT / "data"
+VOCAB_PATH = DATA_DIR / "vocab.txt"
+VOCAB_BASE_PATH = DATA_DIR / "vocab_base.txt"
+PROMPT_PATH = DATA_DIR / "prompt.txt"
+LAST_UPDATE_SHA_PATH = DATA_DIR / "last_update_sha.txt"
+ENV_PATH = PROJECT_ROOT / ".env"
+ICON_PATH = PROJECT_ROOT / "assets" / "icon.png"
+
+_ROOT_DATA_FILES = (
+    "vocab.txt",
+    "vocab_base.txt",
+    "prompt.txt",
+    "last_update_sha.txt",
+)
+
+
+def migrate_root_data_files() -> None:
+    """Move legacy root-level data files into data/ on first run."""
+    DATA_DIR.mkdir(parents=True, exist_ok=True)
+    for name in _ROOT_DATA_FILES:
+        src = PROJECT_ROOT / name
+        dst = DATA_DIR / name
+        if src.is_file() and not dst.exists():
+            src.rename(dst)
+
+
+migrate_root_data_files()
+
+
+def ensure_vocab_file() -> None:
+    """Create data/vocab.txt from vocab_base.txt when missing."""
+    migrate_root_data_files()
+    if VOCAB_PATH.is_file():
+        return
+    if VOCAB_BASE_PATH.is_file():
+        VOCAB_PATH.write_text(VOCAB_BASE_PATH.read_text(encoding="utf-8"), encoding="utf-8")
+    else:
+        VOCAB_PATH.write_text("", encoding="utf-8")