Engineering docs / Phase 2Audio → DSP control program
AI GUITAR PEDAL / V4 / WHISPER BASE

From a guitar recording
to DSP control instructions.

Follow the data through this training project: how samples are paired, how stereo audio enters Whisper, how program tokens become learning targets, and how the model is evaluated and saved. Each stage explains the logic first, then shows the actual code.

Task Audio-to-program Seq2SeqScope V4 Phase 2 trainingPipeline 15 stages
3,206DSP vocabulary size
160Combined left/right Mel bins
6 sDefault waveform crop length
768Maximum target token length

Training flowchart

Read in order · Select a stage for its logic and code
DryRun → stops after Python resolves paths, before scanning dataValidateOnly → stops after the data split12 Resume and callbacks ↔ 13 Training loop: memory / PAUSE

14 Evaluation runs at intervals during 13 Training, then once more after training returns. PAUSE saves state and stops the training loop; final evaluation and export still run.

How one sample changes shape

Audio [T,C]→Waveform [2,96000]→Features [160,3000]→logits [B,L,3206]

With the default Whisper Base feature configuration, the waveform is first cropped to 6 seconds, then padded to 30 seconds by the feature extractor. Target labels have shape [B,L], where L ≤ 768 and B is the batch size.

Training uses processed guitar audio as input, with supervision from target_program_json.model_token_ids. Data rendering, program serialization, and real-time effects inference are outside the current V4 project.
PROJECT MAP

File structure and roles

Read the project by entry points, training implementation, fixed resources, runtime outputs, and documentation. Training code stays in Main, while common commands use the unified entry point in scripts.

AI_Guitar_Pedal_V4/
├── README.md                 Project navigation
├── scripts/
│   ├── train.ps1             Unified training entry point
│   └── README.md             Entry point guide
├── Main/
│   ├── train_program_seq2seq.py
│   ├── requirements.txt
│   ├── ai_guitar_pedal/
│   │   ├── __init__.py          Package marker
│   │   ├── phase2_training_utils.py
│   │   ├── storage_redirect.py
│   │   ├── memory_guard.py
│   │   └── tokenizer/
│   │       ├── __init__.py      Package marker
│   │       └── dsp_control_tokenizer.py
│   ├── resources/
│   │   └── dsp_control_vocab.json
│   ├── tools/
│   │   └── run_phase2_full_train_safe.ps1
│   │                         Legacy entry point compatibility
│   └── storage/              Runtime outputs (ignored)
│       ├── artifacts/
│       │   ├── checkpoints/<experiment>/
│       │   ├── exports/<experiment>/
│       │   ├── runs/<experiment>/
│       │   └── logs/
│       ├── cache/whisper/features/
│       ├── data/
│       └── tmp/
└── docs/
    ├── README.md             Documentation navigation
    ├── reference/            Original migration notes
    ├── maintenance/          Organization and validation records
    └── site/
        ├── guide.json / guide.en.json   Chinese/English explanations and code references
        ├── template.html / template.en.html Page template
        ├── style.css / app.js
        ├── build.py          Build the page from source
        ├── verify.py         Check links and source consistency
        ├── dependency-excerpts.json
        │                     Dependency source excerpts
        └── dist/             Open directly or host
            ├── index.html / en.html
            ├── source/       Source with addressable line numbers
            └── source-manifest.json
File / directoryRole in training
scripts/train.ps1Resolves data locations, configures cache directories, selects Python, records logs, and invokes the training script.
train_program_seq2seq.pyMain control flow: label loading, data splitting, Dataset/Collator, model construction, Trainer, evaluation, and export.
phase2_training_utils.pyStereo processing, feature caching, first-convolution adaptation, pause, and resume.
tokenizer/ + resources/Strict DSP token↔ID mapping; the vocabulary defines the output protocol.
storage_redirect.pyCentralizes output and dependency cache directories, and checks that outputs are under the specified root.
memory_guard.pyChecks RAM/VRAM usage and emits warnings at configured intervals.
Main/storage/Created at runtime; holds checkpoints, exports, logs, and feature caches.
docs/site/dist/Built website. Includes snapshots of training code, without datasets or model weights.
requirements.txtTraining dependencies; versions are currently unpinned.
__init__.pyMarks Python packages so the training entry point can import their modules.
.gitignoreExcludes virtual environments, Python caches, and Main/storage runtime outputs.

External read-only inputs: the Wet_Main manifest, audio, and Presets/programs under the sibling AI_Guitar_Pedal_V2/Audio directory. V4 includes its own vocabulary; -DataProjectRoot can select another complete data project.

STEP 01 / Preparation

01Launch entry point and storage paths

Resolve where the data is read from and where training artifacts are written before entering the main Python workflow.

Input
PowerShell arguments + data project path
Output
Training command, cache environment, artifact directories
Core files
scripts/train.ps1 · storage_redirect.py

Logic · How it works

Run scripts/train.ps1 from the project root. The launcher first checks V4's own Audio/Samples/Wet_Main/manifest_full.jsonl, then tries the sibling V2 project if it is missing. You can also pass -DataProjectRoot explicitly. Audio and program JSON files are read from the data project; training artifacts are written to V4's Main/storage by default.

The launcher configures directories for Hugging Face, Torch, pip, temporary files, and other storage. It prefers V4's Main/.venv, then the data project's virtual environment, and finally the system Python. Before importing heavy dependencies, the training script also bootstraps the storage environment from --e_root.

-DryRun shows the resolved command. It still starts Python, imports dependencies, and creates directories, but exits before scanning the data or loading the model. -ValidateOnly continues through data validation and splitting before exiting; it likewise does not extract features or load Whisper weights.

Code · Core implementation

Training entry point · choosing the data project
$scriptDir = Split-Path -Parent $MyInvocation.MyCommand.Path
$projectDir = (Resolve-Path (Join-Path $scriptDir "..")).Path
$mainDir = (Resolve-Path (Join-Path $projectDir "Main")).Path
$workspaceDir = Split-Path -Parent $projectDir

if (-not $DataProjectRoot) {
    $localManifest = Join-Path $projectDir "Audio\Samples\Wet_Main\manifest_full.jsonl"
    $v2ProjectRoot = Join-Path $workspaceDir "AI_Guitar_Pedal_V2"
    $v2Manifest = Join-Path $v2ProjectRoot "Audio\Samples\Wet_Main\manifest_full.jsonl"
    if (Test-Path $localManifest) {
        $DataProjectRoot = $projectDir
    } elseif (Test-Path $v2Manifest) {
        $DataProjectRoot = $v2ProjectRoot
    } else {
        throw "No complete Phase 2 data tree found. Pass -DataProjectRoot explicitly."
    }
}
$DataProjectRoot = Resolve-ExistingPath -PathValue $DataProjectRoot
scripts/train.ps1 · L67–L84 ↗
Set up storage before importing heavy dependencies
def bootstrap_storage_env_from_argv(argv: Sequence[str]) -> None:
    raw_e_root = early_arg_value(argv, "--e_root") or E_ROOT
    try:
        e_root = resolve_e_root(raw_e_root)
        storage = ensure_storage_tree(e_root)
        apply_env_defaults(storage, set_temp_tmp=True)
    except Exception:
        return
Main/train_program_seq2seq.py · L34–L41 ↗
STEP 02 / Data

02Define the output language the model will learn

A fixed vocabulary maps effect structures, parameter names, and discrete values to 3,206 IDs.

Input
dsp_control_vocab.json
Output
token ↔ ID; PAD/BOS/EOS = 0/1/2
Core class
DSPControlTokenizer

Logic · How it works

V4 uses a dedicated DSP control vocabulary. Each token represents a structural marker, effect type, parameter name, or discrete value. At every output position, the model predicts one of 3,206 classes. It neither transcribes natural language nor directly predicts an audio waveform.

When the tokenizer is constructed, IDs must be unique, contiguous from 0, and include PAD, BOS, and EOS. encode_tokens and decode_ids perform strict dictionary lookups; unknown tokens or IDs immediately raise an error. Training primarily reads the already encoded model_token_ids.

The real program excerpt below first declares a chain, then a gate effect, supplies each parameter and value, and finally closes the effect and the full chain. Q_0169 is a quantization symbol; its physical unit and dequantization rules are defined by the external DSP protocol.

Symbol groupCountMeaning
Structural symbols6PAD, BOS, EOS, CHAIN, END_CHAIN, END_FX
FX / CAT / P141 / 32 / 1,740Effects / categories / parameter names
B / I / E2 / 63 / 221Boolean / integer / enumerated values
Q1,001Q_0000 … Q_1000 quantized values
<BOS> <CHAIN> <FX_19000> <CAT_GATE>
<P_19000_A> <Q_0000> … <P_19000_T> <Q_0169>
<END_FX> … <END_CHAIN> <EOS>

Example excerpted from the external V2 programs/0001.json. Some parameters and effects are omitted; this is not a complete, directly executable program.

Code · Core implementation

DSPControlTokenizer.__init__
    def __init__(self, token_to_id: dict[str, int]):
        if not token_to_id:
            raise ValueError("DSP control vocabulary is empty")
        ids = list(token_to_id.values())
        if len(ids) != len(set(ids)):
            raise ValueError("DSP control vocabulary contains duplicate token IDs")
        if sorted(ids) != list(range(len(ids))):
            raise ValueError("DSP control token IDs must be contiguous from zero")

        self.token_to_id = dict(token_to_id)
        self.id_to_token = {token_id: token for token, token_id in token_to_id.items()}
        self.vocab_size = len(token_to_id)
        self.pad_token_id = self._required_id("<PAD>")
        self.bos_token_id = self._required_id("<BOS>")
        self.eos_token_id = self._required_id("<EOS>")
Main/ai_guitar_pedal/tokenizer/dsp_control_tokenizer.py · L11–L25 ↗
DSPControlTokenizer.decode_ids
    def decode_ids(self, token_ids: Iterable[int]) -> list[str]:
        tokens = []
        for token_id in token_ids:
            token_id = int(token_id)
            if token_id not in self.id_to_token:
                raise ValueError(f"Unknown DSP control token ID: {token_id}")
            tokens.append(self.id_to_token[token_id])
        return tokens
Main/ai_guitar_pedal/tokenizer/dsp_control_tokenizer.py · L43–L50 ↗
STEP 03 / Data

03Connect each sample to its target program

Each manifest row binds an audio path to an already serialized DSP program.

Input
manifest JSONL + program JSON
Output
audio, labels, program_hash, target_length
Core functions
load_program_rows → read_program_token_ids

Logic · How it works

The training manifest is read line by line for audio_filepath and target_program_json. Relative paths are resolved against the directory containing the manifest. If the original program path does not exist, the loader tries program_root / filename. Audio paths do not have the same filename fallback.

Targets come from model_token_ids in the program JSON. By default, the leading BOS is removed, the final token must be EOS, and every token ID is checked against the vocabulary range. Samples longer than max_target_length fail validation rather than being silently truncated.

Each program JSON only needs to be parsed once: the loader caches its labels, length, and SHA-256 hash. Missing files and invalid fields are collected into a combined report. The upstream processes of rendering audio and converting effect parameters into model_token_ids are outside V4.

Code · Core implementation

read_program_token_ids
def read_program_token_ids(
    program_path: Path,
    tokenizer: DSPControlTokenizer,
    strip_bos: bool,
) -> List[int]:
    payload = json.loads(program_path.read_text(encoding="utf-8"))
    token_ids = payload.get("model_token_ids")
    if not isinstance(token_ids, list) or not token_ids:
        raise ValueError(f"program missing model_token_ids: {program_path}")
    out = [int(x) for x in token_ids]
    if strip_bos and out and out[0] == tokenizer.bos_token_id:
        out = out[1:]
    if not out or out[-1] != tokenizer.eos_token_id:
        raise ValueError(f"program token sequence must end with EOS: {program_path}")
    bad = [x for x in out if x < 0 or x >= tokenizer.vocab_size]
    if bad:
        raise ValueError(f"program token id out of vocabulary: {program_path}, examples={bad[:10]}")
    return out
Main/train_program_seq2seq.py · L255–L272 ↗
Assemble each training sample
            cached = program_cache[program_path]
            rows.append(
                {
                    "audio": str(audio_path),
                    "labels": cached["labels"],
                    "program_hash": cached["program_hash"],
                    "target_length": cached["target_length"],
                }
            )
Main/train_program_seq2seq.py · L339–L347 ↗
STEP 04 / Data

04Separate training and validation by complete program

Different audio samples associated with the same DSP program stay in the same split.

Input
All rows containing labels and program_hash
Output
train_rows / eval_rows + split manifests
Defaults
seed = 42; eval_ratio = 0.02

Logic · How it works

The integers in the full labels sequence are joined with commas and hashed with SHA-256 to produce program_hash. Programs with equivalent semantics but different token sequences still receive different hashes: groups are defined by exact sequence equality.

Samples with the same hash are grouped together. The sorted group keys are shuffled with seed=42. By default, approximately 2% of program groups form the validation set and the rest form the training set. Where possible, at least one validation group and one training group are retained. There is no validation set when eval_ratio=0 or only one group exists.

After splitting, the sample rows are shuffled separately. train_split_desensitized.jsonl and eval_split_desensitized.jsonl are written to the experiment's checkpoint directory, recording audio paths, hashes, and lengths. ValidateOnly ends here.

Code · Core implementation

token_hash
def token_hash(token_ids: Sequence[int]) -> str:
    text = ",".join(str(int(x)) for x in token_ids)
    return hashlib.sha256(text.encode("utf-8")).hexdigest()
Main/train_program_seq2seq.py · L250–L252 ↗
split_by_program_hash
def split_by_program_hash(
    rows: Sequence[Dict[str, Any]],
    eval_ratio: float,
    seed: int,
) -> Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]:
    if not (0.0 <= eval_ratio < 1.0):
        raise ValueError(f"eval_ratio must be in [0,1), got {eval_ratio}")
    grouped: Dict[str, List[Dict[str, Any]]] = defaultdict(list)
    for row in rows:
        grouped[str(row["program_hash"])].append(row)

    keys = sorted(grouped)
    rng = random.Random(seed)
    rng.shuffle(keys)
    if eval_ratio <= 0.0 or len(keys) <= 1:
        eval_keys = set()
    else:
        eval_count = int(round(len(keys) * eval_ratio))
        eval_count = max(1, eval_count)
        eval_count = min(eval_count, len(keys) - 1)
        eval_keys = set(keys[:eval_count])

    train_rows: List[Dict[str, Any]] = []
    eval_rows: List[Dict[str, Any]] = []
    for key in keys:
        if key in eval_keys:
            eval_rows.extend(grouped[key])
        else:
            train_rows.extend(grouped[key])
    rng.shuffle(train_rows)
    rng.shuffle(eval_rows)
    if not train_rows:
        raise ValueError("No train rows after program-hash split.")
    return train_rows, eval_rows
Main/train_program_seq2seq.py · L367–L400 ↗
STEP 05 / Features

05Standardize audio to two channels at 16 kHz

Preserve the left and right channels, take the first 6 seconds, and pad shorter clips with zeros at the end.

Input
wet WAV specified in the manifest [T,C]
Output
float32 waveform [2,96000]
Core functions
load_audio_stereo_16k / fix_audio_length

Logic · How it works

soundfile.read(always_2d=True) reads audio as [samples, channels]. Mono audio is duplicated into left and right channels; for audio with more than two channels, only the first two are retained. The array is then transposed to [2,T], and each channel is resampled to 16 kHz.

max_len = max_seconds × 16000, which defaults to 96,000 samples. Longer audio is cropped to its first 6 seconds; shorter audio is zero-padded at the end. Both channels use the same time interval and are not first mixed down to mono.

This step reads only wet audio. The manifest's dry-audio provenance fields are not used as a second model input. There is currently no random cropping, loudness normalization, or data augmentation.

Code · Core implementation

load_audio_stereo_16k
def load_audio_stereo_16k(path: Path, target_sr: int = TARGET_SR) -> np.ndarray:
    audio, sr = sf.read(str(path), always_2d=True)
    audio = np.asarray(audio, dtype=np.float32)
    if audio.size == 0:
        raise ValueError(f"Empty audio file: {path}")

    if audio.shape[1] == 1:
        audio = np.repeat(audio, STEREO_CHANNELS, axis=1)
    elif audio.shape[1] < STEREO_CHANNELS:
        raise ValueError(f"Expected at least one audio channel, got shape {audio.shape}: {path}")
    elif audio.shape[1] > STEREO_CHANNELS:
        audio = audio[:, :STEREO_CHANNELS]

    stereo = audio.T
    if int(sr) != int(target_sr):
        stereo = np.stack(
            [
                librosa.resample(channel, orig_sr=int(sr), target_sr=int(target_sr))
                for channel in stereo
            ],
            axis=0,
        )
    return np.asarray(stereo, dtype=np.float32)
Main/ai_guitar_pedal/phase2_training_utils.py · L153–L175 ↗
fix_audio_length
def fix_audio_length(waveform: np.ndarray, max_len: int) -> np.ndarray:
    if waveform.ndim != 2 or waveform.shape[0] != STEREO_CHANNELS:
        raise ValueError(f"Expected stereo waveform with shape [2, T], got {waveform.shape}")
    if waveform.shape[1] > max_len:
        waveform = waveform[:, :max_len]
    elif waveform.shape[1] < max_len:
        waveform = np.pad(
            waveform,
            ((0, 0), (0, max_len - waveform.shape[1])),
            mode="constant",
        )
    return waveform.astype(np.float32, copy=False)
Main/ai_guitar_pedal/phase2_training_utils.py · L178–L189 ↗
STEP 06 / Features

06Extract each channel's features, then concatenate along the Mel dimension

Two 80-bin Log-Mel matrices are combined into one 160-bin input.

Input
[2,96000] waveform
Output
[160,3000] float32 features (default configuration)
Layout
First 80 bins = L; last 80 bins = R

Logic · How it works

The same Whisper feature extractor performs the time-frequency transform on each channel. The default Base configuration uses a 16 kHz sample rate, a 400-sample FFT window, a 160-sample hop, and 80 Mel bins. Log-Mel compresses the spectral magnitude's dynamic range, allowing the model to read time-frequency structure.

V4 does not specify max_length or padding when calling the feature extractor. Checking the local Base preprocessor_config and the Transformers implementation confirms that the default still pads to 30 seconds (480,000 samples), producing [80,3000] per channel. Therefore, retaining 6 seconds of audio does not also reduce Whisper's input frame count.

torch.cat(channel_features, dim=0) concatenates the left and right channels along the Mel dimension to produce [160,3000]. The time axis remains 3000 frames. This step neither mixes the left and right waveforms nor joins them into 12 seconds of audio.

Left[96000] → padding 480000 → [80,3000]Right[96000] → padding 480000 → [80,3000]Concat dim=0[160,3000]

Code · Core implementation

stereo_to_features
def stereo_to_features(
    feature_extractor: AutoFeatureExtractor,
    waveform: np.ndarray,
) -> torch.Tensor:
    if waveform.ndim != 2 or waveform.shape[0] != STEREO_CHANNELS:
        raise ValueError(f"Expected stereo waveform with shape [2, T], got {waveform.shape}")

    channel_features: list[torch.Tensor] = []
    for channel in waveform:
        features = feature_extractor(
            channel,
            sampling_rate=TARGET_SR,
            return_tensors="pt",
        ).input_features
        channel_features.append(features[0].cpu().to(torch.float32))

    if len(channel_features) != STEREO_CHANNELS:
        raise ValueError(f"Expected {STEREO_CHANNELS} channel feature tensors")
    if channel_features[0].shape != channel_features[1].shape:
        raise ValueError(
            "Stereo channel feature shapes differ: "
            f"{channel_features[0].shape} vs {channel_features[1].shape}"
        )
    return torch.cat(channel_features, dim=0).to(torch.float32)
Main/ai_guitar_pedal/phase2_training_utils.py · L192–L215 ↗
STEP 07 / Features

07Avoid recomputing spectra every epoch

Read tensors directly on cache hits; compute and save .pt files on cache misses.

Input
Audio path, model_name, max_seconds
Output
CPU feature tensor + .pt cache on disk
Default switches
use_feature_cache / preextract_features = true

Logic · How it works

The cache directory is composed from the model identifier, sample rate, duration, and channel mode. Audio under audio_root receives cache names based on relative paths, with renders prefixes in the directory path normalized. External audio instead uses the SHA-1 of its absolute path. audio_root is primarily used for this cache-location process.

By default, all rows are traversed before model construction to pre-extract training and validation features. This is a fixed feature transform; the validation set is not used to fit normalization parameters. The Dataset can subsequently read the cache directly, reducing CPU work during each training epoch.

Cache writes first save a temporary file, then replace the destination using os.replace. On read, the code checks that the object is a tensor with a first dimension of 160, then converts it to float32. With caching disabled, features are recomputed whenever the Dataset accesses a sample.

Code · Core implementation

feature_cache_base_dir
def feature_cache_base_dir(
    feature_cache_root: Path,
    model_name: str,
    max_seconds: float,
) -> Path:
    model_id = normalize_model_name_for_cache(model_name)
    model_tag = re.sub(r"[^A-Za-z0-9._-]+", "_", model_id).strip("_")
    max_tag = f"{max_seconds:.3f}".replace(".", "p")
    return feature_cache_root / model_tag / f"sr{TARGET_SR}_max{max_tag}_{AUDIO_CHANNEL_MODE}"
Main/ai_guitar_pedal/phase2_training_utils.py · L298–L306 ↗
load_or_compute_input_features
def load_or_compute_input_features(
    wav_path: Path,
    feature_extractor: AutoFeatureExtractor,
    max_len: int,
    use_feature_cache: bool,
    cache_path: Path,
    save_on_miss: bool,
) -> torch.Tensor:
    if use_feature_cache and cache_path.exists():
        tensor = torch.load(cache_path, map_location="cpu")
        if not isinstance(tensor, torch.Tensor):
            raise ValueError(f"Invalid cached tensor format: {cache_path}")
        if tensor.ndim < 2 or int(tensor.shape[0]) != STEREO_FEATURE_BINS:
            raise ValueError(
                f"Cached features are not stereo {STEREO_FEATURE_BINS}-bin tensors: "
                f"{cache_path} shape={tuple(tensor.shape)}"
            )
        return tensor.to(torch.float32)

    tensor = compute_input_features(
        wav_path=wav_path,
        feature_extractor=feature_extractor,
        max_len=max_len,
    )
    if use_feature_cache and save_on_miss:
        atomic_torch_save(tensor, cache_path)
    return tensor
Main/ai_guitar_pedal/phase2_training_utils.py · L345–L371 ↗
atomic_torch_save
def atomic_torch_save(tensor: torch.Tensor, path: Path) -> None:
    path.parent.mkdir(parents=True, exist_ok=True)
    tmp = path.with_suffix(path.suffix + f".tmp_{os.getpid()}")
    torch.save(tensor.cpu(), tmp)
    os.replace(tmp, path)
Main/ai_guitar_pedal/phase2_training_utils.py · L328–L332 ↗
STEP 08 / Data assembly

08Assemble variable-length targets into a batch

Stack features directly and pad labels only to the longest target in the current batch.

Input
Per-sample input_features + labels
Output
[B,160,3000] and [B,L]
Ignore marker
labels padding = -100

Logic · How it works

ProgramSeq2SeqDataset.__getitem__ finds the audio by index, computes its cache path, and loads the features while converting program token IDs to torch.long. The Dataset does not parse program syntax again.

The Collator combines fixed-shape input features with torch.stack. Target programs may have different lengths, so it finds the maximum length L in the current batch, creates a [B,L] matrix filled with -100, and writes each sequence of actual labels into the beginning of its row.

-100 marks a supervision position to ignore. It is not one of the 3,206 DSP tokens and must not be confused with PAD ID=0. Both the subsequent cross-entropy loss and accuracy calculation skip these positions.

Code · Core implementation

ProgramSeq2SeqDataset.__getitem__
    def __getitem__(self, idx: int) -> Dict[str, torch.Tensor]:
        row = self.rows[idx]
        wav_path = Path(str(row["audio"])).expanduser().resolve()
        cache_path = feature_cache_path_for_audio(
            wav_path=wav_path,
            audio_root=self.audio_root,
            cache_base=self.cache_base,
        )
        input_features = load_or_compute_input_features(
            wav_path=wav_path,
            feature_extractor=self.feature_extractor,
            max_len=self.max_len,
            use_feature_cache=self.use_feature_cache,
            cache_path=cache_path,
            save_on_miss=self.use_feature_cache,
        )
        return {
            "input_features": input_features,
            "labels": torch.tensor(row["labels"], dtype=torch.long),
        }
Main/train_program_seq2seq.py · L442–L461 ↗
Seq2SeqCollator
class Seq2SeqCollator:
    def __init__(self, pad_token_id: int, label_pad_token_id: int = -100) -> None:
        self.pad_token_id = int(pad_token_id)
        self.label_pad_token_id = int(label_pad_token_id)

    def __call__(self, items: List[Dict[str, torch.Tensor]]) -> Dict[str, torch.Tensor]:
        input_features = torch.stack([item["input_features"] for item in items], dim=0)
        labels = [item["labels"] for item in items]
        max_len = max(int(label.shape[0]) for label in labels)
        padded = torch.full(
            (len(labels), max_len),
            self.label_pad_token_id,
            dtype=torch.long,
        )
        for idx, label in enumerate(labels):
            padded[idx, : label.shape[0]] = label
        return {"input_features": input_features, "labels": padded}
Main/train_program_seq2seq.py · L464–L480 ↗
STEP 09 / Model

09Change the speech model's output to DSP programs

Reuse compatible encoder–decoder weights, replace the target vocabulary, and expand the output position capacity.

Input
Pretrained Whisper Base weights + DSP tokenizer
Output
DSP vocabulary size: 3206; target positions ≥ 768
Architecture
Base: d_model 512; 6 encoder and 6 decoder layers

Logic · How it works

Read the model configuration and set vocab_size to 3206. PAD, BOS, EOS, and the decoder start token all follow the DSP protocol. Set the decoder's maximum position capacity to the larger of its original capacity and the requested limit; by default, this expands Base's original 448 positions to 768.

Clear the forced decoder IDs and suppressed-token lists used for speech transcription so the decoder can learn DSP tokens. Call from_pretrained(ignore_mismatched_sizes=True) to load compatible parameters; token embeddings, the output projection, or position embeddings whose dimensions have changed cannot be inherited unchanged.

Next, adapt the model's first layer for stereo input. During training, set use_cache=False and enable gradient checkpointing according to the configuration, trading extra activation recomputation during backpropagation for lower memory usage. V4 does not add LoRA or a strategy that freezes the encoder.

[B,160,3000] → conv1 [B,512,3000] → conv2 [B,512,1500] → encoder [B,1500,512] → decoder [B,L,512] → [B,L,3206]

Dimensions are based on Whisper Base. The Transformers dependency implements the encoder and decoder Transformer operations and the output projection.

Code · Core implementation

build_model
def build_model(args: argparse.Namespace, tokenizer: DSPControlTokenizer) -> WhisperForConditionalGeneration:
    config = WhisperConfig.from_pretrained(args.model_name)
    requested_target_positions = int(args.max_target_length)
    if requested_target_positions <= 0:
        raise ValueError(f"max_target_length must be positive, got {requested_target_positions}")
    config.max_target_positions = max(
        requested_target_positions,
        int(getattr(config, "max_target_positions", 0) or 0),
    )
    config.vocab_size = int(tokenizer.vocab_size)
    config.pad_token_id = int(tokenizer.pad_token_id)
    config.bos_token_id = int(tokenizer.bos_token_id)
    config.eos_token_id = int(tokenizer.eos_token_id)
    config.decoder_start_token_id = int(tokenizer.bos_token_id)
    config.forced_decoder_ids = None
    config.suppress_tokens = []
    config.begin_suppress_tokens = []

    model = WhisperForConditionalGeneration.from_pretrained(
        args.model_name,
        config=config,
        ignore_mismatched_sizes=True,
    )
    adapt_whisper_model_for_stereo(model)
    model.config.use_cache = False
    if hasattr(model, "generation_config"):
        model.generation_config.pad_token_id = int(tokenizer.pad_token_id)
        model.generation_config.bos_token_id = int(tokenizer.bos_token_id)
        model.generation_config.eos_token_id = int(tokenizer.eos_token_id)
        model.generation_config.decoder_start_token_id = int(tokenizer.bos_token_id)
        model.generation_config.forced_decoder_ids = None
        model.generation_config.suppress_tokens = []
        model.generation_config.begin_suppress_tokens = []
        model.generation_config.max_length = int(args.max_target_length)
    if args.gradient_checkpointing:
        model.gradient_checkpointing_enable()
    return model
Main/train_program_seq2seq.py · L507–L543 ↗
STEP 10 / Model

10Let the first layer accept left and right channels

Expand conv1's input channels and initialize each half with half of the original weights.

Input
Original conv1: 80 input channels
Output
New conv1: 160 input channels
Initialization
W_left = W_right = 0.5 × W_old

Logic · How it works

If the model's first layer already has 160 input channels, update the configuration and return. If it has the original 80 channels, create a Conv1d with 160 input channels, preserving the output channels, kernel, stride, padding, device, and data type.

Copy the original convolution weights into both the first 80 and the last 80 input channels, multiplying each copy by 0.5 and leaving the bias unchanged. When the left and right features are identical, the new layer produces the same output as the original mono convolution, avoiding the doubled response that simple duplication would cause.

Finally, replace encoder.conv1 and set config.num_mel_bins to 160. The two halves can learn separate weights during training; the initialization itself does not mean the model has already learned spatial relationships.

Conv([L; R]) = 0.5 · W ∗ L + 0.5 · W ∗ R + b

Code · Core implementation

adapt_whisper_model_for_stereo
def adapt_whisper_model_for_stereo(model: torch.nn.Module) -> None:
    conv1 = whisper_encoder_first_conv(model)
    if conv1.in_channels == STEREO_FEATURE_BINS:
        setattr(model.config, "num_mel_bins", STEREO_FEATURE_BINS)
        return
    if conv1.in_channels != MONO_FEATURE_BINS:
        raise RuntimeError(
            f"Unsupported Whisper conv1 input channels: {conv1.in_channels}; "
            f"expected {MONO_FEATURE_BINS} or {STEREO_FEATURE_BINS}"
        )

    new_conv = torch.nn.Conv1d(
        in_channels=STEREO_FEATURE_BINS,
        out_channels=conv1.out_channels,
        kernel_size=conv1.kernel_size,
        stride=conv1.stride,
        padding=conv1.padding,
        dilation=conv1.dilation,
        groups=conv1.groups,
        bias=conv1.bias is not None,
        padding_mode=conv1.padding_mode,
        device=conv1.weight.device,
        dtype=conv1.weight.dtype,
    )
    with torch.no_grad():
        new_conv.weight[:, :MONO_FEATURE_BINS, :] = conv1.weight * 0.5
        new_conv.weight[:, MONO_FEATURE_BINS:, :] = conv1.weight * 0.5
        if conv1.bias is not None and new_conv.bias is not None:
            new_conv.bias.copy_(conv1.bias)

    candidates = [
        getattr(model, "encoder", None),
        getattr(getattr(model, "model", None), "encoder", None),
        getattr(getattr(model, "whisper", None), "encoder", None),
    ]
    for encoder in candidates:
        if encoder is not None and getattr(encoder, "conv1", None) is conv1:
            encoder.conv1 = new_conv
            break
    setattr(model.config, "num_mel_bins", STEREO_FEATURE_BINS)
Main/ai_guitar_pedal/phase2_training_utils.py · L231–L270 ↗
STEP 11 / Learning objective

11Predict the next program token from the true prefix

The loss is token cross-entropy. The dependency inserts BOS when it shifts labels to the right.

Input
Audio features + ground-truth labels [B,L]
Output
logits [B,L,3206] + scalar loss
Implementation
V4 configuration / Transformers internal forward

Logic · How it works

V4 passes input_features and labels to Trainer. When no explicit decoder inputs are supplied, Whisper's forward shifts the labels one position to the right and inserts BOS at the start. At each position, the decoder receives the true preceding tokens: this is teacher forcing.

By default, the initial BOS is removed from labels and the final EOS is retained. The learning target is therefore <CHAIN> … <EOS>, while the input prefix is <BOS> <CHAIN> …. The model must learn both the program's content and when to stop.

The output projection converts each position's hidden state into scores for 3206 classes. The dependency's CrossEntropyLoss() computes the negative log-probability of the true ID and ignores -100; structural symbols and parameter tokens both contribute. V4 does not render audio through DSP to compute a differentiable reconstruction loss.

labels<CHAIN> <FX_19000> … <END_CHAIN> <EOS>decoder input<BOS> <CHAIN> … preceding_token <END_CHAIN>
Loss = mean[-log p(yₜ | audio, y<ₜ)]
Average only over positions where labels ≠ -100

Code · Core implementation

Transformers 5.12.1 · shift_tokens_right
def shift_tokens_right(input_ids: torch.Tensor, pad_token_id: int, decoder_start_token_id: int):
    """
    Shift input ids one token to the right.
    """
    shifted_input_ids = input_ids.new_zeros(input_ids.shape)
    shifted_input_ids[:, 1:] = input_ids[:, :-1].clone()
    shifted_input_ids[:, 0] = decoder_start_token_id

    if pad_token_id is None:
        raise ValueError("self.model.config.pad_token_id has to be defined.")
    # replace possible -100 values in labels by `pad_token_id`
    shifted_input_ids.masked_fill_(shifted_input_ids == -100, pad_token_id)

    return shifted_input_ids
Transformers 5.12.1 · WhisperForConditionalGeneration.forward
        if labels is not None:
            if labels.shape[1] > self.max_target_positions:
                raise ValueError(
                    f"Labels' sequence length {labels.shape[1]} cannot exceed the maximum allowed length of {self.max_target_positions} tokens."
                )
            if decoder_input_ids is None and decoder_inputs_embeds is None:
                decoder_input_ids = shift_tokens_right(
                    labels, self.config.pad_token_id, self.config.decoder_start_token_id
                )

        outputs: Seq2SeqModelOutput = self.model(
            input_features,
            attention_mask=attention_mask,
            decoder_input_ids=decoder_input_ids,
            encoder_outputs=encoder_outputs,
            decoder_attention_mask=decoder_attention_mask,
            past_key_values=past_key_values,
            decoder_inputs_embeds=decoder_inputs_embeds,
            decoder_position_ids=decoder_position_ids,
            use_cache=use_cache,
            **kwargs,
        )
        lm_logits = self.proj_out(outputs.last_hidden_state)

        loss = None
        if labels is not None:
            loss_fct = CrossEntropyLoss()
            # move labels to correct device to enable PP
            labels = labels.to(lm_logits.device)
            loss = loss_fct(lm_logits.view(-1, self.config.vocab_size), labels.reshape(-1))
STEP 12 / Training control (throughout training)

12Configure resuming and monitoring callbacks

Select the resume path before training starts; memory and pause callbacks run at the end of training steps.

Input
Memory status / PAUSE / resume arguments
Output
Warnings, pause_ckpt, or a resume path
Priority
Explicit checkpoint → pause_ckpt → latest regular checkpoint

Logic · How it works

This stage spans the training loop: register callbacks and select the resume source first, then enter Trainer.train in the next stage. Trainer triggers memory checks and pause saves during execution. The flowchart numbers indicate the configuration and reading order.

MemoryWarningCallback checks the first positive step, then samples RAM and VRAM at a default interval of 25 steps. It issues a warning when usage reaches the 90% threshold, with at least 200 steps between warnings of the same type by default. It does not automatically reduce the batch size or pause training.

After an empty PAUSE file is created in the experiment's checkpoint directory, the callback saves the model, training arguments, optimizer, scheduler, optional AMP scaler, Trainer state, and random-number state to pause_ckpt at the end of an optimization step. On success, it deletes PAUSE and requests that the training loop stop.

At the next launch, an explicitly specified checkpoint takes priority. If none is specified and automatic resuming is enabled, try pause_ckpt first, followed by the regular checkpoint with the highest number. The configuration must have num_mel_bins set to 160. Training state is restored, as supported by the dependency, only through Trainer.train(resume_from_checkpoint=...).

Code · Core implementation

MemoryWarningCallback.on_step_end
    def on_step_end(self, args, state, control, **kwargs):
        step = int(getattr(state, "global_step", 0))
        if not self._should_check(step):
            return control

        ram = get_ram_usage()
        if ram is not None and float(ram["ratio"]) >= self.ram_threshold:
            if (step - self._last_ram_warn_step) >= self.repeat_warning_steps:
                self._last_ram_warn_step = step
                print(
                    "[MemWarn][RAM] "
                    f"step={step}, usage={float(ram['ratio']) * 100:.1f}%, "
                    f"used={_format_bytes_gb(int(ram['used']))}, "
                    f"total={_format_bytes_gb(int(ram['total']))}"
                )

        for item in get_vram_usage():
            device = int(item["device"])
            ratio = float(item["ratio"])
            if ratio < self.vram_threshold:
                continue
            last_step = self._last_vram_warn_step.get(device, -10**9)
            if (step - last_step) < self.repeat_warning_steps:
                continue
            self._last_vram_warn_step[device] = step
            print(
                "[MemWarn][VRAM] "
                f"step={step}, cuda:{device}, usage={ratio * 100:.1f}%, "
                f"used={_format_bytes_gb(int(item['used']))}, "
                f"total={_format_bytes_gb(int(item['total']))}"
            )
        return control
Main/ai_guitar_pedal/memory_guard.py · L116–L147 ↗
PauseSaveAndStopCallback.on_step_end
    def on_step_end(self, args, state, control, **kwargs):
        if self._pause_handled or not self.pause_flag_file.exists():
            return control

        print("[PAUSE] detected -> saving checkpoint then stopping.")
        try:
            if self._trainer is None:
                raise RuntimeError("Pause callback has no trainer reference.")
            if self._trainer.args.should_save:
                self._save_pause_checkpoint()
                print(f"[PAUSE] checkpoint saved: {self.pause_ckpt_dir}")
                if self.remove_pause_flag and self.pause_flag_file.exists():
                    self.pause_flag_file.unlink()
                    print(f"[PAUSE] flag removed: {self.pause_flag_file}")
            else:
                print("[PAUSE] non-saving process received pause signal, stopping.")
        except Exception as exc:
            print(f"[PAUSE] failed to save pause checkpoint: {exc}")

        self._pause_handled = True
        control.should_training_stop = True
        return control
Main/ai_guitar_pedal/phase2_training_utils.py · L476–L497 ↗
resolve_resume_checkpoint
def resolve_resume_checkpoint(
    args: argparse.Namespace,
    pause_ckpt_dir: Path,
) -> Tuple[Path | None, str]:
    if str(args.resume_from_checkpoint).strip():
        resume_path = Path(str(args.resume_from_checkpoint)).expanduser().resolve()
        if not resume_path.exists():
            raise FileNotFoundError(f"resume_from_checkpoint not found: {resume_path}")
        if not is_stereo_checkpoint(resume_path):
            raise ValueError(
                f"Explicit resume checkpoint is not stereo-compatible: {resume_path} "
                f"(num_mel_bins={checkpoint_input_feature_bins(resume_path)}, "
                f"expected={STEREO_FEATURE_BINS})"
            )
        return resume_path, "explicit"

    if not bool(args.auto_resume):
        return None, "disabled"

    if is_trainer_checkpoint(pause_ckpt_dir) and is_stereo_checkpoint(pause_ckpt_dir):
        return pause_ckpt_dir, "pause_ckpt"
    if is_trainer_checkpoint(pause_ckpt_dir):
        print(
            f"[Resume] skipping mono/incompatible pause checkpoint: {pause_ckpt_dir} "
            f"(num_mel_bins={checkpoint_input_feature_bins(pause_ckpt_dir)})"
        )

    latest_ckpt = find_latest_trainer_checkpoint(Path(str(args.output_dir)))
    if latest_ckpt is not None and is_stereo_checkpoint(latest_ckpt):
        return latest_ckpt, "latest_checkpoint"
    if latest_ckpt is not None:
        print(
            f"[Resume] skipping mono/incompatible latest checkpoint: {latest_ckpt} "
            f"(num_mel_bins={checkpoint_input_feature_bins(latest_ckpt)})"
        )
    return None, "none"
Main/ai_guitar_pedal/phase2_training_utils.py · L115–L150 ↗
STEP 13 / Training

13Let Trainer run backpropagation and weight updates

By default, each forward pass processes 1 sample; 32 gradient accumulation passes make one optimization step.

Input
Model, Dataset, Collator, TrainingArguments
Output
Updated weights and regular checkpoints
Defaults
3 epochs; learning_rate=3e-5; batch=1 × 32

Logic · How it works

Trainer handles batch iteration, forward loss computation, backpropagation, and optimizer updates. On one GPU with a complete accumulation cycle, batch size 1 × gradient accumulation 32 is equivalent to using gradients from 32 samples per update. The step counter counts optimization steps, not individual samples.

Training runs for 3 epochs by default; setting a positive max_steps makes the step limit control training instead. fp16 is enabled only when CUDA is available. Gradient checkpointing reduces the amount of stored activations while adding some computation. Settings such as the optimizer and scheduler, which V4 does not explicitly override, use the defaults of the installed TrainingArguments version.

By default, saving and evaluation occur every 250 steps, with at most 5 regular checkpoints retained. Logging occurs every 10 steps. report_to=[] disables integration reporting, so the existence of a runs or TensorBoard directory does not mean TensorBoard events have been written.

Code · Core implementation

Trainer hyperparameters
    kwargs: Dict[str, Any] = {
        "output_dir": str(Path(args.output_dir).expanduser().resolve()),
        "logging_dir": str(Path(args.log_dir).expanduser().resolve()),
        "per_device_train_batch_size": int(args.per_device_train_batch_size),
        "per_device_eval_batch_size": int(args.per_device_eval_batch_size),
        "gradient_accumulation_steps": int(args.gradient_accumulation_steps),
        "learning_rate": float(args.learning_rate),
        "num_train_epochs": float(args.num_train_epochs),
        "max_steps": int(args.max_steps),
        "fp16": bool(args.fp16 and torch.cuda.is_available()),
        "save_strategy": strategy,
        "logging_strategy": "steps",
        "logging_steps": 10,
        "save_total_limit": max(1, int(args.save_total_limit)),
        "report_to": [],
        "run_name": sanitize_experiment_name(args.experiment_name),
        "remove_unused_columns": False,
        "seed": int(args.seed),
        "gradient_checkpointing": bool(args.gradient_checkpointing),
        "dataloader_num_workers": int(args.num_workers),
        "dataloader_pin_memory": True,
        "eval_accumulation_steps": max(1, int(args.eval_accumulation_steps)),
    }
    if int(args.max_steps) <= 0:
        kwargs.pop("max_steps")

    params = inspect.signature(TrainingArguments.__init__).parameters
Main/train_program_seq2seq.py · L557–L583 ↗
Connect the model, data, and trainer
    model = build_model(args, tokenizer)
    trainer = Trainer(
        model=model,
        args=build_training_args(args, has_eval=eval_dataset is not None),
        train_dataset=train_dataset,
        eval_dataset=eval_dataset,
        data_collator=Seq2SeqCollator(pad_token_id=tokenizer.pad_token_id),
        compute_metrics=compute_token_metrics if eval_dataset is not None else None,
        preprocess_logits_for_metrics=preprocess_logits_for_metrics
        if eval_dataset is not None
        else None,
    )
Main/train_program_seq2seq.py · L749–L760 ↗
Run training or resume from a checkpoint
    resume_ckpt, resume_source = resolve_resume_checkpoint(args, pause_ckpt_dir)
    if resume_ckpt is not None:
        print(f"[Resume] source={resume_source}, resume_from_checkpoint={resume_ckpt}")
        trainer.train(resume_from_checkpoint=str(resume_ckpt))
    else:
        print(f"[Resume] source={resume_source}, starting from scratch")
        trainer.train()
Main/train_program_seq2seq.py · L779–L785 ↗
STEP 14 / Validation

14Evaluate predicted tokens and select the best weights

token_acc and seq_exact measure prediction quality when the true preceding tokens are provided.

Input
Validation forward-pass logits and labels
Output
eval_token_acc / eval_seq_exact / eval_loss
Best-model criterion
Higher eval_token_acc is better

Logic · How it works

preprocess_logits_for_metrics takes argmax over the vocabulary dimension at each position and retains only the predicted IDs, reducing the memory cost of caching all logits during evaluation. compute_token_metrics aligns predictions and labels and ignores positions marked -100.

token_acc measures the proportion of all valid tokens that are predicted correctly. seq_exact requires every valid position in a sample to be correct, then measures the proportion of samples meeting that condition. EOS and structural tokens both count toward these metrics.

When a validation set exists, set load_best_model_at_end=True and select the checkpoint with the highest eval_token_acc; evaluate again after training returns. This selection is not performed when there is no validation set.

token_acc = correctly predicted valid tokens / total valid tokens
seq_exact = samples with all valid positions correct / total samples

Code · Core implementation

preprocess_logits_for_metrics
def preprocess_logits_for_metrics(logits, labels):
    if isinstance(logits, tuple):
        logits = logits[0]
    return torch.argmax(logits, dim=-1)
Main/train_program_seq2seq.py · L483–L486 ↗
compute_token_metrics
def compute_token_metrics(eval_pred) -> Dict[str, float]:
    preds = np.asarray(eval_pred.predictions)
    labels = np.asarray(eval_pred.label_ids)
    if preds.shape != labels.shape:
        preds = preds.reshape(labels.shape)
    mask = labels != -100
    if not np.any(mask):
        return {"token_acc": 0.0, "seq_exact": 0.0}
    token_acc = float(np.mean(preds[mask] == labels[mask]))
    seq_ok = []
    for pred_row, label_row, mask_row in zip(preds, labels, mask):
        seq_ok.append(bool(np.all(pred_row[mask_row] == label_row[mask_row])))
    return {
        "token_acc": token_acc,
        "seq_exact": float(np.mean(seq_ok)) if seq_ok else 0.0,
    }
Main/train_program_seq2seq.py · L489–L504 ↗
Enable best-model selection when a validation set exists
        kwargs["load_best_model_at_end"] = True
        kwargs["metric_for_best_model"] = "eval_token_acc"
        kwargs["greater_is_better"] = True
Main/train_program_seq2seq.py · L593–L595 ↗
STEP 15 / Artifacts

15Save model assets for later loading

The model, feature configuration, DSP vocabulary, and run summary together describe the training result.

Input
Model after training returns + extractor + vocab
Output
Main/storage/artifacts/exports/<experiment>/
Run record
run_summary.json

Logic · How it works

Call trainer.save_model to write the model configuration and weights, save the feature extractor configuration, and copy the current DSP vocabulary. The model uses 160-bin input, while the feature extractor still operates with a single-channel 80-bin configuration. The same left/right channel processing must therefore be applied when loading and using the model.

run_summary.json records the target definition, data locations, sample counts, program-hash counts, target lengths, metrics, hyperparameters, resume source, feature-cache path, and effective environment variables. It is the starting point for reviewing the training setup and results.

Regular checkpoints and pause_ckpt are used to restore training state; exports are used for subsequent model loading. V4 currently ends at export. It does not include the downstream execution code to turn predicted tokens into runnable DSP, render audio, or integrate a real-time pedal.

Code · Core implementation

Run final evaluation and save the model and protocol
    if eval_dataset is not None:
        metrics = trainer.evaluate()
        print(
            f"[Eval] eval_token_acc={metrics.get('eval_token_acc', 0.0):.6f}, "
            f"eval_seq_exact={metrics.get('eval_seq_exact', 0.0):.6f}"
        )
    else:
        metrics = {}

    export_dir.mkdir(parents=True, exist_ok=True)
    trainer.save_model(str(export_dir))
    feature_extractor.save_pretrained(str(export_dir))
    shutil.copy2(vocab_path, export_dir / "dsp_control_vocab.json")
Main/train_program_seq2seq.py · L787–L799 ↗
Define the complete run summary
    summary = {
        "objective": "audio_to_dsp_control_program_tokens",
        "desensitized_from_preset_number": True,
        "uses_preset_number_as_label": False,
        "target_source_field": "target_program_json.model_token_ids",
        "manifest": str(manifest_path),
        "vocab": str(vocab_path),
        "program_root": str(program_root),
        "vocab_size": int(tokenizer.vocab_size),
        "audio_root": str(audio_root),
        "total_rows": int(len(rows)),
        "train_rows": int(len(train_rows)),
        "eval_rows": int(len(eval_rows)),
        "unique_program_hashes": int(program_hash_count),
        "train_program_hashes": int(len({r["program_hash"] for r in train_rows})),
        "eval_program_hashes": int(len({r["program_hash"] for r in eval_rows})),
        "target_length": {
            "min": int(min(lengths)),
            "max": int(max(lengths)),
            "mean": float(np.mean(lengths)),
        },
        "metrics": metrics,
        "training_config": {
            "model_name": str(args.model_name),
            "audio_channel_mode": AUDIO_CHANNEL_MODE,
            "input_feature_bins": int(STEREO_FEATURE_BINS),
            "max_seconds": float(args.max_seconds),
            "max_target_length": int(args.max_target_length),
            "strip_bos_from_labels": bool(args.strip_bos_from_labels),
            "eval_ratio": float(args.eval_ratio),
            "num_train_epochs": float(args.num_train_epochs),
            "max_steps": int(args.max_steps),
            "checkpoint_strategy": str(args.checkpoint_strategy),
            "save_steps": int(args.save_steps),
            "eval_steps": int(args.eval_steps),
            "save_total_limit": int(args.save_total_limit),
            "per_device_train_batch_size": int(args.per_device_train_batch_size),
            "per_device_eval_batch_size": int(args.per_device_eval_batch_size),
            "gradient_accumulation_steps": int(args.gradient_accumulation_steps),
            "learning_rate": float(args.learning_rate),
            "gradient_checkpointing": bool(args.gradient_checkpointing),
            "fp16": bool(args.fp16 and torch.cuda.is_available()),
            "gpu_mem_frac": float(args.gpu_mem_frac),
            "auto_resume": bool(args.auto_resume),
            "resume_from_checkpoint": str(resume_ckpt) if resume_ckpt is not None else "",
            "resume_source": str(resume_source),
        },
        "feature_cache_base": str(cache_base),
        "e_root": str(e_root),
        "output_dir": str(output_dir),
        "log_dir": str(log_dir),
        "export_dir": str(export_dir),
        "pause_flag_file": str(pause_flag_file),
        "pause_ckpt_dir": str(pause_ckpt_dir),
        "env": env_effective,
    }
Main/train_program_seq2seq.py · L801–L856 ↗
Write the run summary to disk
    (export_dir / "run_summary.json").write_text(
        json.dumps(summary, indent=2, ensure_ascii=False),
        encoding="utf-8",
    )
    print(f"Parameter sequence model exported to: {export_dir}")
Main/train_program_seq2seq.py · L857–L861 ↗
PRACTICAL REFERENCE

Commands and reading reference

Run all commands below from the AI_Guitar_Pedal_V4 project root. Check paths first, validate the data, then start training.

1. Environment and checks

python -m venv Main/.venv
Main/.venv/Scripts/python.exe -m pip install -r Main/requirements.txt
powershell -ExecutionPolicy Bypass -File .\scripts\train.ps1 -DryRun
powershell -ExecutionPolicy Bypass -File .\scripts\train.ps1 -ValidateOnly

If a V4 environment has not been created, the launcher tries the data project's Main/.venv, then falls back to system Python. Versions in requirements.txt are currently unpinned; check the installed PyTorch/CUDA and Transformers versions when moving to a new environment.

2. New experiments and data selection

powershell -ExecutionPolicy Bypass -File .\scripts\train.ps1 `
  -ExperimentName Phase2_V4_Experiment_0002 `
  -DataProjectRoot ..\AI_Guitar_Pedal_V2

# Start a new experiment from exported weights (without restoring the old optimizer)
# Resolve the absolute path first; the launcher changes to Main before starting Python
$exportPath = (Resolve-Path .\Main\storage\artifacts\exports\Phase2_V4_Experiment_0002).Path
powershell -ExecutionPolicy Bypass -File .\scripts\train.ps1 `
  -ExperimentName Phase2_V4_FromExport `
  -ModelName $exportPath

The ModelName above is an example path; its export must already exist. Reusing an experiment directory attempts automatic resume by default. Use a new ExperimentName to start a separate experiment. The launcher can override MaxSteps, MaxSeconds, MaxTargetLength, and other settings.

3. Pause and resume

# Use only while this experiment is training; the name must match the active experiment
New-Item -ItemType File -Force `
  .\Main\storage\artifacts\checkpoints\Phase2_V4_Experiment_0002\PAUSE

# Later, rerun the original command; automatic resume is enabled by default
powershell -ExecutionPolicy Bypass -File .\scripts\train.ps1 `
  -ExperimentName Phase2_V4_Experiment_0002 `
  -ResumeFromPauseCkpt

Default settings at a glance

SettingDefaultMeaning
ModelNameopenai/whisper-baseInitializes the model and feature extractor
MaxSeconds / MaxTargetLength6 / 768Uses the first 6 seconds; rejects labels longer than 768 tokens
EvalRatio / seed0.02 / 42Splits by program group
TrainBatch / GradientAccumulation1 / 32Effective batch size of 32 on one GPU with full accumulation
LearningRate / NumTrainEpochs3e-5 / 3Learning rate and default epoch count
SaveSteps / EvalSteps250 / 250Save and evaluation intervals in optimizer steps
SaveTotalLimit5Maximum number of regular checkpoints retained
GradientCheckpointing / fp16true / true when CUDA is availableMemory and computation precision settings
Cache / pre-extraction / automatic resumetrue / true / trueCan be disabled through the launcher

Local reading and maintenance

Open docs/site/dist/index.html directly to read offline. After updating the training source or explanations, run python docs/site/build.py to refresh code excerpts, line numbers, and source snapshots, then run python docs/site/verify.py to check internal links and snapshot consistency.

The guide consists of static files in docs/site/dist/, suitable for offline reading or hosting in a portfolio subdirectory. All resources use relative links, with no backend, external fonts, or CDN.

Implementation scope

  • Implemented in V4: data validation and splitting, stereo features, caching, Whisper program-token training, teacher-forced metrics, pause/resume, and export.
  • Provided externally: audio rendering, DSP program serialization, and training data; V2 is the default source.
  • Not yet implemented: free-generation evaluation, complete program grammar constraints, DSP dequantization and execution, rendered-audio comparisons, and a real-time inference UI.

Organization validation · 2026-09-16

A full ValidateOnly run through the new entry point completed with exit code 0; GPU training was not started. Data loading and label validation took 27.7 seconds, with the following results:

ItemObserved result
Total samples / unique program hashes60,600 / 1,510
Training samples / training programs59,400 / 1,480
Validation samples / validation programs1,200 / 30
Target token length (after removing BOS)Minimum 14; maximum 685; mean 169.11
New and legacy entry point consistency41 matching parameters; 52 matching lines of effective path and parameter output; both DryRun exit codes were 0