MidiPilot - Your AI Copilot

MidiPilot is the AI brain embedded directly in MidiEditor AI. Open the sidebar panel, type what you want in plain English, and watch it compose, edit, transform, and analyze your MIDI data automatically.

Key Features

Chat Panel

The MidiPilot panel features a clean chat interface with context-aware track info, mode selection (Agent / Simple), and model switching - all accessible without leaving the editor. Assistant bubbles and the live reasoning stream render Markdown (bold, italic, lists, fenced code, links); user and system bubbles stay plain text.

The input bar at the bottom provides quick access to everything you need:

Agent/Simple mode toggle · FFXIV checkbox · Model selector · Reasoning level

• Mode selector - Switch between Agent (multi-step) and Simple (single response)
• New Chat - Start a fresh conversation (previous one is auto-saved to history)
• History - Browse and resume past conversations
• FFXIV checkbox - Enable FFXIV Bard Performance constraints
• Model dropdown - Switch models on the fly without opening settings
• Reasoning level - Adjust thinking effort (when supported by the model)
• Token counter - Shows session usage vs. context window size
• Send / Stop - Send your prompt or cancel an in-progress Agent run
• Settings gear - Open MidiPilot Settings… or save per-file AI presets
• Status indicator - Shows Ready, Thinking, or error states in real time

Simple Mode (One-Shot)

Simple mode sends a single API call to the AI model containing your instruction, the current editor state, selected events, and surrounding musical context. The model responds with one complete answer - no follow-up calls, no iterative loop.

How It Works

1. Your prompt is bundled with a JSON snapshot of the editor (cursor position, active track, tempo, time signature, selection, and ±5 measures of surrounding events)
2. Everything is sent as one request to the model
3. The model returns a single JSON response with one or more actions (edit, delete, create_track, set_tempo, etc.)
4. MidiEditor executes all actions from that response at once

When to Use Simple Mode

• Quick edits - transpose a selection, quantize notes, change velocities
• Small segments - work on a specific track or a few measures
• Focused tasks - delete notes, change instruments, adjust tempo
• Fast turnaround - one call means instant results with no waiting for multi-step loops

Limitations

• Token limit risk - Complex requests (many tracks, long passages) can exceed the model’s output token limit. When this happens the response is truncated, which can result in incomplete or broken edits
• No tool access - The model cannot call tools to inspect the file or request additional context. It works only with whatever was included in the initial snapshot
• No self-correction - If something goes wrong, it cannot verify or retry. You have to undo and try again manually

If MidiPilot detects a truncated response (finish_reason: "length"), it will display a warning suggesting you switch to Agent mode for the task.

Agent Mode (Multi-Step)

Agent mode is the powerhouse for complex compositions and large-scale edits. Instead of squeezing everything into one response, the AI works iteratively - planning, executing, inspecting, and adjusting across multiple API calls until the task is complete.

How It Works

1. Your prompt is sent along with the system prompt and a set of 16 tools the model can call
2. The model responds with one or more tool calls (e.g., create_track, insert_events, get_editor_state)
3. MidiEditor executes each tool call and sends the results back to the model
4. The model reviews the results, decides the next step, and issues more tool calls
5. This loop continues until the model decides the task is complete (or the max steps limit is reached)

Why This Matters

• No token limit risk - Work is split across many smaller API calls, so each individual response stays well within token limits. No truncation, no data loss
• Self-monitoring - The model can call get_editor_state or query_events at any time to inspect what it has built so far, catch mistakes, and correct them
• Complex compositions - Multi-track arrangements, genre conversions, full orchestrations - tasks that would overwhelm a single response are handled step by step
• Parallel tool calls - The model can issue multiple tool calls in a single step (e.g., create three tracks at once), speeding up large jobs
• Granular undo - Each tool call from an Agent run gets its own undo step. Press Ctrl+Z to revert one action at a time, keeping the rest intact

Agent Steps Panel

During an Agent run, a collapsible Agent Steps panel appears below the chat showing real-time progress: each tool call, its parameters, and results. The agent loop can also stream live reasoning summaries, assistant text, and tool-call argument progress while the step is still being generated. Step indicators use theme-aware colors that adapt to dark and light mode (⏳ pending, 🔄 active, ✅ done, ⚠ retrying, ❌ failed).

Gemini 3.1 native streaming with live thought summaries and Agent tool progress

When to Use Agent Mode

• Composing from scratch - “Create a jazz waltz with piano, bass, and drums”
• Multi-track work - Adding harmonies, arranging for multiple instruments
• Genre conversion - “Make this classical piece into a metal version”
• FFXIV setups - Preparing an 8-track octet with channel mapping and drum conversion
• Analysis tasks - “Check all tracks for issues and fix them”

Configuration

Agent Conductor & Working State

The agent loop is wrapped by a lightweight conductor that owns a compact, program-managed working state: the user goal, the inferred task type, a list of confirmed facts (what the model has already accomplished), the last tool result, the active constraints (e.g. FFXIV rules), and a counter of consecutive failed write attempts.

• After every successful tool call, the conductor distils a one-line confirmation into the working state (e.g. “Tempo set to 82 BPM”, “8 tracks created”) so the model resumes from real progress instead of re-deriving it from raw tool output.
• After a rejected write (validation error, FFXIV violation, capability error…), the conductor injects a short, positive rejection guidance - describing what should be tried next instead of restating what failed.
• If two consecutive write attempts fail without making progress, the conductor performs a bounded-failure stop and reports the remaining problem in plain language instead of looping forever.

GPT-5.5 model-isolation policy

For OpenAI gpt-5.5*, MidiPilot applies a small set of model-specific mitigations through a central policy table - nothing else is touched. Currently:

• Schema-light tools - the insert_events / replace_events JSON schema drops the pitch_bend branch that this model frequently mis-emits.
• Sanitised, positive rejection guidance - rejection messages avoid phrasing the model is known to over-react to.
• Serial tool calls + low reasoning effort on the OpenAI Responses API (parallel_tool_calls: false, reasoning.effort: "low").

On OpenRouter, only the schema-light tools and prompt sanitisation apply (no Responses-API knobs). On every other model, including all current OpenAI gpt-5*/gpt-4o/o-series, Pitch Bend and parallel tool calls remain unchanged.

FFXIV Bard Mode

When the FFXIV checkbox is enabled, MidiPilot appends additional constraints to the system prompt that enforce Final Fantasy XIV Bard Performance rules. This works with both Simple and Agent mode.

• Agent + FFXIV - Full FFXIV context (~3500 chars) with detailed guidance + 3 FFXIV-specific tools (validate_ffxiv, convert_drums_ffxiv, setup_channel_pattern)
• Simple + FFXIV - Compact FFXIV context (~1300 chars) optimized for token budget. Includes a warning that large compositions may exceed limits

Enforced Rules Overview

• ✅ Note range C3-C6 (MIDI 48-84) - strict boundaries, no notes outside
• ✅ Monophonic tracks - one note at a time (Lute/Harp/Piano allow limited chords)
• ✅ Track name = instrument - the track name determines which instrument the bard equips in-game (29 valid names)
• ✅ Instrument → GM program mapping - correct program_change at tick 0 per channel
• ✅ Channels are cosmetic - the channel number does not affect the instrument. Unique channels per track are recommended for clean editing, but not required
• ✅ Drums can stay on channel 9 - the track name (Bass Drum, Snare Drum, Cymbal, etc.) determines the percussion instrument, not the channel
• ✅ GM drum conversion - kick → Bass Drum C4, snare → Snare C5, hi-hat → Cymbal C6, crash → Cymbal C5, toms → Timpani
• ✅ Guitar switches - the only case where channels matter functionally. Each of the 5 variants (Clean, Muted, Overdriven, PowerChords, Special) needs its own channel with program_change at tick 0. Switch sound mid-track by changing the note’s channel
• ✅ No pitch bend / no velocity editing - velocity fixed at 100
• ✅ Instrument transposition - native ranges mapped per instrument (e.g., Piano C4-C7 = −1 octave, Tuba C1-C4 = +2 octaves)
• ✅ Track ordering - melodic instruments first, drums at end (highest track indices)

See FFXIV Prompt Examples for tested prompts.

Fix X|V Channels

The Fix X|V Channels tool provides a one-click, deterministic channel fixer that sets up the complete MidiBard2 channel mapping - no AI calls needed. Find it in the toolbar or via Tools → Fix X|V Channels.

👉 Full documentation: Fix X|V Channels - the 5-step algorithm, Rebuild vs Preserve modes, supported instruments, guitar variant switching, before/after screenshots, and tips.

Mode Comparison

Token Tracking & Context Window

MidiPilot tracks token usage per API call and per session, with automatic normalization across providers (OpenAI, Anthropic, Gemini). The token counter is displayed at the bottom of the chat panel:

<last call> | <session total>🔥 / <context window> [<limit>✂]

• Last call - Prompt + completion tokens from the most recent API call
• Session total - Cumulative tokens since the chat was opened (🔥 fire icon)
• Context window - The model’s known context limit (e.g., 128k for GPT-4o, 1M for GPT-5). Turns yellow when usage exceeds 80%
• Limit - Only shown when a max token limit is enabled (✂ scissors icon)

Context Window Management

When conversations grow long, MidiPilot automatically manages context to prevent exceeding the model’s limit:

• Sliding window - When estimated token usage exceeds 70% of the context window, older messages are dropped while keeping the system prompt, first 2 messages (task context), and most recent messages
• Truncation marker - A “[Context truncated]” message is inserted where older messages were removed
• Visual warning - The token label turns yellow when approaching 80% of the context window

Multi-Provider Token Normalization

Different providers report token usage in different formats. MidiPilot normalizes all of them:

• OpenAI Chat Completions - prompt_tokens / completion_tokens (native format)
• OpenAI Responses API - input_tokens / output_tokens → normalized
• Anthropic - input_tokens / output_tokens → normalized
• Google Gemini - usageMetadata.promptTokenCount / candidatesTokenCount → normalized
• Estimation fallback - When a provider returns no usage data, MidiPilot estimates ~4 characters per token

AI Settings

Configure your AI connection from Settings → MidiPilot AI. Select a provider, enter your API key, choose a model, and customize behavior.

Model Refresh & Favorites

MidiEditor AI can fetch provider model lists directly instead of relying on a fixed dropdown. The refresh button is available both in Settings → MidiPilot AI and in the MidiPilot footer, so you can update models without leaving the chat panel. Refreshed models are cached for seven days and used for context-window estimates.

When the footer refresh completes, MidiPilot reports the result in the chat/status area so you know whether the cache was updated or the provider rejected the request.

Model update feedback after a successful refresh

If a model fails live streaming but still works without streaming, MidiPilot retries automatically and marks that model with a warning icon for the current app session. The mark is scoped per mode - Simple Mode (no tools) and Agent Mode (with tools) are tracked independently, because some models support live streaming in only one of the two paths. The dropdown shows which mode is blocked: ⚠ <model> (Simple), (Agent), or (Simple+Agent). Use Force Streaming for This Model to clear the temporary mark after switching providers, updating the model, or testing a fixed endpoint.

Capability-aware error handling - if the active provider returns HTTP 404 with “No endpoints found that support tool use” (or any equivalent “tools not supported” error), MidiPilot stops retrying immediately, posts a clear “Model does not support tool calling - pick a different model in Settings → AI, or switch to Simple mode for this request” bubble, and remembers the flag per provider:model for the rest of the session. Picking a different model re-enables Agent Mode automatically.

Custom System Prompts

Click Edit System Prompts… in settings to open the built-in editor. Each mode (Simple, Agent, FFXIV, FFXIV Compact) has its own tab with fully customizable instructions.

Built-in System Prompt Editor with tab-based mode selection

• Load Default - Reset the current tab to the built-in default prompt
• Export to JSON - Save all prompts to a file for backup or sharing
• Import from JSON - Load prompts from a previously exported file
• Reset All to Defaults - Restore all modes to factory defaults

Prompts are saved as system_prompts.json in the application directory. If no custom file exists, MidiPilot uses the hardcoded defaults.

Looking for per-model overrides? The newer Prompt Profiles dialog lets you attach a dedicated system prompt to one or more provider:model combinations - great for guiding a single “quirky” model without touching the global mode prompts.

Prompt Profiles (Per-Model System Prompts)

Open Settings → AI → Prompt Profiles… to manage profiles. A profile binds a custom system prompt to one or more model patterns and decides whether the profile replaces or appends to the default mode prompt.

MidiPilot ships with one built-in profile: GPT-5.5 Decisive, bound to openai:gpt-5.5* and openrouter:openai/gpt-5.5*. It nudges the model to commit to a tool call instead of asking clarifying questions, which is a known weak spot of that family. You can edit, duplicate, or disable it like any other profile.

Prompt Profiles dialog with the built-in GPT-5.5 Decisive profile bound to openai:gpt-5.5* and openrouter:openai/gpt-5.5*

Auto-Save

MidiEditor AI automatically saves a backup copy of your work at regular intervals, so you never lose progress to a crash or accidental close. Your original file is never overwritten - the backup is stored as a separate .autosave sidecar file alongside your MIDI file.

How It Works

• Debounce timer - after each edit, a countdown resets. The backup is only written once you pause editing for the configured interval (default: 120 seconds).
• Named files - saved as YourSong.mid.autosave next to the original file.
• Untitled documents - saved to AppData/MidiEditor AI/autosave/untitled.autosave.mid.
• Cleanup - the .autosave backup is automatically deleted when you save normally or exit cleanly.
• Protocol entry - each auto-save appears as an “Auto-saved” marker in the undo history panel.

Crash Recovery

• When opening a file, if a newer .autosave backup exists, MidiEditor AI offers to recover it.
• On startup, orphaned untitled backups from previous crashes are detected and a recovery dialog is shown.
• Recovered files open as unsaved - use Save As to give them a permanent name.

Settings

Auto-save options are in Settings → System & Performance:

Auto-Save settings with enable toggle and interval configuration

AI Tools Reference

In Agent mode, the AI has access to 16 tools (12 base + 4 FFXIV-specific) for inspecting and modifying MIDI files:

Supported Providers

Getting Started

1. Open Settings (gear icon or Edit → Settings) and click the MidiPilot AI tab
2. Select your Provider (Google Gemini is a great free option)
3. Enter your API Key (get one from OpenAI, OpenRouter, or Google Gemini)
4. Choose a Model (e.g., gemini-2.5-flash)
5. Click Test Connection to verify everything works
6. Close settings and open the MidiPilot panel from the sidebar
7. Type a prompt and press Enter:

"Create an 8-bar jazz waltz in Bb major with piano, bass, and drums"

The AI will compose the requested music directly into the editor using its built-in tools. In Agent mode, it works iteratively - creating tracks, setting tempo, inserting notes, and validating the result step by step.

See Prompt Examples for more real-world prompts and a full demo.

Conversation History

MidiPilot automatically saves every conversation as a JSON file. You can browse, search, and resume past sessions at any time.

How It Works

• Auto-save - After every assistant response, the conversation is saved (debounced 2 seconds) to AppData/MidiPilotHistory/
• Associated with MIDI file - Each conversation records which MIDI file it was working on
• History button - Click the 📜 history icon in the toolbar (next to New Chat) to browse past conversations
• Turn metadata - Agent reasoning, executed steps, latency, provider, model, and token counts are stored with each assistant turn
• Resume - Click any past conversation to load it back into the chat and continue where you left off
• Persistent across sessions - Conversations survive app restarts

Conversation File Format

Each conversation is stored as a single JSON file containing the full message history, model/provider info, token usage, per-turn metadata, and the associated MIDI file path. Files are human-readable and can be exported or shared.

Response Streaming

MidiPilot uses Server-Sent Events (SSE) to stream responses in real time. Instead of waiting for the entire response to complete, Simple mode can show text or action composition immediately, and Agent mode can show live reasoning, assistant text, and tool-call progress while each step is still being generated.

How It Works

• Simple text - A chat bubble appears immediately and grows as text arrives
• Simple actions - JSON action responses show a subdued composing indicator, then execute once the complete payload is available
• Agent thoughts - Reasoning summaries stream into the grey italic thought block when the provider exposes them
• Agent tools - Tool calls are reconstructed from streamed deltas, then dispatched through the same undo-safe tool path as non-streaming Agent mode
• Fallback - Broken streams automatically retry without streaming and mark the model temporarily for the current app session
• Token capture - Usage statistics are captured from the final SSE event when the provider reports them

When Streaming Is Used

Mode-scoped streaming fallback - failed paths are marked (Simple), (Agent), or (Simple+Agent) in the model dropdown until you re-enable streaming for that model

Non-Streaming Reference Run

For comparison, here is the same Agent loop running without live streaming - either because the provider does not support it, the user disabled Live Streaming in settings, or a previous request hit the per-session streaming-fallback marker. Tool calls and assistant text still arrive correctly, just in one chunk per round-trip instead of progressively.

Non-streaming Agent run - same multi-step composition flow, batched tool results per round-trip

Per-File AI Presets

Different MIDI files may need different AI settings. A 16-track orchestral arrangement needs different guidance than a 3-track FFXIV bard song. Per-file presets let you save and auto-load settings for each file.

What’s Saved

• Provider - OpenAI, OpenRouter, Gemini, or Custom
• Model - e.g., gpt-5.4, gemini-2.5-flash
• Mode - Simple or Agent
• FFXIV mode - On or off
• Reasoning effort - None / Low / Medium / High / Extra High
• Custom instructions - Free-text notes appended to the system prompt (e.g., “This is a jazz arrangement, keep swing feel”)

How to Use

1. Click the ⚙ gear button in the MidiPilot footer
2. Select “Save AI preset for this file”
3. The current settings are saved as a .midipilot.json sidecar file next to your MIDI file
4. Next time you open that MIDI file, the preset is auto-loaded

Sidecar File

Presets are stored as <filename>.midipilot.json next to the MIDI file. For example:

Sweet Child O Mine.mid
Sweet Child O Mine.mid.midipilot.json   ← preset

The preset file is a simple JSON object. All fields are optional - any field not present falls back to the global default.

MCP Server - External AI Clients

MidiEditor AI includes a built-in MCP (Model Context Protocol) server that exposes all 15 MidiPilot tools to external AI clients. Instead of using the built-in chat panel, you can connect Claude Desktop, VS Code Copilot, Cursor, Windsurf, or any other MCP-compatible client and let it edit your MIDI files directly.

Enable the MCP server in Settings → AI → MCP Server, copy the config JSON, paste it into your AI client, and you’re ready. All tool calls appear in the Protocol panel with the client name (e.g. “MidiPilotMCP (VS Code Copilot Claude Opus 4.6)”) and support full undo.

📖 Full MCP Server Documentation →

API Log

MidiPilot writes every API request and response to a log file for debugging and transparency. The log is saved as midipilot_api.log in the same directory as the MidiEditor AI executable.

If the AI produces unexpected results, open the log to inspect the raw JSON sent to and received from the provider. This is especially useful for debugging tool-call sequences in Agent mode.