SECourses Video Captioner Pro: 1-Click Video, Image & Audio Captions (Patreon)
Downloads
Content
SECourses Video Captioner Pro captions video, images and sound, transcribes speech with Whisper and builds training datasets using 21 checkpoint variants.
Supports Qwen3-Omni, TimeChat Captioner GRPO and AVoCaDO among five model families, with 19 workflow presets, 52 task presets, batch folders, multi-GPU jobs and Kohya / Musubi TOML export. 1-click installers for Windows, RunPod and Massed Compute; V2 brought many bug fixes.
Patreon exclusive posts index to find our scripts easily, Patreon exclusive posts index to see which scripts updated or added last and amazing Patreon special generative scripts list that you can use in any of your task.
Join discord to get help, chat, discuss and also tell me your discord username to get your special rank : SECourses Discord
Please also Star, Watch and Fork our Stable Diffusion & Generative AI GitHub repository and join our Reddit subreddit and follow me on LinkedIn (my real profile)
=======
Latest installer zip file : SECourses_Video_Captioner_Pro_v2_0.zip
Windows Requirements
Please use latest Windows Installer tutorial and zip file literally 1-click Bat file
1-time only necessary : https://youtu.be/ptuLL0C72Ms
Setup zip file : https://www.patreon.com/SECourses/posts/1-click-windows-172290854
28 October 2026 Update V2
Lots of bug fixes and improvements
What you can do with Video Captioner Pro:
Caption video, images, and sound. Generate scene and motion descriptions, image captions, or audio descriptions with a model suited to the media and task.
Choose the right AI for the job. Access five model families, 21 checkpoint variants, 19 workflow presets, and 52 task presets, with custom prompts and a reusable prompt library.
Turn speech into subtitles. Use Whisper for transcription, English translation, and optional word timestamps; export SRT, VTT, TXT, LRC, TSV, or JSON.
Ask follow-up questions about your media. Chat with Qwen3 Instruct or Thinking using video, audio, images, and text to explore details beyond the first caption.
Break long videos into useful clips. Split by detected scenes, fixed duration, or trainer length; preview sampled frames and filter clips by sharpness, darkness, motion, or silence.
Process folders and ZIPs in batches. Scan subfolders, filter filenames and media types, skip completed items, retry failures, and distribute compatible folder jobs across multiple GPUs.
Combine visuals, speech, and sound. Keep separate video and audio caption parts, reuse existing visual captions, and merge descriptions, transcripts, and sound captions with your own templates.
Review and improve captions in one editor. Browse media, filter the queue, find and replace text, apply bulk edits, regenerate captions with a before/after diff, and approve the items you want to keep.
Prepare datasets for training. Export approved media/text pairs and ZIPs, inspect caption statistics and clip fitness, sub-split videos, and generate Kohya / Musubi TOML configurations.
Choose a memory plan that fits your hardware. Use VRAM presets, native BF16/INT8/INT4 variants or supported Qwen GGUF variants, block swapping, and model unloading to manage GPU memory.
Carry a good workflow into the next job. Save prompts and presets, recover settings from previous runs, download result files or ZIPs, and use the supplied Windows or Linux/cloud setup scripts.
SECourses Video Captioner Pro — Complete Practical Guide
Turn raw media into captions, subtitles, and training datasets.
Describe what happens. Capture what is said. Build a dataset you can review and export. SECourses Video Captioner Pro connects AI captioning, Whisper transcription, media chat, clip preparation, and caption editing in one browser workspace, running on your own machine or a cloud GPU.
Built for dataset creators, LoRA trainers, video editors, and anyone turning a media collection into useful text. Go from one clip to a reviewed batch with the same connected workflow.
From your first caption to a reviewed training dataset. This guide covers version 1.7.0, its Windows and Linux distribution, all ten tabs, all five model families, 21 checkpoint variants, 19 shipped workflow presets and 52 task presets.
You will learn to describe video, images and sound; transcribe speech; ask follow-up questions about media; split long recordings; review and approve captions; and prepare paired media and text for training. The examples use actual application runs. Start with the first-caption walkthrough, then follow the workflow that matches your files. The complete control reference at the end is for the moments when you need a specific setting.
A caption describes what happens or what can be heard. A transcript records spoken words. A training caption is the text paired with a media file. These serve different purposes: a detailed sound description should not be substituted for an accurate transcript.
1. Install and open the application
Windows: keep the distribution together
Extract the complete release into a folder with enough free disk space. Keep the BAT files, Models_Downloader.py and video_caption_requirements.txt together above the SECourses_Video_Captioner_Pro application folder.
Install the prerequisites expected by the distribution: 64-bit Python 3.12, Git, FFmpeg and FFprobe, a compatible NVIDIA driver and the supplied CUDA/cuDNN stack. The Windows installer calls for CUDA 13 and cuDNN 9.17 or newer. Full TorchInductor compilation also needs the Visual Studio C++ build workload.
Run Windows_Install_and_Update.bat. Leave the terminal open and read its final success or error message.
Run Windows_Run_Video_Captioner_Pro.bat. Keep that terminal running while using the interface.
Open the local URL printed by the application. It normally starts at http://127.0.0.1:7860 and can choose another free port. The screenshots used a separate tutorial instance on port 7865.
To download models ahead of time, run Windows_Download_Models_and_or_Resume.bat, or use System & Models inside the app.
The update installer resets tracked application files before pulling updates. Back up local code changes before running it. Your ordinary launch command should be the Run BAT file; an installation/update is not required for every session.
The supplied stack is specific to this release. The tested Windows environment used Python 3.12.10, PyTorch 2.13.0+cu130 and Gradio 6.26.0. CUDA 13.x requires NVIDIA driver branch 580 or newer; choose a driver that also supports your particular GPU. See NVIDIA's compatibility table. A model's download size is not its total VRAM requirement.
RunPod / SimplePod
Upload the complete distribution to a persistent /workspace volume. The release includes the SECourses CUDA 13 template and a SimplePod template. Storage size must cover the selected checkpoints, environment, temporary clips and outputs; a collection of large variants can exceed a small default volume.
Run these commands from a terminal:
cd /workspace export HF_HOME="/workspace" chmod +x RunPod_Install_SECourses_Video_Captioner_Pro.sh ./RunPod_Install_SECourses_Video_Captioner_Pro.sh
For a later launch:
export HF_HOME="/workspace" cd /workspace/SECourses_Video_Captioner_Pro source venv/bin/activate unset LD_LIBRARY_PATH python secourses_app.py --share
The share link exposes the running interface to people who have it. Keep model storage and outputs on persistent storage, and end the rented instance when your work is finished. This guide makes no pricing or availability claims.
Massed Compute / Ubuntu
Open a terminal in the extracted distribution folder and run:
chmod +x Massed_Compute_Install.sh ./Massed_Compute_Install.sh
Then launch:
cd SECourses_Video_Captioner_Pro source venv/bin/activate unset LD_LIBRARY_PATH python secourses_app.py --share
The Linux installers provision prerequisites and Python, install the release requirements, and attempt to build the pinned CUDA llama.cpp runtime. GGUF on Linux needs the CUDA toolkit with nvcc, CMake and a C++ toolchain. A failed optional build can be retried with System & Models → Install / repair llama.cpp. Windows live workflows were verified for this guide; the cloud installers received source and syntax checks, not a rented-cloud installation.
Understand the workspace
The top bar applies one workflow preset across the relevant tabs. Selecting a preset applies it immediately. A new launch starts from the shipped default; Load Last Values explicitly restores the last-used preset. Use Open / Close All, or F4, to expand the sections of the current tab.
Use Caption for generated descriptions, Processing Pipeline for sampling and splitting, Transcribe for Whisper, and Chat for questions. Caption Editor handles review. Dataset & Export prepares trainer inputs. The remaining tabs manage global paths, recover recorded settings, inspect system/model health and display release history.
2. Your first useful caption
For a first run, use a short local video with a clear subject. The walkthrough uses a 20.020-second street scene with lightning and sound. If you have a 24–32 GB GPU and the checkpoint is available, Qwen3-Omni Instruct → GGUF Q4_K_M + mmproj Q8_0 is a practical path to try. On a smaller GPU, start with the app's matching VRAM tier and its suggested variant rather than copying this machine's selection.
In Caption, choose File path and paste the media path. A path refers to the machine running the app; on a cloud instance, upload the file or use its cloud path.
Alternatively, choose Upload files and select supported media. Quoted paths, mixed slash styles and Unicode names are supported.
Check the preview, duration, dimensions, frame rate and audio details. Use the player to inspect a meaningful moment.
Leave trimming alone for this first example. Trim range can later set Start and End in seconds; an empty end means through the end of the file.
Select the model and a compatible Task / prompt preset. For the example, use the dense audiovisual Qwen3 task. Keep sampling conservative and use a sufficient reply budget; the recorded first run allowed 512 new tokens per segment.
Click Start Captioning. A first use may download and verify a checkpoint before loading it.
The player's visual trim produces an edited input; numeric trimming is applied after that. Input trimming re-encodes for accurate boundaries using the configured quality. The separate Fast stream copy setting controls later clip splitting, not the accuracy of the input trim.
When generation finishes, read the text before exporting it. Use the JSON, SRT preview, Files and Clips tabs for their corresponding results. Files exposes the artifacts produced by that run; enabled formats determine which files appear. Clips is useful when split media was retained.
Copy caption copies the result text. Results ZIP packages the run's relevant artifacts for download. Open Output, Open Caption and Reveal Clip open the corresponding local locations where supported. Open in Caption Editor carries the output into the review workflow. A remote browser cannot make a server-side folder appear on your own PC; use the downloadable files or ZIP there.
The recorded example completed with 1 done, 0 skipped, 0 failed, five detected segments and 755 generated tokens in 42.18 seconds. That is a measured run on an RTX 5090, not a promise for other hardware.
The total time includes preparation and loading. The displayed token rate describes model decoding, often the most recently completed segment. Context usage reports how much of the prompt/media/reply window was consumed. Read the item table and live log together when a run is skipped, rejected, cancelled or fails.
3. Choose the right family, precision and memory plan
Five model families with different jobs
TimeChat Captioner GRPO 7B takes audiovisual video and produces native timestamped structured segments. Its task variants can flatten those fields into motion paragraphs, audiovisual captions, chapters or subtitles. It always needs an audio timeline; the app supplies a silent timeline for silent video. A single native request has a 60-second model limit, and preprocessing may impose a lower practical limit.
AVoCaDO handles visual or audiovisual video. Start with its aligned audiovisual or visual-only task. Its declared duration limit is 100 seconds, subject to the current media/context budget. Unrelated custom prompts may move away from its training distribution.
Qwen3-Omni Instruct handles video, video with audio, standalone audio, images and text. It is the general-purpose choice for custom captions, OCR, transcription-style tasks and multi-turn chat.
Qwen3-Omni Thinking accepts the same media categories and separates intermediate reasoning from the final answer. Give it enough output budget for both. Keep reasoning out of training captions unless you explicitly need a separate analysis artifact.
Qwen3-Omni Captioner is a specialized audio-only, prompt-free sound-description model. Direct use takes one audio file up to 30 seconds, with no system or user prompt. It has no Chat mode. The dataset audio-caption stage can handle longer extracted audio by captioning windows of at most 30 seconds and joining them.
Folder batching is an application workflow: it can process individual compatible items even when a family does not support multiple media inputs inside one model request. A model's per-request constraints still apply to every item.
Native Transformers or GGUF
BF16, INT8 ConvRot and INT4 ConvRot W4A8 use the native Transformers path. Lower-precision variants reduce checkpoint size and often memory pressure. GGUF Q4 and Q8, available for the three Qwen3 families, use a private llama-server process plus a multimodal projector. The projector is required; downloading only the main GGUF file is incomplete.
Native Qwen audiovisual processing and GGUF video delivery are different. GGUF receives chronological sampled images plus separate audio; those frame and audio tokens are not interleaved by timestamp. Test the backend against the kinds of motion and sound alignment that matter in your dataset.
Choose VRAM preset before manually refining other settings. It coordinates precision, frames, pixels, output tokens, attention and offload. Selecting a universal preset then applies its saved values exactly; it does not normally reapply the tier and silently replace the workflow. An incompatible selected variant can still be substituted, with a note in the interface.
GPU chooses a physical card. Keep Subprocess mode enabled for isolated CUDA work and reliable unloading. Data-parallel GPUs is for distributing folder items across two or more workers, each with its own model copy; it does not combine the cards into one larger pool of VRAM. A batch needs enough compatible items to benefit.
Understand block swap
Automatic block swap estimates the decoder layers that can remain resident while leaving a VRAM reserve. Swapped layers live in system RAM and travel through the GPU during decoding. More swapping can fit a larger checkpoint, but transfers can make generation much slower.
The yellow estimate in this example is a useful warning: even an all-swapped plan can exceed the currently available budget because model towers, activations, staging buffers and the current media request still need memory. Reduce the media/context budget or use a better-fitting variant. Turning every layer into a swapped layer does not make the rest of the model free.
Decoder layers to block-swap becomes manual when Automatic block swap is off; zero keeps all decoder layers resident. VRAM to keep free, Swap slots, Plan slack and Pinned RAM budget refine the plan. Offload MoE experts is a separate legacy path and disables block swap. The CUDA allocator cap aims to produce a recoverable OOM instead of a severe Windows shared-memory slowdown.
Keep model loaded reuses the worker between jobs. Idle unload frees it after inactivity; zero disables that timer. Unload model explicitly frees the resident model and worker. Changing variants also releases the old model. The next run recreates the worker when needed.
Attention backend offers available optimized implementations with an SDPA fallback. torch.compile can spend substantial time on the first run. Start with it off while establishing a reliable workflow. The two compile modes avoid explicit CUDA graph replay for these decoders; failures can fall back to eager execution. Clear compile caches is a troubleshooting action, not a step to repeat before every job.
Native example
A second live example used TimeChat INT4, a five-second numeric trim and TimeChat — Wan motion paragraph. The native model still ran its structured task, and the application flattened the relevant fields. With enough token allowance, this run completed in 17.8 seconds.
A small token budget can stop a structured reply before it covers the entire input. Check the finish reason and the actual temporal coverage. A short probe that reaches its budget is useful for checking loading, but it is not a complete caption-quality evaluation. AVoCaDO INT4 was also exercised on the trimmed audiovisual input and completed successfully.
4. Prompts, reusable instructions and generation
Pick a task before editing its wording
The Task / prompt preset list is filtered by family and the first selected input modality. Changing the task can also change sampling and other recommended generation values. Read the resulting User prompt, not just the preset name.
System prompt sets the broader instruction; User prompt contains the concrete request. If you want a fully independent instruction, select Custom — free text and then fill both fields as needed. For example: “Describe only visible objects, actions, camera position and lighting. Use one concise paragraph.” The prompt-free Captioner is an exception and cannot be turned into a custom-prompt model.
Use Prompt name → Save prompt to retain a system/user pair in My prompts. Load prompt reapplies the selected pair; Delete prompt removes that library entry. This is separate from a universal preset, which stores the broader workflow. A save/edit/load round trip was checked with a personal visible-only prompt.
Template variables
Use the fields under Template variables to personalize compatible tasks. A trigger word such as ohwx identifies the concept you plan to train. The trigger is a deliberate dataset choice, not a word to leave in every unrelated caption.
Caption language, Source language and Target language have different roles: descriptive language, spoken language and translation destination. Caption length is a natural-language request, while Maximum new tokens is a hard generation limit. Subject class, Avoid list and Extra instructions only affect templates that insert them. Resetting prompts to the preset restores its template wording.
The ordinary variables are TRIGGER, LANGUAGE, SOURCE_LANGUAGE, TARGET_LANGUAGE, CAPTION_LENGTH, SUBJECT_CLASS, AVOID and EXTRA_INSTRUCTIONS, written with double braces. {{TRANSCRIPT}} is resolved later using the Whisper speech overlapping each clip. Context carry and dataset caption templates have their own placeholders, described below.
Control the answer budget and repeatability
Keep Sample tokens off, or temperature at zero, when you want greedy decoding. With sampling on, Temperature, Top-p and Top-k control candidate selection. A fixed Seed can make sampled runs more reproducible under the same model and execution conditions; -1 requests a fresh seed. Metadata records the seed actually used.
Use Repetition penalty gently. Native No-repeat n-gram size can stop loops but may suppress legitimate repetition; GGUF ignores that control. Use KV cache improves autoregressive decoding on native backends at a memory cost. Post-processing's repeated-sentence removal is a separate cleanup step.
Context length includes the prompt, media, history and reply. It is not the same as Maximum new tokens. The app's families expose up to a 32,768-token window. A reply limit above the remaining context does not create more space. GGUF reserves its window at server load, while native caches grow with use; long chats trim older turns when needed.
GGUF-specific tuning
GGUF maximum frames is an additional cap after the general sampling plan. JPEG quality affects the images sent to the server. Physical batch size controls prompt micro-batching and is a useful value to lower when prefill runs out of memory; Logical batch size serves a different server-level batching role.
The app normally clamps GGUF context by VRAM tier: 8k at 16 GB and below, 16k at 24 GB and 32k above that. Ignore tier context cap requests the larger selected window, but the server's fitter may still reduce it. Prompt cache reuse is exposed but the current multimodal llama.cpp path disables that optimization, including text-only turns with a projector loaded.
Min-p, Repeat last N, Presence penalty and Frequency penalty belong to llama.cpp sampling. GPU layers = 0 lets the app use automatic fitting; another value pins the layer count. MoE expert layers on CPU can lower GPU residency with a speed cost. Fit headroom leaves extra space for projector encoder buffers.
Startup timeout allows a large checkpoint to load from slow storage. Stream idle timeout stops a generation that sends no data for too long; zero waits indefinitely. Use Extra llama-server arguments only when you understand the option and its interaction with the app's own arguments. Start by changing one control, then inspect the log for clamping or fallback.
5. Sampling, scene splitting and caption cleanup
Processing Pipeline settings are saved and recovered with the Caption workflow. They change what reaches the model, how long inputs are divided, and what is written afterward.
Frame sampling and resolution
Sampling FPS asks for samples over time. Maximum frames caps the count. Maximum pixels caps the area of each resized frame while preserving aspect ratio; it is not a width or height. Minimum pixels is the lower smart-resize bound. Total pixel cap limits the whole request and can lower per-frame detail when many frames are selected.
The available strategies distribute the budget differently: target FPS follows a temporal cadence, uniform timestamps spread samples through the input, keyframes use keyframe locations, and adaptive sampling favors visual changes. Adaptive threshold only matters for the adaptive strategy. Lower values keep more changes. Setting Maximum frames to zero requests audio-only processing for Qwen3; the 7B families require visual frames and clamp that value with a logged warning.
Preview sampled frames runs a CPU preview without loading the caption model. The gallery displays a subset of the plan, with timestamps. Check whether important events are represented. Normalize clip re-encodes to the chosen format, frame rate and model-divisible shape. Use video audio follows family support. Models receive 16 kHz internally; the audio sample-rate control affects extracted/saved or normalized media.
Automatic rejection
Enable automatic rejection only after testing thresholds on representative footage. Minimum duration rejects tiny clips. Black-ratio and black-luma thresholds measure near-black content. Motion and sharpness thresholds can exclude static or blurred footage. Quality frames chooses how many samples are analyzed. Audio requirements and silence thresholds deal with missing or mostly silent audio.
A dark night scene can be a valid sample. A deliberately static shot can also be useful. Aggressive rejection settings can remove good data, so review the reported rejection reason before widening a batch.
Whole input, scenes, fixed length or trainer length
Whole keeps the input together unless a model/context limit requires splitting. Scenes uses detected content boundaries when scene detection is enabled. Fixed uses a chosen number of seconds. Trainer uses a trainer's frame-count and FPS guidance. The model-limit ceiling shown by the app reflects the current frame, pixel, audio and output-token budgets.
In Scenes mode, choose the detector and threshold, then control minimum/maximum scene length, merging and fade detection. Lower content thresholds tend to create more cuts. Downscaling speeds analysis. Minimum scene length and merge-below duration solve different problems: one affects detection, the other combines short detected pieces.
The real detector found five segments in the lightning video. A flash can look like a content change even when the camera does not cut. Use this preview to decide whether Scenes mode is suitable or whether fixed chunks better match your training objective.
Split quality and continuity
Fast stream copy is quick but cannot always place exact boundaries at arbitrary timestamps; Precise re-encode trades processing time for precise cuts. The dedicated sub-split tool can fall back to precise encoding when copy timing is inexact. The selected codec, CRF/CQ, encoder preset and audio bitrate control re-encoding. Lower CRF/CQ preserves more detail and creates larger files. An unavailable NVENC encoder can fall back to libx264 with a logged warning.
Use Save produced clips when you need media paired with the generated text. Sub-split overlap repeats some source time across neighboring chunks. That helps boundary context but can duplicate words or events. The recorded speech-caption test used three five-second-bounded windows with half-second overlaps.
Carry previous chunk context can feed the tail of the preceding caption into the next request for AVoCaDO and Qwen3 Instruct/Thinking. Context carry words limits that tail; Context carry prompt places it through {{CONTEXT}}. It is not used for TimeChat or the prompt-free Captioner.
Clean the final caption
Set a prefix or suffix when every caption needs consistent surrounding text. Trigger injection uses the shared Trigger word. Word replacements accepts one find;replace pair per line, with case, whole-word and regex options. Preview broad changes in the Caption Editor when you need to inspect their impact on an existing dataset.
Whitespace collapse and repeated-sentence removal clean model text. Caption join separator can be a space or a tag-style comma separator. Maximum caption characters is applied to the final assembled text and trims at a sensible boundary. It does not reduce inference cost the way a smaller token budget does.
Plain text is always written. Other choices add structured JSON, caption SRT/VTT or dataset JSONL. Minimum subtitle cue and Subtitle line width shape caption subtitle files. Caption subtitles describe events; use Whisper when the goal is spoken-word subtitles.
Save reasoning keeps Thinking output separate. Summarize segments performs an extra text-model pass over completed segment captions and writes a summary/chapter artifact. It needs Qwen3 Instruct or Thinking; other families log a skip. The summary has its own prompt and token limit. An actual three-segment audio run produced TXT, SRT, VTT, JSONL and a summary file for this guide.
6. Transcribe speech with Whisper
Choose Transcribe when the deliverable is the words people said, timed subtitles or transcript formats. Upload audio/video, use a local File path, or choose Folder batch. Inspect the audio preview and trim only when needed.
Select Whisper Quality (large-v1) in the universal preset bar.
Set a known language when practical, or use automatic detection.
Choose the device/GPU and retain float16 for the tested CUDA path.
Click Start Transcription.
Review the transcript and timing, then download the desired files or Results ZIP.
The recorded 11-second speech sample completed in 9.7 seconds total and produced two timed segments. Completed segments stream during work; the final Transcript view is plain text. SRT preview shows cues, Segments exposes segment information/confidence, JSON gives structure, and Files offers the exported artifacts.
Copy transcript, Open Transcript, Open Output, Retry failed, Cancel and Open in Caption Editor follow the same practical pattern as Caption. Folder batches have recursive scanning, item limits, overwrite/skip and next-to-source output options.
Model and runtime
The quality preset uses large-v1. Turbo is a faster alternative to evaluate, and the model field also accepts a custom Hugging Face repository ID containing a slash. The app reports download size and readiness. Download / Verify, Refresh and Delete model files manage the selected Whisper model.
Whisper runs in its own CTranslate2 worker and uses CUDA runtime libraries independently of PyTorch's CUDA 13 stack. Device: auto can fall back to CPU/int8 if CUDA is unavailable; forcing CUDA exposes its runtime error. Compute type, GPU and CPU threads should match the device you intend to use. A CPU fallback can be much slower than the GPU example.
Language and vocabulary
Set Language to the spoken language, not the language you want in a visual caption. Translate to English asks Whisper for English translation. Initial prompt can supply names and domain vocabulary. Repeating it every window keeps that guidance present on long recordings. Prefix biases the first decoded window; Hotwords provides recognition hints. These are guidance, not guaranteed corrections.
Decoding and long recordings
The quality preset starts with beam/best-of 5, temperature 0, repetition penalty 1.2 and 30-second chunks. Beam size affects beam search; Best of is relevant to nonzero-temperature sampling. Patience, length/repetition penalties and n-gram controls refine search.
Compression-ratio, log-probability and no-speech thresholds help detect questionable output. Previous-text conditioning carries language context between windows and can also propagate mistakes. The prompt-reset threshold controls when that context is discarded. Suppress-blank, suppress-token IDs and maximum initial timestamp are specialist decoding controls.
Max new tokens = 0 uses the Whisper model default per window. Chunk length controls windowing. Hallucination silence threshold, language detection probability/segments, and Use batched inference / Batch size handle difficult or large recordings. Increase batch size only after checking memory and output quality.
Word timing and output formats
Enable Word timestamps for word-level timing. Normalize word timestamp output rebuilds sentence-aware cues. Word highlighting is unavailable while normalization is enabled, so choose the mode that matches your subtitle presentation. Prepend/append punctuation fields decide which word receives punctuation during alignment.
Choose from SRT, VTT, TXT, LRC, TSV and JSON. Add timestamps to filenames when repeated exports must coexist. Keep JSON or TSV when you need structured timing downstream; use TXT for a plain transcript and SRT/VTT for subtitle players.
Voice activity detection
Silero VAD filters nonspeech before decoding. Its threshold controls speech probability; minimum speech, maximum speech, minimum silence and speech padding control region boundaries. Padding can protect word edges, while over-aggressive filtering can remove quiet speech.
Review nonspeech recordings especially carefully. In the thunderstorm test, Whisper produced a short word-like description even though the source was intended as environmental sound. That is a reason to review the transcript and choose a sound-caption workflow for sound effects, rather than accepting every transcription as spoken evidence.
Use Whisper during a caption run
Enable Also transcribe speech with Whisper during caption runs to transcribe each audio/video input before generating clip captions. Select transcript sidecar formats and a suffix. It uses the model and decoding settings in Transcribe.
When Inject transcript into the caption prompt is enabled, each clip receives only the overlapping recognized speech. Put {{TRANSCRIPT}} directly in a compatible prompt, or let the wrapper append it automatically. The recorded audio test confirmed clip-local text and injection flags in the structured result. Verify the transcript itself before treating it as exact dialogue.
7. Batch folders and separate video/audio caption parts
Build a predictable batch
Use Folder batch for a directory of media. A ZIP upload is extracted below Outputs/uploaded_batches and becomes the selected folder. The light scan reads filenames; media probing happens during processing.
Choose media kinds, turn Scan subfolders on when needed, and narrow File name filter with glob patterns such as *.mp4 or multiple patterns separated by semicolons. Limit items = 0 processes the whole eligible queue. Start with a small limit while checking a new workflow.
Batch output folder mirrors source names and relative directories. Save outputs next to the source files instead puts captions, layout folders, transcript sidecars and saved clips beside their sources. Metadata, the run log and working files remain in a numbered batch run folder. Keep a backup before overwriting valuable reviewed captions.
Overwrite existing captions controls repeated work. A split audio/video layout needs its required parts, not merely any TXT with the same stem. The verified ZIP example began with a visual TXT, generated its missing audio part, and then skipped the completed item on the next run.
Choose speech, sound or both
An Audio caption source other than Off creates separate video_caption/ and audio_caption/ parts. Choose Whisper for spoken words, Captioner for environmental/music/sound description, or both.
Video caption source → Reuse existing caption files skips the main visual-caption model. It searches a clean video_caption part first, then a same-stem caption, then a caption beside the source. This is useful for adding sound to a visually reviewed dataset. Sound-caption model can auto-match precision/backend or use an explicitly selected Captioner variant.
The audio template accepts {{TRANSCRIPT}}, {{SOUND_CAPTION}} and {{FILENAME}}. The merged template additionally accepts {{VIDEO_CAPTION}} and {{AUDIO_CAPTION}}. Choose paragraph, per-segment lines or clip-local timestamped Whisper text. Empty parts collapse cleanly.
A typical next-to-source layout is:
dataset/ clip.mp4 clip.txt video_caption/ clip.txt audio_caption/ clip.txt
Write merged caption beside each clip creates the outer clip.txt. Turn it off when separate parts are the desired deliverable. The empty-audio policy can keep only the visual text or write a chosen placeholder. Scene/fixed/trainer splits use equivalent parts in their segment folders.
In the verified existing-caption batch, the original visual text stayed intact in video_caption, the audio part was added, and the merged TXT contained both. The Captioner also made unsupported claims about stereo placement and a rural setting. Remove such claims during review. The app resamples model audio internally; a model's confident wording does not establish the source's channel layout or location.
8. Ask questions in Chat
The Chat model comes from Caption → Model variant. Qwen3 Instruct and Thinking support multi-turn conversations with text, image, audio and video attachments. TimeChat and AVoCaDO support one question about exactly one video per send. Captioner has no chat mode.
Select a chat-capable model.
Add media through Attachments or Media path when the question needs it.
Write a specific question. Enter sends; Shift+Enter adds a line.
Read the answer, then ask a follow-up.
Save the exchange when it is useful.
The live image example first asked for a two-sentence street description, then asked for one short training caption. The second turn used the conversation context and returned “Nighttime street intersection with cars and streetlights.”
Attachments belong to the turn they accompany and clear from the composer after sending. Earlier media remains associated with its conversation turn. Stop interrupts a reply; Clear history starts over; Copy last answer copies the final reply; Save conversation writes JSON and Markdown in a new output folder.
Use System prompt for conversation-wide directions. Caption prompt as first message copies the current Caption prompt into the composer. Chat has its own temperature, Top-p, Top-k, token limit, repetition penalty, seed and thinking controls, stored by universal presets. The context window is shared with the selected model and includes history/media.
Thinking exposes a separate reasoning panel and a final answer. The recorded image question completed with 133 generated tokens and an EOS finish. Reasoning is a model-produced explanation; verify its claims against the media. Keep only the reviewed final answer when preparing ordinary training captions.
9. Review captions before exporting a dataset
Scan, select and edit
Open the editor from a completed run or set Caption folder and click Scan. Enable Recursive for nested batches and segment folders. The queue shows file, caption excerpt, length, approximate tokens, flag and status. Table and Gallery views provide different browsing styles; page controls and Per page handle larger queues.
Select an item, inspect its media and edit Caption. With autosave enabled, changes are written atomically after about 0.7 seconds without another edit. Save is also available explicitly. TXT edits are mirrored into the text field of a JSON sibling. Caption parts displays separate video/audio parts read-only when present.
Approve marks an item and advances; Reject marks it for rejection. Previous/Next and the editor keyboard shortcuts speed review. Caption-only items can be reviewed even when source media is unavailable, but they cannot become a normal paired-media export.
Filter the review queue
Choose filters for the desired flags, text, length, status or missing captions, then click Apply filters. Changing a dropdown alone does not apply the filter. Reset clears the filter values. The filtered scope matters before bulk editing or regeneration; inspect the queue count.
Find and replace
Enter Find and Replace values, choose case/whole-word/regex behavior and scope, then run the preview. Inspect the highlighted changes and affected counts before applying. The live test previewed four replacements across three caption files before saving them. Regex errors should be corrected before applying; a broad expression can affect much more than one word.
Bulk editing
Bulk edits can add a consistent prefix, suffix or trigger across the chosen captions and apply available cleanup options. The editor's bulk-trigger field is separate from the Caption tab's template Trigger word. Align them deliberately when they refer to the same training concept.
Regenerate selected or filtered captions
Use regeneration for a caption that needs a new model answer. Choose the model and prompt override when needed, run the selected item, then inspect the old/new diff. The app preserves a before state and offers Revert for the last change.
Bulk regeneration requires confirming the number and scope of captions. The live test filtered to the one approved item, confirmed one caption, and completed with no failures. Review the regenerated text again even when its previous caption was approved.
Export approved pairs
Select the destination and available export options, then export approved items. Read all counts: exported, not approved, missing media and errors. The verified export produced one approved video/TXT pair, excluded five unapproved items, and created a ZIP. The exported segment measured 5.1051 seconds and shared its stem with the TXT file.
An export is a snapshot. Later regeneration of a source caption does not retroactively rewrite an already created export. Export again into the intended destination when you want the new text.
Dataset statistics
Statistics summarize caption coverage, length, token estimates, vocabulary and trigger usage for the scanned items. Trigger coverage uses the current Caption task Trigger word. In this example, the task trigger was ohwx while a bulk edit had inserted stormscene, so zero coverage for ohwx was expected. A count is useful only when it measures the token you actually intend to train.
10. Prepare clips and trainer configuration
Clip fitness
Set Source folder, choose the trainer, resolution bucket and resize policy, then Analyze. The tool checks duration, source FPS/frame counts and suggested fit. Keep AR preserves shape, Letterbox pads, Crop removes edges, and Area targets area while respecting the selected policy. The bucket preview reports the proposed dimensions and crop/pad amounts without changing the source.
Write plan JSON saves a timestamped inspection plan. It does not materialize all the proposed processing. The displayed frame recommendation is a target; a source at a different FPS must be resampled or configured consistently in the trainer.
Sub-split existing videos
The dedicated tool scans a video folder recursively and writes collision-safe per-source output directories. Set Target seconds, Overlap and Cut mode, then Run sub-split. The final segment can be shorter. The real 20.020-second input produced five clips in the recorded five-second test: four approximately five-second clips and a final 0.033-second tail after frame rounding. Remove or merge such tiny tails before training, and recheck clip fitness.
Keep source-time overlap in mind when counting unique training content. Verify resulting duration and audio alignment on a few outputs before preparing a large dataset.
Generate Kohya / Musubi TOML
Set Dataset root to reviewed direct-media folders or their parent. Kind can auto-detect video. Configure caption extension, repeats, resolution, batch size, buckets and upscaling behavior. A folder prefix such as 10_character can supply its repeat count.
For video, check Target frames, frame extraction, stride/sample, maximum frames and source FPS. The Wan 81f, LTX 121f/25fps and MiniMax H3 124f buttons are shortcuts; confirm they match the training setup you will actually use. Set Output path and click Generate. Inspect the TOML before passing it to a trainer; generation does not start training.
11. Recover, maintain and troubleshoot
Reuse previous runs
Run history lists recent caption, batch, transcription, regeneration and saved chat outputs. Refresh after saving new work. Select a row before using its actions. Chat rows show their model, message count and last answer, and can open their export folder. Conversation exports do not pretend to be caption datasets or complete run-setting metadata, so incompatible editor/recovery actions remain disabled.
Recover recorded settings
Use a metadata file, Metadata path or Recent run, then Load to compare saved and current values. Apply to UI restores compatible settings; Apply model + prompt only narrows the scope.
Also restore input/output paths is a separate opt-in. Application model/temp/theme paths are not restored from a run, and unavailable GPU indices are skipped. Loading settings is not the same as restarting an unfinished job. Confirm the input paths still exist and rerun deliberately.
Global paths and preferences
Outputs apply to new runs after saving. Temporary, Models, Logs and FFmpeg paths need a restart. The repaired Windows launcher and manual downloader honor the saved Models directory; an explicitly supplied VCAP_MODELS_DIR still takes precedence. Other VCAP directory environment overrides also take precedence over saved preferences.
Theme is stored in this browser and excluded from workflow presets/run metadata. Dark, Light and System are available. Save-processed-files keeps intermediate media; enable it when inspection matters and budget extra disk space. The recursive default, desktop/sound notifications and automatic single-job folder opening are everyday convenience choices. Desktop notifications may require the browser's own permission.
System health
Copy the environment report when asking for support. It identifies OS, Python, libraries, FFmpeg, attention backends, GPU/driver information and free disk space. Live VRAM/RAM/shared-memory indicators help separate model memory pressure from unrelated GPU activity. Polling pauses on hidden tabs and catches up when visible.
Check for updates compares release information without installing an update. Changelog provides expandable release notes. Check them before updating, especially when a new version changes preset behavior or file formats.
The runtime panel shows the pinned llama.cpp build, whether llama-server runs, CUDA support and checkpoint readiness. Install / repair llama.cpp retries the managed installation; use Refresh afterward. Build logs help diagnose missing nvcc, compiler or CUDA dependencies. The compile-toolchain section reports whether full or fallback compilation is available.
Select an exact model variant before Download, Verify or Delete. Verification checks local integrity; the recorded TimeChat BF16 verification passed. Cancellation leaves resumable download state where supported, and a later download resumes it. Partial files do not count as ready checkpoints. Model deletion names the selected files and their disk usage in a confirmation step. Do not use it as a routine unload action.
Whisper has a separate alias inventory and download/verify/delete controls. Refresh model status, Unload model and Open logs folder help inspect a resident worker or a failure. Unloading releases memory while retaining the checkpoint on disk.
Cancel and retry
The Cancel control becomes available during a job. It opens a confirmation bar; Yes, cancel stops the worker, while Keep running dismisses it. Caption Esc cancellation uses an eight-second confirmation window. The live cancellation test finished with one cancelled item and no failed items.
Completed artifacts remain useful after cancellation. Retry failed uses the saved failed-item context when available; it is not a universal resume button for any old folder. To resume a batch, preserve its output location, keep overwrite off and start the remaining eligible work again. Check the item counts after each run.
Fast diagnosis
The file will not preview: check that the path exists on the server machine, inspect its extension/media details, and verify FFmpeg and FFprobe. Set Global Settings → FFmpeg path and restart when automatic lookup fails.
The model is missing: confirm Models directory, environment overrides, the exact variant and disk space. Use Download/Verify. GGUF requires its projector and a working llama-server.
GPU memory runs out: close unrelated GPU-heavy work, use the correct tier, reduce frames/pixels/context, prefer a fitting precision and leave reserve. OOM retries reduce frame/pixel budgets and record that degradation.
Windows becomes extremely slow: inspect shared GPU memory. Paging can be slower than an explicit OOM. Prefer a smaller resident model or a realistic block-swap plan.
The first compiled run is slow: allow for compilation, verify the toolchain or disable torch.compile to isolate it. Clearing caches forces recompilation later.
The caption ends early: inspect finish_reason and token budget. Structured tasks and Thinking may need more reply tokens; a larger reply allowance also reduces room for media.
A prompt seems ignored: verify family/modality filtering and the rendered prompt. Select Custom before supplying independent wording. Captioner deliberately takes no prompt.
Captions contain guesses or repeated phrases: review the source, simplify the task, check sampling and token limits, use appropriate repetition controls and edit before approval.
A batch is skipped: inspect existing requested outputs, layout requirements, filters, recursion, item limit and overwrite state.
The editor export is empty: check approval flags, active filters, media availability and the explicit export counts.
Settings changed after a preset choice: selection applies immediately. Load reapplies it; Reset uses defaults. Save your own named workflow after tuning.
A speech-only task returns sound descriptions: use Transcribe/Whisper or a compatible ASR task; the specialized Captioner describes audio and does not guarantee word-for-word transcription.
Shortcuts are scoped to the active tab. Keep focus outside text fields when using arrow-key item navigation. Ctrl+S works while editing the Caption textbox; Ctrl+Enter and Ctrl+Delete approve or reject. Use the visible confirmation/status as the authority when stopping work.
Related posts
All JoyCaption Apps: 1-Click Installers With Batch Captioning
Ultimate Image Processing App: 1-Click Batch Cropping, SAM 3.1 Masking
Ultimate Image Captioner Pro: Qwen3 VL & JoyCaption JSON Captioning
All posts in this topic: Captioning, Dataset and Image Processing Tools · New to SECourses? Start Here: Best and Most Popular SECourses Posts