字幕工厂
背景与动机：在 AGI 来临时做视频与自动化
阐明做视频的动机与自动化目标
在家这段时间，思考了很久是否要自己做视频。最终决定开始做，原因很简单，AGI 来临之际，把自己做成一个真人是一件很重要的事。毕竟不能比 AI 更准，比 AI 更快那就更不可能了。但怎么着比 AI 更有人的味道那我们肯定是更擅长的。 所以第一天就疯狂 Vibe Coding，捣鼓一个 AI 驱动的视频后期制作流水线，目标呢，自动完成从原始视频到专业成片的全部工作。
在肝了一天一夜后，我的第一个视频终于上线了。 显然，专业没有，流水线是有了！
能力与产出：它能做什么
这个搞了一天一夜的小流水线产品用来干嘛呢？
构建之初是想要能够自动完成视频中的音频识别、自动配字幕、中英互译翻译、指出内容重点、带上一些有趣的包装、自动制作封面、还能够渲染动画（Maybe）
它能做什么？
想象一下：你拍了一段 20 分钟的知识讲解视频，手动加字幕、翻译、做重点高亮、设计封面,这些工作可能需要一个剪辑师忙上大半天。而这个系统只需要你给它一个视频文件，它就能自动完成以下所有事情：
- 听懂你说了什么 — 用 AI 语音识别把每一个字、每一个词精确到毫秒级别地记录下来
- 变成漂亮字幕 — 自动断句、换行，像专业字幕组一样，中文不会断在奇怪的地方
- 翻译成英文 — 不是机械翻译，而是针对视频字幕这个场景专门优化的简洁译文
- 读懂内容重点 — AI 像一个资深编辑一样阅读全文，标出核心概念、关键数据、转折观点
- 自动做视觉包装 — 重点词句变成黄色高亮、关键论点弹出大字卡片、结尾自动生成号召行动的画面
- 生成封面 — 根据视频内容自动选配色、选版式、写标题，生成 YouTube 风格的封面图
- 渲染最终成片 — 把原视频、字幕、特效、音效全部合成为一个可以直接上传的 MP 4 文件
整个过程，你只需要运行一条命令，然后去喝杯咖啡。
工作原理与流程：它是怎么工作的？
AI 理解 + 规则执行，分步解释从听到合的机制
系统的核心思路可以用一句话概括：让 AI 做它擅长的理解**，让程序做它擅长的**执行****。
第一步：听 — 语音变文字
目前使用一个叫做 MLX Whisper 的 AI 模型来听视频。这不是普通的语音转文字。它把每个词出现的精确时间点都记下来。这就像给每个字打上了一个时间邮戳，后面所有的字幕对齐、特效出现都依赖这些精确的时间信息。
由于运行在苹果芯片上，这一步完全在你自己的电脑上完成，不需要联网，也不用担心隐私问题。
第二步：排 — 智能断句
拿到带时间戳的文字后，系统要把它变成人类看着舒服的字幕。这里有一套精心设计的规则：每行中文不超过 16 个字，在标点和自然停顿处断开，避免把因为和后面的内容拆到两行,这些看似简单的规则，背后需要考虑的关键点是，能够断句简单，不要出现太多长句子。
第三步：译 — 字幕级翻译
翻译不是把中文扔给 Google Translate。系统对翻译 AI 说的是：「你现在翻译的是视频字幕，不是文章。每行不超过 30 个英文字符，用口语化的短句，省略不必要的冠词。」这让英文字幕读起来像是原生英文视频的字幕，而不是生硬的翻译腔。
但显然这不是翻译的精华，一个简单的重写并不能带来高质量的翻译，但多次翻译后在优化，甚至抽取全文关键信息后再优化也会有诸多问题：翻译速度慢了怎么办？质量如何保障等等需要优化
第四步：懂 — AI 编辑审稿
这是整个系统最精妙的部分，也分成两个阶段：
阶段 A — AI 语义发现： 系统把完整的字幕脚本交给 AI。他的任务不是做视频特效，而是像一个资深内容编辑那样阅读全文，回答这些问题：
哪些是核心概念？哪里有关键数据？哪个观点出现了转折？哪句话是全文的结论？
阶段 B — 规则引擎编排： AI 标出重点之后，一套确定性的规则引擎接手。它决定：这个重点用什么颜色？放在屏幕什么位置？显示多大的字？停留几秒？需不需要配一个叮的提示音？这些规则保证了视觉风格的一致性，不管 AI 每次的理解有多少微小差异，最终画面的风格总是统一、克制、专业的。
为什么要分两步？因为 AI 擅长判断什么重要，但不擅长保持视觉风格一致。
把两个任务分开，各取所长。
第五步：画 — React 视频合成
有了所有素材和编排方案，系统使用一个叫 Remotion 的框架来画最终视频。Remotion 是一个特殊的画布：原始视频是最底层，字幕叠在上面，重点高亮再叠一层，结尾的感谢画面是最顶层,就像 Photoshop 的图层一样，一层层叠加出最终效果。
第六步：合 — 渲染输出
最后一步是把这些图层逐帧渲染成真正的视频文件。系统不是调用一个黑盒的命令行工具，而是通过一个精心设计的通信协议与渲染引擎实时对话：「现在渲染到 42% 了」第 1260 帧完成
这样你可以在终端里看到实时的进度条，而不是干等着不知道还要多久。
考虑还是用 ffmpeg 渲染可能更快一些。
它的大脑：VideoSummary
一次全局摘要指导后续封面、结尾与风格
整个系统有一个重要的设计：全局摘要（VideoSummary）。
在流水线启动的早期，AI 会对视频做一次整体分析，产出一份内容档案,包括视频的核心论点、目标受众、情感基调、内容类型等。这份档案就像一个导演的创作备忘录，后续所有环节都参考它：
- 封面设计参考它选配色和标题
- 结尾画面参考它决定是感谢观看还是下期预告
- 视觉风格参考它判断是科技蓝还是历史暖调
一次理解，处处复用。
这避免了不同模块各自为政、风格割裂的问题。
总结
智能与可控兼得的分工原则
流水线的核心是：AI 负责理解，规则负责美学，两者协同但互不越界。
AI 不会直接决定一个字该显示成多大,它只说这句话很重要，然后由人类预设的美学规则来决定重要的话应该怎么呈现。
这让最终产出同时具备两个通常矛盾的特质：智能（AI 理解内容语义）和可控（视觉风格 100% 确定性）。
对于创作者来说，它就是一个「把视频扔进去，拿成片出来」的魔法盒子。
Background and motivation: building video and automation in the age of AGI
A personal explanation of why I started making videos and why automation matters
While staying at home for a while, I spent a long time thinking about whether I should start making videos myself. In the end, I decided to do it for a simple reason: as AGI gets closer, turning yourself into a real, visible human presence becomes increasingly important. I cannot be more accurate than AI, and I definitely cannot be faster than AI. But being more human, more textured, more personal than AI is still something we do better. So on day one, I went straight into full-on vibe coding and started building an AI-driven video post-production pipeline. The goal was ambitious: automate everything from raw footage to a finished, polished video.
After grinding on it for a full day and night, my first video finally went live. Clearly, the professionalism still needs work. But the pipeline is real.
🎬 Tao Te Ching: Harmony Matters
⸻
Capabilities and output: what it is for
So what exactly is this one-day-and-one-night pipeline product supposed to do?
At the beginning, I wanted it to automatically handle speech recognition, subtitle generation, Chinese-English translation, key point extraction, some light visual packaging, automatic thumbnail creation, and maybe even animation rendering.
What can it do?
Imagine this: you record a 20-minute educational video. Adding subtitles, translating it, highlighting key ideas, and designing a thumbnail could easily take an editor half a day. This system, by contrast, only needs a video file from you, and it can automatically do all of the following:
- Understand what you said — It uses AI speech recognition to capture every sentence and every word with millisecond-level timing
- Turn it into clean subtitles — It automatically breaks lines and segments sentences so the subtitles feel polished, like those from a professional subtitle team
- Translate it into English — Not as stiff machine translation, but as concise subtitle-style English tailored specifically for video captions
- Understand the key ideas — AI reads the full transcript like an experienced editor, then identifies core concepts, important numbers, and turning points in the argument
- Add visual packaging automatically — Important words can become yellow highlights, major points can trigger large on-screen text cards, and the ending can automatically generate a call-to-action frame
- Generate a thumbnail — Based on the content of the video, it can choose color schemes, layout styles, and titles to produce a YouTube-style thumbnail
- Render the final video — It combines the original footage, subtitles, effects, and sound cues into a ready-to-upload MP 4 file
In the ideal version, you run one command and then go make yourself a coffee.
⸻
How it works: what is actually happening under the hood?
AI for understanding + rules for execution, explained step by step
The core idea of the system can be summed up in one sentence: let AI handle what it is good at, which is understanding, and let software handle what it is good at, which is execution**.**
Step one: listen — speech to text
Right now, I use an AI model called MLX Whisper to listen to the video. This is not just ordinary speech-to-text. It records the exact timing of each word. It is like attaching a timestamp to every spoken unit, which later becomes the foundation for subtitle alignment and timed visual effects.
Because it runs on Apple silicon, this step happens entirely on your own machine. No internet connection is required, and privacy stays intact.
Step two: arrange — smart subtitle segmentation
Once the transcript with timestamps is ready, the system needs to turn it into subtitles that feel comfortable for humans to read. This is where a carefully designed rule set comes in: each line of Chinese stays under 16 characters, breaks happen around punctuation and natural pauses, and common word groups are not split awkwardly across lines. These rules may sound simple, but the real challenge is to keep segmentation natural and avoid overly long sentences.
Step three: translate — subtitle-level translation
Translation here is not just throwing Chinese text into Google Translate. The system instructs the translation model like this: “You are translating video subtitles, not an article. Keep each line under 30 English characters, use short spoken phrases, and drop unnecessary articles where appropriate.” That makes the English subtitles feel closer to native video captions rather than textbook translation.
But obviously, that is not the full story of good translation. Simple rewriting alone does not produce high-quality subtitle translation. Multiple rounds of translation and refinement can help, and using extracted key information from the full transcript can improve quality too. But that introduces new tradeoffs: what happens when translation becomes too slow, and how do you guarantee quality consistently? These are still open problems that need more work.
Step four: understand — AI as editor and reviewer
This is the most interesting part of the whole system, and it has two stages:
Stage A — AI semantic discovery: The system passes the full subtitle script to the AI. Its job is not to generate visual effects directly, but to read the whole thing like an experienced content editor and answer questions such as:
What are the core concepts? Where are the key numbers? Where does the argument shift? Which sentence best captures the conclusion?
Stage B — rule engine orchestration: Once the AI marks the important parts, a deterministic rule engine takes over. It decides: what color should this highlight use? Where should it appear on screen? How large should the text be? How many seconds should it stay visible? Should there be a small notification sound? These rules ensure visual consistency. Even if the AI’s interpretation varies slightly from run to run, the final style remains unified, restrained, and professional.
Why split it into two steps? Because AI is good at deciding what matters, but not very good at keeping visual style consistent.
So the system separates the two jobs and lets each part do what it does best.
Step five: draw — React-based video composition
Once all the assets and layout decisions are ready, the system uses a framework called Remotion to compose the final video. Remotion works like a special kind of canvas: the original video sits at the bottom, subtitles are layered on top, keyword highlights sit above that, and the final thank-you screen becomes the top layer. It is a lot like Photoshop, where the final result is built by stacking layers together.
Step six: render — output generation
The last step is to render all of those layers frame by frame into a real video file. Instead of calling a black-box command-line renderer and waiting blindly, the system talks to the rendering engine through a more deliberate communication layer: “rendering is now at 42%,” “frame 1260 completed.”
That means I can see a live progress bar in the terminal instead of sitting there with no idea how long it will take.
That said, I am still considering whether using ffmpeg for rendering might be faster in some cases.
⸻
Its brain: VideoSummary
A single global summary that guides thumbnails, endings, and style
One of the most important design ideas in the system is the global summary (VideoSummary).
Early in the pipeline, the AI performs one full-pass analysis of the video and produces a content profile containing the main argument, target audience, emotional tone, content type, and more. This profile works like a director’s creative memo, and every later stage refers back to it:
- The thumbnail generator uses it to choose color palettes and titles
- The ending screen uses it to decide whether the right tone is thanks for watching or see you in the next episode
- The visual style system uses it to decide whether the piece should feel like cool tech blue or warm historical tones
One round of understanding, reused everywhere.
This prevents different modules from drifting apart and producing a fractured style.
⸻
Summary
The principle: intelligence with control
The core principle of the pipeline is this: AI handles understanding, rules handle aesthetics, and the two collaborate without crossing each other’s boundaries.
AI does not directly decide how large a word should appear on screen. It only says, this sentence matters, and then a human-defined aesthetic system decides how important content should be presented.
That gives the final output two qualities that are often hard to combine: intelligence because AI understands the meaning of the content, and control because the visual style remains 100% deterministic.
For a creator, it becomes a kind of magic box: drop a video in, get a finished video out.
讨论
登录后可评论、点赞、引用回复。