My first project experience report. Some fixes that worked.

[doctorrajvanshi]

project_experience_report.md

[doctorrajvanshi]

Project Experience Report: History of the Internet (OpenMontage)

Overview

This document serves as a post-mortem and experience report for the creation of the "History of the Internet" explainer video using the OpenMontage pipeline. We successfully generated a script, visual scene plan, and required assets, and composed them into a final MP4 video using Remotion. However, several critical roadblocks in the open-source pipeline required manual intervention and codebase patching.

What Worked Well

• Pipeline Structure: The modular split between script.json, scene_plan.json, and asset_manifest.json provided a very clean architecture for reasoning about the video.
• Asset Generation: The AI tools (video selectors, TTS, music generation) were successfully orchestrated by generate_assets.py to create the building blocks for the video.
• Props Injection: Passing the composition state to Remotion via .remotion_props.json made it easy to diagnose what the React layer was receiving.

Roadblocks & How They Were Fixed

1. Remotion Cross-Platform URI Resolution Bug

• What didn't work: Remotion composition initially failed because OffthreadVideo components threw errors. The pipeline silently downgraded to text cards.
• The cause: Explainer.tsx contained a bug in the resolveAsset function. When processing absolute paths on Linux/POSIX (paths starting with /), it blindly prepended file:///, resulting in malformed file://// URIs that the browser engine rejected.
• The fix: Patched resolveAsset in Explainer.tsx to handle POSIX absolute paths correctly by returning file://${posix} when the path starts with a slash, and file:///${posix} for Windows drive letters.

2. CORS / Public Directory Asset Fetching

• What didn't work: Remotion failed to load video assets due to CORS and absolute path restrictions when fetching local files.
• The cause: The project assets were located outside the remotion-composer root, and Remotion's staticFile() utility could not access them. video_compose.py attempted to inject absolute paths to bypass this, but that caused subsequent NameErrors and runtime conflicts.
• The fix: Updated the CLI command in video_compose.py (and testing via run_compose.py) to include the --public-dir=H:\OpenMontage\projects\history-of-the-internet flag. This correctly mounted the project directory, allowing staticFile() to serve assets dynamically via http://localhost:3000/assets/... without breaking CORS.

3. Asset Generation Misclassification (MP4s disguised as JPGs)

• What didn't work: Remotion crashed entirely with an EncodingError: The source image cannot be decoded.
• The cause: The generate_assets.py script contained a logic flaw. For assets designated as "type": "image", it correctly set the output extension to .jpg, but it mistakenly called the vid_tool (video generation model) with num_frames=33. As a result, the tool generated 15MB MP4 video files and saved them with a .jpg extension. Remotion understandably crashed when attempting to decode a video file as a static image.
• The fix: Rather than paying the cost/time to re-generate the assets, we used ffmpeg via the terminal to extract the first frame of the corrupt .jpg (MP4) files and overwrite them as true JPEG images.

4. Missing Audio / Prop Mapping Disconnect

• What didn't work: The final video rendered successfully, but it was completely silent.
• The cause: There was a structural mismatch between the Python pipeline and the React composition layer. video_compose.py generated .remotion_props.json with an array of narration segments (e.g., asset_id: "narration-s1") and a music asset_id. However, Explainer.tsx was hardcoded to look for a single string at audio.narration.src and completely ignored the segments array.
• The fix: Rewrote the audio mapping logic in Explainer.tsx. Added a .map() over audio.narration.segments that dynamically resolves the string path (e.g., converting "narration-s1" to assets/narration/s1.mp3) and injects individual <Audio> components inside <Sequence> blocks timed to their respective startFrame. Also added fallback logic to parse audio.music.asset_id.

Conclusion

While the end output wasn't a masterpiece visually, the core pipeline operated exactly as intended once the connective tissue between the Python orchestrator and the Remotion React runtime was repaired. The main takeaway is that the OpenMontage codebase requires stronger type-safety contracts between what the Python backend outputs (.remotion_props.json) and what the React frontend expects (Explainer.tsx), as well as stricter file-signature validation during asset generation.