Implementing a JPEG2000 codec plugin for Adobe Premiere — a technical overview

We have implemented a JPEG2000 codec plugin for video editing with Adobe Premiere. In this article I describe how it was integrated with Premiere and what are the benefits of JPEG2000 codec for Ultra HD 8K or VR video editing.
JPEG2000 codec is very popular in video processing workflows such as broadcasting or film production. When it comes to video editing, Adobe Premiere Pro CC is among the top choices for video editing professionals. It is a non-linear video editing system which is an all-in-one solution for editing, processing and producing high quality video.
One of the great features of JPEG2000 codec is its ability to open video in different resolutions. It is called resolution scalability and it allows users to instantly switch between different video resolutions without the need to create and manage proxy files. For example, while editing 8K video you can switch it to 4K or HD with one click whenever you want.
We work closely with Adobe to take advantage of these JPEG2000 codec features to improve the experience of editing high-resolution video content such as Ultra HD 8K or VR video. Thanks to this partnership, we have created our own JPEG2000 codec plugin for Adobe Premiere.

Performance is key for 8K editing

The JPEG2000 codec plugin is built on top of a Comprimato JPEG2000 GPU decoder. This is a software JPEG2000 decoder optimized to run on GPUs. It allows faster than real-time playback of 8K video and enables you to search and jump along a timeline like you do when editing standard HD video.
The plugin itself is designed to be as lightweight as possible. There is no additional pre- or post-processing involved, so the decoder is fully utilized all the time. In particular, the plugin always maps decoded video to the pixel formats supported by Premiere in order to avoid time consuming operations such as pixel buffer conversions. The plugin also takes advantage of the Adobe pixel buffer cache subsystem (see the code snippet) to avoid decoding the same frame multiple times.

// try to load requested frame from cache prSuiteError err = PPixCacheSuite->GetFrameFromCache(importerID, 0, frameIndex, 1, inFrameFormat, outFrame, NULL, NULL)); if (err != suiteError_NoError) { // frame not in cache, decode it the usual way … }

Editing and playback of VR or full 8K video is very hardware demanding. A system would typically require lots of computing power and very fast storage, in particular. To overcome this problem and to enable 8K editing even on lower-end systems, the JPEG2000 codec plugin integrates with the resolution scale-down interface of Premiere, see the picture. This allows users to edit 8K video in e.g. HD or even lower quality. And users can switch between the full resolution and any of the multiple lower resolutions instantly. Therefore they can always go back to the full resolution to see how a particular frame looks in 8K. And there is no need to generate any proxy files. This resolution switching is entirely under control of the user.

Picture 1: Setting playback resolution in Adobe Premiere Pro

Slow and fast storage

A typical problem with high bitrate video is storage bandwidth. Disks are simply not fast enough. The resolution scalability of JPEG2000 codec helps here as well. When editing a video in the scaled-down mode, there is no need to read the whole video file from the disk. Instead, only the part corresponding to the lower resolution is read. This is possible thanks to discrete wavelet transform (DWT), which effectively enables the video to be stored as a mipmap of cascading resolutions.
The plugin then loads only as much data as is necessary to decode the requested video resolution.

Picture 2: Mipmap structure of image with DWT decomposition ^[1]
 
The bandwidth issue is even more acute with NAS networked storage. NAS is shared among a group of users who are constantly competing for the disk bandwidth. On top of that, the network itself represents another bottleneck. Users are typically connected to the NAS with a 1G Ethernet connection. Which means either the NAS storage cannot support that many parallel users, or the network cannot handle multi-gigabit streams.
For example, lightly compressed 8K VR/360° video is about 450 MB/s or 3.6 Gbps. This is more than a 1Gbps network can handle. The same holds true for 4K VR video, as can be seen from Table 1 below.

Table 1: VR/360° video at 8K is about 450MB/s or 3.6Gbps. The same video at 4K resolution is about 150MB/s or 1.2Gbps.
8K VR video: 8192×8192, 4:2:2, 8bit, 15:1 compression. 4K VR video: 4096×4096, 4:2:2, 8bit, 10:1 compression
 
Ultra HD 4K, 8K or VR video editing in bigger groups is a really tough problem to crack. We are currently working on improvements to our Comprimato JPEG2000 codec plugin that will remove the differences between local and network editing. We will be announcing it soon. Please stay tuned …

Future plans

We will continue to develop and improve the JPEG2000 codec plugin for Adobe Premiere. In the very near future we will introduce a JPEG2000 Exporter plugin which will accelerate the encoding using our Comprimato JPEG2000 GPU engine. The Exporter plugin will implement a set of carefully selected codecs to target all the main exporting use cases while still preserving ease of use.
The plugin currently works with sequences only. Based on users’ feedback, we will soon be releasing support for the MXF container. The MXF will allow users to store JPEG2000 sequences and audio channels.
Soon we will be releasing support for editing from networked NAS storage which will preserve the interactivity of very fast local SSD storage. The goal is that users will not see any difference between editing from NAS and editing from a fast local SSD disk.
UPDATE: The plugin is now available as a Comprimato UltraPix product and ready for FREE DOWNLOAD
References
[1] By Alessio Damato (Own work) [GFDL (http://www.gnu.org/copyleft/fdl.html), CC-BY-SA-3.0 (http://creativecommons.org/licenses/by-sa/3.0/) or CC BY-SA 2.5-2.0-1.0 (http://creativecommons.org/licenses/by-sa/2.5-2.0-1.0)], via Wikimedia Commons.