Deep Learning (DL) purposes usually require processing video knowledge for duties akin to object detection, classification, and segmentation. Nevertheless, standard video processing pipelines are sometimes inefficient for deep studying inference, resulting in efficiency bottlenecks. On this put up will leverage PyTorch and FFmpeg with NVIDIA {hardware} acceleration to attain this optimisation.
The inefficiency comes from how video frames are sometimes decoded and transferred between CPU and GPU. The usual workflow that we might discover within the majority of tutorials observe this construction:
- Decode Frames on CPU: Video information are first decoded into uncooked frames utilizing CPU-based decoding instruments (e.g., OpenCV, FFmpeg with out GPU assist).
- Switch to GPU: These frames are then transferred from CPU to GPU reminiscence to carry out deep studying inference utilizing frameworks like TensorFlow, Pytorch, ONNX, and many others.
- Inference on GPU: As soon as the frames are in GPU reminiscence, the mannequin performs inference.
- Switch Again to CPU (if wanted): Some post-processing steps might require knowledge to be moved again to the CPU.
This CPU-GPU switch course of introduces a major efficiency bottleneck, particularly when processing high-resolution movies at excessive body charges. The pointless reminiscence copies and context switches decelerate the general inference pace, limiting real-time processing capabilities.
For example, the next snippet has the standard Video Processing pipeline that you simply got here throughout if you find yourself beginning to be taught deep studying:
The Resolution: GPU-Based mostly Video Decoding and Inference
A extra environment friendly method is to hold all the pipeline on the GPU, from video decoding to inference, eliminating redundant CPU-GPU transfers. This may be achieved utilizing FFmpeg with NVIDIA GPU {hardware} acceleration.
Key Optimisations
- GPU-Accelerated Video Decoding: As a substitute of utilizing CPU-based decoding, we leverage FFmpeg with NVIDIA GPU acceleration (NVDEC) to decode video frames immediately on the GPU.
- Zero-Copy Body Processing: The decoded frames stay in GPU reminiscence, avoiding pointless reminiscence transfers.
- GPU-Optimized Inference: As soon as the frames are decoded, we carry out inference immediately utilizing any mannequin on the identical GPU, considerably decreasing latency.
Arms on!
Stipulations
In an effort to obtain the aforementioned enhancements, we shall be utilizing the next dependencies:
Set up
Please, to get a deep perception of how FFmpeg is put in with NVIDIA gpu acceleration, observe these instructions.
Examined with:
- System: Ubuntu 22.04
- NVIDIA Driver Model: 550.120
- CUDA Model: 12.4
- Torch: 2.4.0
- Torchaudio: 2.4.0
- Torchvision: 0.19.0
1. Set up the NV-Codecs
2. Clone and configure FFmpeg
3. Validate whether or not the set up was profitable with torchaudio.utils
Time to code an optimised pipeline!
Benchmarking
To benchmark whether or not it’s making any distinction, we shall be utilizing this video from Pexels by Pawel Perzanowski. Since most movies there are actually brief, I’ve stacked the identical video a number of instances to offer some outcomes with completely different video lengths. The unique video is 32 seconds lengthy which provides us a complete of 960 frames. The brand new modified movies have 5520 and 9300 frames respectively.
Authentic video
- typical workflow: 28.51s
- optimised workflow: 24.2s
Okay… it doesn’t look like an actual enchancment, proper? Let’s take a look at it with longer movies.
Modified video v1 (5520 frames)
- typical workflow: 118.72s
- optimised workflow: 100.23s
Modified video v2 (9300 frames)
- typical workflow: 292.26s
- optimised workflow: 240.85s
Because the video period will increase, the advantages of the optimization change into extra evident. Within the longest take a look at case, we obtain an 18% speedup, demonstrating a major discount in processing time. These efficiency positive factors are notably essential when dealing with giant video datasets and even in real-time video evaluation duties, the place small effectivity enhancements accumulate into substantial time financial savings.
Conclusion
In right this moment’s put up, we’ve got explored two video processing pipelines, the standard one the place frames are copied from CPU to GPU, introducing noticeable bottlenecks, and an optimised pipeline, by which frames are decoded within the GPU and go them on to inference, saving a significantly period of time as movies’ period improve.
