CV VideoPlayer — Once and For All | by Daniel Tomer | Dec, 2024

A Python video participant bundle made for laptop imaginative and prescient analysis

When growing laptop imaginative and prescient algorithms, the journey from idea to working implementation typically includes numerous iterations of watching, analyzing, and debugging video frames. As I dove deeper into laptop imaginative and prescient initiatives, I discovered myself repeatedly writing the identical boilerplate code for video visualization and debugging.

Sooner or later, I made a decision sufficient was sufficient, so I created CV VideoPlayer, a Python-based open-source video participant bundle, particularly designed for laptop imaginative and prescient practitioners that may remedy this downside as soon as and for all.

In the event you’ve ever developed an algorithm for video evaluation, you’ve most likely written some model of the next code that can assist you visualize and debug it:

import cv2
cap = cv2.VideoCapture()
ret = True
whereas ret:
ret, body = cap.learn()
algo_output = some_video_analsys_algorithm(body)
frame_to_display = visualizer(body, algo_output)
cv2.imshow(frame_to_display)
cv2.waitKey()

However in nearly all initiatives I’ve labored on this code was not often sufficient. Because the challenge went on I discovered myself including increasingly performance to assist me perceive what was occurring.

For instance:

• Navigation via the video backwards and forwards body by body.
• The power to file the output to a file.
• Supporting sources aside from a easy video file (body folder, stream, distant storage, and many others.)

However the factor that irritated me essentially the most was the dearth of interactivity. Utilizing this sort of code, The visualization is created earlier than rendering and can’t change as soon as displayed. And, whereas that is okay for easy algorithms, for the extra advanced ones, there may be simply manner an excessive amount of info wanted for every body. And with out the power to resolve, on the fly, what you need to show, you end up working the identical video time and again, every time with totally different visualization parameters.

This course of was tedious and exhausting.

CV VideoPlayer was born from the necessity for a easy customizable resolution for interactively rendering movies and frames. It permits any variety of overlays, sidebars, or another body edits, every of which may be simply switched on and off by the consumer throughout run time. let’s see an instance of how that is executed:

Set up

We begin by putting in the bundle utilizing pip set up cvvideoplayer

Enjoying vanilla video

We will then import the video participant and run an unedited video with the next code:

from cvvideoplayer import create_video_player
VIDEO_OR_FRAME_FOLDER_PATH = ""
video_player = create_video_player(video_source=VIDEO_OR_FRAME_FOLDER_PATH)
video_player.run()

It will open the video participant and let you play it with the spacebar or utilizing the arrows, it is going to additionally add some default built-in frame-edit-callbacks which we are going to elaborate on within the following part.

So as to add custom-built visualization to the video we are able to use the frame_edit_callbacks argument of the create_video_player constructor perform like so:

from cvvideoplayer import VideoPlayer
VIDEO_OR_FRAME_FOLDER_PATH = ""
video_player = create_video_player(
video_source=VIDEO_OR_FRAME_FOLDER_PATH,
frame_edit_callbacks=[
FitFrameToScreen(),
FrameInfoOverlay(),
KeyMapOverlay(),
]
)
video_player.run()

When unspecified, the default listing will likely be precisely the one within the instance above.

Constructed-in callbacks

There are a bunch of built-in callbacks to make use of corresponding to:

• FitFrameToScreen — Mechanically resizes the body to suit the display dimension.
• FrameInfoOverlay — Prints the body quantity and authentic body decision on the highest left nook.
• KeyMapOverlay — Mechanically detects and prints all accessible keyboard shortcuts (Additionally these added by the consumer).
• DetectionCsvPlotter — Plots Bounding containers laid out in a CSV with the next Header: frame_id, label, x1, y1, width, peak, rating
• FrameNormlizer — Permits the consumer to regulate the dynamic vary of the picture.
• HistogramEqulizer — self-explanatory

And extra are added with every model.

Making a {custom} callback

Right here is the place the usefulness of the bundle shines. So as to add your personal {custom} visualization you create a brand new class that inherits BaseFrameEditCallback and implements the edit_frame technique, for instance:

class MyCallback(BaseFrameEditCallback):
def __init__(
self,
enable_by_default: bool = True,
enable_disable_key: Optionally available[str] = None,
additional_keyboard_shortcuts: Optionally available[List[KeyFunction]] = None
**any_other_needed_params
):
tremendous().__init__(
enable_by_default,
enable_disable_key,
additional_keyboard_shortcuts
)
def edit_frame(
self,
video_player: "VideoPlayer",
body: np.ndarray,
frame_num: int,
original_frame: np.ndarray,
) -> np.ndarray:
"""
This perform receives the displayed body and may return it
after it has been altered in any manner fascinating by the consumer
Args:
video_player: an occasion fo VideoPlayer
body (): the body to be edited and displayed
frame_num ():
original_frame () the body earlier than any alterations
Returns: the edited body
"""
body = add_any_visalizations(body)
return body

Moreover, you’ll be able to add setup and teardown strategies by overriding these strategies within the mother or father class:

class MyCallback(BaseFrameEditCallback):
... 
def setup(self, video_player: "VideoPlayer", body) -> None:
"""
Optionally configure extra parameters in keeping with the
first incoming body
"""
def teardown(self) -> None:
"""
Optionally outline how the callback ought to shut when the
video participant is closed
"""

For every callback, CV Video Participant lets you add {custom} keyboard shortcuts that may change the visualization it does at run time.

Probably the most primary shortcut is enabling/disabling the callback and is created utilizing the enable_disable_key parameter like so:

my_callback = MyCallback(
enable_disable_key="ctrl+a"
)

The string handed right here may be any mixture of modifiers (ctrl, alt, and shift) with a letter or quantity for instance: “crtl+alt+s”, “g”, “shift+v”, “crtl+1” and so forth.

So as to add shortcuts that change the visualization itself, you’ll be able to override theadditional_keyboard_shortcuts property which returns an inventory of the dataclassKeyFunction .

from cvvideoplayer import KeyFunction
class MyCallback(BaseFrameEditCallback):
...
@property
def additional_keyboard_shortcuts(self) -> Checklist[KeyFunction]:
[
KeyFunction(
key="alt+r",
function=self.a_function_to_modify_the_visualiztion,
description="what this does"
)
]

A KeyFunction is constructed utilizing three arguments:

• The key argument — Identical as for enable_disable_key , The string handed right here may be any mixture of modifiers (ctrl, alt, and shift) with a letter or quantity for instance: “crtl+alt+s”, “g”, “shift+v”, “crtl+1”
• The description argument — That is utilized by the KeyMapOverlay callback to print all of the accessible shortcuts on the display.
• The perform argument — Needs to be a perform that accepts no arguments.

In lots of instances, the KeyFunction will obtain a perform that toggles some boolean attribute of the callback, which can change one thing that the edit_frametechnique does. So one thing like:

from cvvideoplayer import KeyFunction
class MyCallback(BaseFrameEditCallback):
...
@property
def additional_keyboard_shortcuts(self) -> Checklist[KeyFunction]:
[
KeyFunction(
key="alt+r",
function=self.a_function_to_modify_the_visualiztion,
description="what this does"
)
]
def a_function_to_modify_the_visualiztion():
self._draw_something = bool(1 - self._draw_somthing)

Many occasions, I discovered myself wanting to check two totally different visualizations facet by facet. For instance, evaluating two detectors or an algorithm’s output with the unique body with out modifications, and so forth.

To try this I added double_frame_mode which may be turned on by:

video_player = create_video_player(
...
double_frame_mode=True
)

The video initially of this weblog is an instance of what this mode seems to be like.

On this mode, you need to use “ctrl+1” and “ctrl+2″ to resolve which body visualization you need to management with the keyboard.

By default, each frames may have the identical callbacks accessible however in order for you totally different callbacks for the fitting body you need to use the right_frame_callback argument to provide the fitting body a unique set of callbacks (the left body may have those handed to the frame_edit_callback argument):

video_player = create_video_player(
...
double_frame_mode=True
right_frame_callbacks = [callback1, callback2, ...]
)

I Hope this software turns out to be useful for all of you. You probably have any concepts on the right way to enhance it, please let me know within the points tab on the challenge’s GitHub page, and don’t neglect to depart a star when you’re at it 🙂 …