Interactive Data Exploration for Computer Vision Projects with Rerun

instruments for information visualization and exploration have allowed me to iterate means faster on my pc imaginative and prescient initiatives, particularly when the issues I’m confronted with should not easy and I have to make algorithm or design choices primarily based on dynamic time-varying alerts. These alerts can typically be difficult to investigate by simply taking a look at quickly altering numbers plotted on the display or saved in a desk.

Whereas engaged on a few of these issues, I explored the built-in interactive components from OpenCV, however other than some sliders the choices there are very restricted, particularly when making an attempt to combine some animated plots. There’s some hacky methods to get dynamic plots from matplotlib into OpenCV, which I explored on this post.

I additionally explored totally different UI frameworks like tkinter, which I utilized in my last project for a sentiment evaluation visualization. I constructed a customized element that allowed me to show dynamic frames. Nonetheless it nonetheless didn’t actually really feel like the suitable device for the duty, particularly when making an attempt to work with interactive plots.

After which I stumbled upon Rerun. Each every now and then I uncover a device or framework that actually excites me, and this was precisely a type of occasions. Rerun is an open supply device for visualizing information sometimes discovered within the robotics area starting from easy time collection information and static photographs to advanced 3D level clouds, video streams or different varieties of multi-modal information. The demos look actually spectacular and the setup and code samples are so easy, I used to be instantly hooked.

So I made a decision to remodel my ball monitoring demo from a earlier mission and plot the information utilizing rerun. Let me present you the way simple it’s to make use of and create interactive functions!

Quickstart

You possibly can set up rerun in any of your python initiatives utilizing pip or uv:

pip set up rerun-sdk
uv add rerun-sdk

You can begin the viewer after putting in the sdk by merely operating it from the command line:

rerun

The viewer can be your essential window the place your experiments can be proven. You possibly can depart it open or shut it between your experiments.

To instantiate a rerun viewer from a python script, you must spawn an occasion with an experiment title:

import rerun as rr

rr.init("ball_tracking", spawn=True)

Ball Monitoring Demo

Rerun experiment recordings may be saved to and loaded from .rrd recording recordsdata. You possibly can obtain the recording file for the ball monitoring demo from here. Press Ctrl + O or choose Open... within the menu on the highest left of the rerun viewer and cargo the downloaded recording file.

You will notice the ball monitoring demo playback as soon as after which the video stops. On the backside of the viewer, you have got the timeline. You possibly can scrub by the timeline by clicking and dragging the deal with.

These recording recordsdata solely comprise the information of the experiment together with the video, its annotations and the time collection of the monitoring. The structure of the viewer is saved individually in a .rbl blueprint file. Obtain the blueprint file for the demo here and open it on prime of the prevailing information file.

Now we’ve a barely higher overview with the place, velocity and acceleration plots separated and the video prominently centered.

Within the video body you may click on on the annotations and within the left Blueprint panel you may cover or present them individually.

Time Collection Plots

To research a particular plot, you may click on on the develop view button on the prime proper of any window, for instance the place plot.

It is a TimeSeriesView. This view can be utilized to plot information in a 2D chart with the x-axis representing the time area, in our case the discrete body index of the video. On this ball monitoring demo, we iterate over the video frames in a loop, the place we are able to set the body index of our timeline explicitly.

frame_index = 0

whereas True:
    ret, body = cap.learn()
    if not ret:
        break

    frame_index += 1
    if frame_index >= num_frames:
        break

    rr.set_time("frame_idx", sequence=frame_index)

To create the plot for the place, you must log a Scalar worth for the place of the tracked ball at each body index. On this case after we calculate the place we are able to merely log it to rerun:

rr.log("ball/position_y", rr.Scalars(pos))

To configure the type of this plot, we additionally have to log one thing to the identical entity path (ball/position_y), however for the reason that type doesn’t change we are able to log it as soon as earlier than the loop and provide a static argument.

rr.log(
    "ball/position_y",
    rr.SeriesLines(colours=[0, 128, 255], names="pos y"),
    static=True,
)

To outline the axis vary that’s seen per default, we have to specify a structure element.

view_pos = rrb.TimeSeriesView(
    origin="ball/position_y",
    axis_y=rrb.ScalarAxis(vary=(0, 700)),
)
structure = rrb.Blueprint(view_pos)
rr.send_blueprint(structure)

Video Stream

Equally we are able to create a view for the video frames by logging the picture to rerun. Since rerun expects an RGB photographs however OpenCV ues BGR for its coloration channel ordering, we have to convert the frames from BGR to RGB earlier than passing them to rerun.

frame_rgb: np.ndarray = cv2.cvtColor(body, cv2.COLOR_BGR2RGB)
rr.log("body", rr.Picture(frame_rgb))

So as to add annotations to the picture view we have to log spatial components to a sub path of the required entity path. For instance, we are able to draw the middle of the tracked ball:

rr.log(
    "body/factors",
    rr.Points2D(
        positions=[center],
        colours=[0, 0, 255],
        radii=4.0,
    ),
)

To incorporate the video body view within the structure, we are able to use a Spatial2DView node:

view_frame = rrb.Spatial2DView(origin="body")

Then we are able to stack the plot from earlier than vertically with the body view by utilizing a Vertical structure element:

structure = rrb.Blueprint(
    rrb.Vertical(
        view_pos,
        view_frame,
        row_shares=[0.33],
    ),
)
rr.send_blueprint(structure)

The row shares specifies how a lot every of the rows occupies in percentages. We are able to omit the second row share entry for the body view for the reason that shares have so as to add as much as 1.

Limitations

Whereas engaged on this mission, I bumped into some limitations of Rerun. Within the unique mission I visualized a prediction of the trajectory at each timestep, however that is at the moment not potential in a time collection view. Additionally the layouts and configuration of the plotted information is proscribed, for instance there’s no built-in means to attract a circle with out fill. However for the reason that mission could be very actively being developed, there’s a very good probability that a few of these could be potential sooner or later.

Conclusion

The developer expertise with this device as a pc imaginative and prescient engineer is extraordinarily good, the consumer interface masses virtually immediately and the timeline scrubbing may be extremely useful for understanding or debugging alerts in time collection plots or in movies. I’ll undoubtedly preserve utilizing and exploring this mission and may solely suggest you to strive it out for your self!

For extra particulars and the complete implementation, checkout the supply code of this mission beneath src/ball_tracking/trajectory_rerun.py.

https://github.com/trflorian/ball-tracking-live-plot

All visualizations on this submit had been created by the writer.