New Gimbal Interface #3386

robertlong13 · 2024-08-04T14:56:23Z

Still a bit to do, but I have some stuff working. It's going to be a massive PR, so I want to get some of this out there to start review early.

I have a test setup using RealFlight that I will need to document so anyone can play around with this.
I have a lua script I need to publish that sends VIDEO_STREAM_INFORMATION messages so Planner can detect the URIs for the video.
ArduPilot's CAMERA_FOV_STATUS message has a bug that causes the wrong units to be used for alt. Needs fixing.
Most of the buttons and right-click options do not work yet
Need a settings page so people can adjust the stuff like keybindings
Things you can do
- WASD to slew. Hold shift to go faster, ctrl to go slower.
- Hold E to zoom in, hold Q to zoom out.
- Clicking sets the POI to the lat/lon of the point you clicked, though more options are intended. (THIS ONLY WORKS IF YOU HAVE PATCHED CAMERA_FOV_STATUS to send the correct units)
- Alt-F to take a picture
- Alt-R to toggle recording
- L to toggle between lock and follow
- N to go neutral
- H to point at home

(Terrible gif quality, had to really squeeze it down to fit in a github post, so there are a ton of artifacts; these obviously aren't in the real thing)

rmackay9 · 2024-08-05T06:59:43Z

This is really great to see! Thanks for also finding the issue with FOV_STATUS. We added support ages ago but no GCS has used it yet so I guess we missed the bug.

Davidsastresas · 2024-08-07T16:56:16Z

Very neat! :)

IuliuNovac · 2024-08-17T11:34:38Z

How did you manage to get the simulation with a video feed up and running, is that gazebo?

robertlong13 · 2024-08-18T11:18:29Z

How did you manage to get the simulation with a video feed up and running, is that gazebo?

I use RealFlight. I'm sure the same thing could be done in Gazebo, but I don't know my way around it nearly as well. You need this model and this lua script. You also need to be running on AP master (until the FOV message bug fix gets backported to 4.5)

I use a gstreamer pipeline to grab a cropped area of a monitor. I toyed with using OBS to stream as RTSP, but that introduced more than a second of latency.

dx9screencapsrc x=10 y=60 width=640 height=360 monitor=1 ! videoconvert ! video/x-raw,format=BGRA ! appsink name=outsink sync=false

Adjust the monitor, x, y, width, and height for your setup.

If you configure your autopilot to provide VIDEO_STREAM_INFORMATION messages, with gst://[pipeline] as the URI, this new UI can grab the pipeline from there automatically (it doesn't auto connect to it yet though; that's a planned feature).

The airport in this video was Pendleton Airport (KPDT), but I don't know if I can share my RealFlight model of that airport publicly. Instead, I'll share Base INFOCA, which has the side benefit of being within the real terrain model RealFlight pulled from Spain, so all the terrain actually lines up: Base INFOCA_AP.zip. The downside is that I went WAY overboard manually adding one metric crapload of trees to it. Disable trees with View > Scenery > Trees to disable them if your computer is choking on them. Launch SITL with -O 37.412620,-3.075734,1052.0,0

robertlong13 · 2024-09-29T06:51:22Z

This PR is getting close to ready. All buttons and (almost) all keybindings are implemented, and I have a nice settings page where you can customize every keybinding.

Still needed:

Click to track (need a testing setup for this one)
Auto connect to first video stream if found
Consider making a way to embed this on the flightdata page with minimap, and have the ability to swap between map with mini video and video with mini map.

meee1 · 2024-10-01T11:01:41Z

need to tweak for android/apple - currently failing. system.drawing cant be used easerly....

meee1 · 2024-10-01T12:22:01Z

Controls/Video.cs

@@ -7,6 +7,8 @@ namespace MissionPlanner.Controls
 {
    public partial class Video : Form
    {
+        private readonly GStreamer stream = new GStreamer();


need to confirm if this can be instance based. i have a feeling it cant. as it will lead to native crashs

I can open multiple GStreamer sources (multiple gimbal windows and a separate stream on the HUD) with no apparent issues if that's what you mean.

robertlong13 · 2024-10-13T08:09:19Z

Tracking added

robertlong13 · 2024-10-21T11:55:42Z

Added new ways to display the new control

Full Sized. In this mode, there is a mini map by default, but it can be hidden with a button in the right-click menu.

Mini

Mini and full-sized can be swapped at any point with the "swap with map" when right clicking on the video.

And pop-out works exactly as before (except now it remains always on top, which I think is always better, you can always intentionally minimize it if it's in the way).

rmackay9 · 2024-11-29T06:19:08Z

I've tested this quite a bit now and it works!

A few issues I found (and have already reported to Rob):

An error warning box some times appears if the camera is unplugged and then plugged back in again
right-mouse-button click on the video screen and select "Close" causes MP to crash

Some of the things I tested include:

confirming the VIDEO_STREAM_INFORMATION message's URL field is being consumed and used as the source
right-mouse-button-menu items are working (Retract, Neutral, Point Down, Yaw Lock, Take Picture, Start Recording, Stop Recording, Settings, Mini map, Swap with map)
Settings screen pops up and listed controls work (e.g. slew left, right, up, down). There were some that I could not test (like Zoom) because my camera doesn't support this

From my point of view, besides the two crash issues listed it looks very usable. I guess one enhancement (also already discussed with Rob) is it would be nice to reduce the number of clicks the user must do before the video appears. In particular the user must currently do this each time MP is started:

Right-mouse-button-click on map and select "Gimbal Video", "Full Sized"
Right-mouse-button-click on Video (which initially shows a big red X) and select "Video Stream"
When the VideoStreamSelector window appears select from the "Detected Streams" drop-down and press "Connect"

robertlong13 · 2024-12-26T10:42:51Z

Implemented autoconnecting to the first reported stream
Fixed up the GStreamer first-time setup. My setup accidentally relied on having put a video on your HUD at some point in time before you used this.
Fixed a bug relating to properly handling an RTSP connection loss.
Fixed a bug with the context menu options for swapping/closing the video control on the map.

rmackay9 · 2025-01-03T02:56:15Z

I've retested this and it's working quite well. Some small issues that I found to do with the short-cut keys

the left click to pan/tilt seems to use the entire screen area instead of the smaller area where the video is shown. Perhaps it's related but I also found that clicking on the top of the video caused only a small change in pitch angle but clicking the bottom of the image caused a much larger change in pitch angle. I would expect that clicking at the very top or very bottom of the image would cause the same pitch change (but in the opposite direction of course)
slew up/down short-cut keys seemed reversed
retract short-cut didn't work
didn't re-request video-stream-info when re-connected (not a blocker)

Everything else seemed to work well. My testing included:

a. tested that all the menu commands worked (e.g. retract, neutral, point down, point home, take picture, start/stop recording, Mini map, swap with map, pop-out)
b. tested all the short-cut keys (slew left, right, up down, zoom in/out, slew fast/slow, take picture, start/stop/toggle recording, lock/follow/toggle, neutral, point-down, home)

I think we should rebase this on master and mark as "Ready for review"

Previously, this function did nothing. It was also trying to do the opposite of what it claimed to do.

This library was originally copy-pasted from ArduPilot and ported to C#, but many mistakes were made in the process. For example, many of the methods in the C++ implementation take a reference variable as input, and the C# code was taking a regular input and outputting nothing. This commit overhauls the whole library, making it more C#-like and documents all methods.

Probably none of these make sense to combine, but the else-ifs made an unwritten priority order that was not clear. This behavior is more expected, and we clearly warn the user now on the settings page about clashes.

rmackay9 · 2025-02-03T03:27:49Z

Thank you so much! This is a huge feature to get in!

rmackay9 requested a review from meee1 August 5, 2024 06:59

robertlong13 mentioned this pull request Aug 5, 2024

AP_Scripting: add SITL RealFlight gimbal driver ArduPilot/ardupilot#27752

Open

robertlong13 force-pushed the pr/gimbal_interface branch 3 times, most recently from 9fb28b3 to 56a2a6b Compare September 15, 2024 03:40

robertlong13 force-pushed the pr/gimbal_interface branch 3 times, most recently from eb3d908 to 15628d5 Compare September 29, 2024 06:45

meee1 reviewed Oct 1, 2024

View reviewed changes

robertlong13 force-pushed the pr/gimbal_interface branch 2 times, most recently from 6cd30be to 6adf829 Compare October 13, 2024 08:02

robertlong13 mentioned this pull request Oct 16, 2024

AP_Scripting: examples: add fake camera tracking ArduPilot/ardupilot#28418

Open

robertlong13 force-pushed the pr/gimbal_interface branch 2 times, most recently from 7cdfb5e to dbb9168 Compare October 21, 2024 11:51

robertlong13 force-pushed the pr/gimbal_interface branch 2 times, most recently from f133b6f to b7075f7 Compare November 19, 2024 06:06

rmackay9 mentioned this pull request Nov 30, 2024

Scripting promote video-stream-information to an applet ArduPilot/ardupilot#28780

Merged

robertlong13 force-pushed the pr/gimbal_interface branch 2 times, most recently from b91248c to 0f81dac Compare December 26, 2024 10:17

robertlong13 added 26 commits February 3, 2025 12:55

CameraProtocol: ack not needed for most requests

020532d

GimbalVideoControl: implement buttons

687c6f3

GimbalManagerProtocol: periodically discover

48240d6

Quaternion: fix earth_to_body

8132a87

Previously, this function did nothing. It was also trying to do the opposite of what it claimed to do.

CameraProtocol: add CalculateImagePointVector

d716cf8

GimbalManagerProtocol: add Get/Set attitude quat

85337a0

CameraProtocol: add pixel to rotation calculation

b49fec9

GimbalVideoControl: add click-to-pan

fc73b29

Controls: add Key and Click binding buttons

4418ace

Add GimbalControlSettingsForm

cace2de

GimbalControlSettings: implement FOV settings

03d8af3

GimbalControlSettings: implement default yaw lock setting

7f2ee45

CameraProtocol: add image tracking

84f3081

GimbalVideoControl: add image tracking

e5c5f1d

CameraProtocol: add tracking feedback

f9c1c81

GimbalVideoControl: draw tracking feedback

e26b323

FlightData: add GimbalVideoControl to map

9b44846

CameraProtocol: add GStreamerPipeline generator

b8d2d5f

GimbalVideoControl: autoconnect to first reported stream

1325f6a

GimbalVideoControl: initialize GStreamer

ae72578

GimbalVideoControl: fix exceptions on disconnect

65444f9

FlightData: fix gimbal video context menu bugs

e2950f0

GimbalVideoControl: fix bug in getMousePosition

80b8fff

GimbalControlSettingsForm: warn about binding clashes

3098277

GimbalVideoControl: allow multiple funcs per key

4af9383

Probably none of these make sense to combine, but the else-ifs made an unwritten priority order that was not clear. This behavior is more expected, and we clearly warn the user now on the settings page about clashes.

meee1 force-pushed the pr/gimbal_interface branch from e659953 to 4af9383 Compare February 3, 2025 01:55

meee1 merged commit 0292ed0 into ArduPilot:master Feb 3, 2025
6 checks passed

rmackay9 mentioned this pull request Feb 3, 2025

Camera/Gimbal control screen #2906

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

New Gimbal Interface #3386

New Gimbal Interface #3386

robertlong13 commented Aug 4, 2024 •

edited

Loading

rmackay9 commented Aug 5, 2024

Davidsastresas commented Aug 7, 2024

IuliuNovac commented Aug 17, 2024

robertlong13 commented Aug 18, 2024 •

edited

Loading

robertlong13 commented Sep 29, 2024

meee1 commented Oct 1, 2024

meee1 Oct 1, 2024

robertlong13 Oct 2, 2024

robertlong13 commented Oct 13, 2024

robertlong13 commented Oct 21, 2024

rmackay9 commented Nov 29, 2024

robertlong13 commented Dec 26, 2024

rmackay9 commented Jan 3, 2025

rmackay9 commented Feb 3, 2025

New Gimbal Interface #3386

New Gimbal Interface #3386

Conversation

robertlong13 commented Aug 4, 2024 • edited Loading

rmackay9 commented Aug 5, 2024

Davidsastresas commented Aug 7, 2024

IuliuNovac commented Aug 17, 2024

robertlong13 commented Aug 18, 2024 • edited Loading

robertlong13 commented Sep 29, 2024

meee1 commented Oct 1, 2024

meee1 Oct 1, 2024

Choose a reason for hiding this comment

robertlong13 Oct 2, 2024

Choose a reason for hiding this comment

robertlong13 commented Oct 13, 2024

robertlong13 commented Oct 21, 2024

rmackay9 commented Nov 29, 2024

robertlong13 commented Dec 26, 2024

rmackay9 commented Jan 3, 2025

rmackay9 commented Feb 3, 2025

robertlong13 commented Aug 4, 2024 •

edited

Loading

robertlong13 commented Aug 18, 2024 •

edited

Loading