Remove SimMode and support different vehicle types

[alonfaraj]

Fixes: #506
Fixes: #1382
Fixes: #1777
Fixes: #2990
Fixes: #4084
Fixes: #4110

About

As a follow up to the great work of @rajat2004 (#4110), this PR is intended support different vehicle types in AirSim and remove SimMode from settings.json file.

There are probably few approaches to achieve this goal.
I chose the one which IMO should be the easiest to implement and maintain.

We have already started working with this version for our custom needs so I thought it would be worth sharing with the community.
This PR should be tested much more, but it already work with different vehicle types - in Unreal and ROS2 (ROS1 modified as well but haven't tested yet).
Any feedbacks/thoughts/issues are welcome.

Changes:

• SimMode is no longer required and ignored
• SensorsDefault changed to SensorsDefaultCar and SensorsDefaultMultirotor
• Record file was split into 3 files - airsim_rec_cv.txt, airsim_rec_car.txt, airsim_rec_multirotor.txt and saved into the same folder
• ApiServerPort and its default value changed to ApiServerPortCV (41451), ApiServerPortCar(41461), ApiServerPortMultirotor(41471)

How Has This Been Tested?

Tested with Unreal, python API and ROS2, with various combinations and features.
Attached an example script and settings to initialize with 3 different types, send commands and spawn another 3 vehicles.

example settings.json

{
	"SettingsVersion": 1.2,

	"Vehicles": {
		"test1_car": {
			"VehicleType": "PhysXCar",
			"X": 0, "Y": 7, "Z": 0		
		},
		"test2_cv": {
			"VehicleType": "ComputerVision",
			"X": 0, "Y": 3, "Z": 0		
		},
		"test3_drone": {
			"VehicleType": "SimpleFlight"
		}
	}

  }

example script

import setup_path 
import airsim
import cv2

cv = airsim.VehicleClient()
cv.confirmConnection()

drone = airsim.MultirotorClient()
drone.confirmConnection()
drone.enableApiControl(True)

car = airsim.CarClient()
car.confirmConnection()
car.enableApiControl(True)


drone.takeoffAsync()

car_controls = airsim.CarControls()
car_controls.steering = 0.0
car_controls.throttle = 1.0
car_controls.brake = 0.0
car.setCarControls(car_controls)

pose = airsim.Pose(airsim.Vector3r(0, -2, 0),
                       airsim.Quaternionr(0, 0, 0, 0))
drone.simAddVehicle("drone2", "SimpleFlight", pose)

pose = airsim.Pose(airsim.Vector3r(2, 0, 0),
                       airsim.Quaternionr(0, 0, 0, 0))
car.simAddVehicle("car2", "PhysXCar", pose)

pose = airsim.Pose(airsim.Vector3r(0, 5, 0),
                       airsim.Quaternionr(0, 0, 0, 0))
cv.simAddVehicle("cv2", "ComputerVision", pose)


camera_name = "0"
image_type = airsim.ImageType.Scene
while True:
    rawImage = cv.simGetImage(camera_name, image_type)
    if not rawImage:
        continue
    png = cv2.imdecode(airsim.string_to_uint8_array(rawImage), cv2.IMREAD_UNCHANGED)
    
    cv2.imshow("AirSim", png)
    if cv2.waitKey(1) & 0xFF == ord('q'):
        break
cv2.destroyAllWindows() 

Screenshots (if appropriate):

clip.mp4

recording files:
airsim_rec_cv.txt
airsim_rec_multirotor.txt
airsim_rec_car.txt

[yvan233]

Thanks for your work. This is the functionality I need now. I compiled and ran the latest branch locally and reproduced example script.py without any bugs. But when I test the drone's moveToPositionAsync function like below. The drone will be abnormal and cannot fly to the designated position.

import setup_path 
import airsim
import os

client = airsim.MultirotorClient()
client.confirmConnection()
client.enableApiControl(True)
client.armDisarm(True)

client.takeoffAsync()  
client.moveToPositionAsync(0, 0, -4, 1).join()
client.moveToPositionAsync(5, 5, -4, 1).join()

Do you know what is causing this abnormality? Thanks!

I have found the problem. The clock type needs to be set in settings.json. like this:

"ClockType": "ScalableClock"

[dzywater]

Is it possible to let each client have respective api control status. Currently, one client disable the api control, all clients cannot control. So hope to let server give each client a control status record.

[A-Dings]

I cloned your remove-simmode branch , ran the example script with the example setting.json you provided on Ubuntu 18.04, UE 4.25. I got this error:
Traceback (most recent call last):
File "hello.py", line 10, in
drone.enableApiControl(True)
msgpackrpc.error.RPCError: rpclib: function 'enableApiControl' (called with 2 arg(s)) threw an exception. The exception contained this information: Vehicle API for '' is not available. This could either because this is simulation-only API or this vehicle does not exist.
Could you please help look at this issue?

[sakurayiannie]

I cloned your remove-simmode branch , ran the example script with the example setting.json you provided on Ubuntu 18.04, UE 4.25. I got this error: Traceback (most recent call last): File "hello.py", line 10, in drone.enableApiControl(True) msgpackrpc.error.RPCError: rpclib: function 'enableApiControl' (called with 2 arg(s)) threw an exception. The exception contained this information: Vehicle API for '' is not available. This could either because this is simulation-only API or this vehicle does not exist. Could you please help look at this issue?

Hi！I also encountered this problem, is there any solution?

[alonfaraj]

Hi @sakurayiannie @A-Dings,
I haven't tested it on ubuntu but I don't see any reason it will cause the error.
Did you run the script clean_rebuild?
There are modifications in AirLib, not only the unreal side

[sakurayiannie]

你好@sakurayiannie @A-Dings， 我没有在 ubuntu 上测试过，但我看不出有什么原因会导致错误。 你运行脚本了clean_rebuild吗？ AirLib有修改，不只是虚幻的一面

Thank you for your reply! I can run your remove-simmode project by c++ API (on Win10), while python client returned the same problem ('enableApiControl') as A-Dings encountered. I ran the script clean_rebuild.bat, it returned that "The system cannot find the specified file".

[hjp10010]

Can I only use his@alonfaraj modified old version of airsim to simulate different types of vehicles and UAVs at the same time? It seems that the new version of airsim does not have this function. In fact, I also encountered some problems with his modified old version of airsim.

[alonfaraj]

Hi everyone @hjp10010 @sakurayiannie,
I haven't use this branch heavily back then so there are probably some issues.
It's not fully tested and airsim is no longer maintained so I didn't work on this branch any further.
It also use pretty old ue4 version, maybe 4.25 so I can't test it at the moment.
It worked for me (and for others) with basics features, recording and api, as you can see in the video.

Sorry I can't help you more.

I think colosseum is the right repo to continue this feature development in case the maintainers interested.
I would be glad to help.

[xxEoD2242]

Just leaving the link here for Colosseum:

Colosseum

[hjp10010]

Thank you for your reply. I would like to try the scheme you recommended

[sakurayiannie]

Hi everyone @hjp10010 @sakurayiannie, I haven't use this branch heavily back then so there are probably some issues. It's not fully tested and airsim is no longer maintained so I didn't work on this branch any further. It also use pretty old ue4 version, maybe 4.25 so I can't test it at the moment. It worked for me (and for others) with basics features, recording and api, as you can see in the video.

Sorry I can't help you more.

I think colosseum is the right repo to continue this feature development in case the maintainers interested. I would be glad to help.

Thank you for your contribution to AirSim, my project need to work on air-ground coordination robots, so your work really helped me a lot. I appreciate that. :)

[fellowchrononaut]

I am using the remove-simmode repository and I am facing the following issue:
Whenever I launch the airsim_node.launch file, Unreal crashes with an access violation error in line 533 of SimModeBase.cpp where I think the getImageCapture () is returning a null pointer. I've added an image of the exception below.

And here is the call stack:

I am running the remove-simmode fork on Unreal 4.27.2 with the airsim_ros_pkgs (ROS 1) running on Ubuntu 20.04 on WSL2.

I have tested with the main Airsim repository, also with the Colosseum repository(with UE 5.1). All of them work fine with the ros package. I can get the airsim_node to work and I can get images. But this issue is seen only with the remove-simmode fork. The other topics from the airsim_node are received well if I isolate the images by setting the "update_airsim_img_response_every_n_sec" parameter in the airsim_node.launch file to a high value. So I assume it's an issue with just the image requests.

Can someone please help me debug this issue because I want this feature (Drone and Car in the same environment) for my project?

Thanks in advance

[dzywater]

Fixes: #506 Fixes: #1382 Fixes: #1777 Fixes: #2990 Fixes: #4084 Fixes: #4110

About

As a follow up to the great work of @rajat2004 (#4110), this PR is intended support different vehicle types in AirSim and remove SimMode from settings.json file.

There are probably few approaches to achieve this goal. I chose the one which IMO should be the easiest to implement and maintain.

We have already started working with this version for our custom needs so I thought it would be worth sharing with the community. This PR should be tested much more, but it already work with different vehicle types - in Unreal and ROS2 (ROS1 modified as well but haven't tested yet). Any feedbacks/thoughts/issues are welcome.

Changes:

• SimMode is no longer required and ignored
• SensorsDefault changed to SensorsDefaultCar and SensorsDefaultMultirotor
• Record file was split into 3 files - airsim_rec_cv.txt, airsim_rec_car.txt, airsim_rec_multirotor.txt and saved into the same folder
• ApiServerPort and its default value changed to ApiServerPortCV (41451), ApiServerPortCar(41461), ApiServerPortMultirotor(41471)

How Has This Been Tested?

Tested with Unreal, python API and ROS2, with various combinations and features. Attached an example script and settings to initialize with 3 different types, send commands and spawn another 3 vehicles.

example settings.json
example script

Screenshots (if appropriate):

clip.mp4

recording files: airsim_rec_cv.txt airsim_rec_multirotor.txt airsim_rec_car.txt

Hi, @alonfaraj , airsim seems active again, we can continue this topic.

[alonfaraj]

@dzywater Do you mean colosseum?
It's not so active as far as I can tell.
This feature requires some more work and tests which I don't think we can get.

[xxEoD2242]

@dzywater Do you mean colosseum? It's not so active as far as I can tell. This feature requires some more work and tests which I don't think we can get.

@alonfaraj Is it possible to port some of this work to Colosseum? I have another approach I've been working on but it's a behemoth of a PR that I'm attempting to break down.

@dzywater AirSim has been abandoned by Microsoft. Colosseum is our effort to continue the open source project with UE5

[alonfaraj]

@xxEoD2242 sure.
Feel free

[dzywater]

behemoth

Could you import this pr to colosseum? Afterwards, we maintain colosseum together.

[Willis814]

I encountered some problems. When I opened the block environment after running build.cmd, an error occurred:
Severity Code Description Item File Line Suppressed Status Details
Error Unable to instantiate module 'AirSim': System.IO.DirectoryNotFoundException: Part of the path 'D:\Unreal Projects\Test\Plugins\AirSim\Source\AirLib\deps\MotionCore' could not be found. Test D:\Unreal Projects\Test\Intermediate\ProjectFiles\UnrealBuildTool 1
What should I do to solve this problem? Thanks for the answer!

[JeroZe]

这是来自QQ邮箱的假期自动回复邮件。   您好，我最近正在休假中，无法亲自回复您的邮件。我将在假期结束后，尽快给您回复。

[Willis814]

我正在使用remove-simmode存储库，并且面临以下问题： 每当我启动airsim_node.launch文件时，Unreal都会崩溃，并在SimModeBase.cpp的第533行中出现访问冲突错误，我认为getImageCapture（）返回null指针。我在下面添加了异常的图像。

这是调用堆栈：

我正在 Unreal 4.27.2 上运行remove-simmode fork，并在 WSL2 上的 Ubuntu 20.04 上运行airsim_ros_pkgs (ROS 1)。

我已经使用主 Airsim 存储库以及 Colosseum 存储库（使用 UE 5.1）进行了测试。它们都与 ros 包一起工作得很好。我可以让airsim_node 工作并且可以获取图像。但只有在使用remove-simmode 分支时才会出现此问题。如果我通过将airsim_node.launch 文件中的“update_airsim_img_response_every_n_sec”参数设置为较高值来隔离图像，则来自airsim_node 的其他主题会得到很好的接收。所以我认为这只是图像请求的问题。

有人可以帮我调试这个问题吗，因为我想要为我的项目提供此功能（无人机和汽车在同一环境中）？

提前致谢

Have you solved this problem? I want to know what causes the error?🥺

[JeroZe]

这是来自QQ邮箱的假期自动回复邮件。   您好，我最近正在休假中，无法亲自回复您的邮件。我将在假期结束后，尽快给您回复。

[ahworld22]

我正在使用remove-simmode存储库，并且面临以下问题： 每当我启动airsim_node.launch文件时，Unreal都会崩溃，并在SimModeBase.cpp的第533行中出现访问冲突错误，我认为getImageCapture（）返回null指针。我在下面添加了异常的图像。
这是调用堆栈：
我正在 Unreal 4.27.2 上运行remove-simmode fork，并在 WSL2 上的 Ubuntu 20.04 上运行airsim_ros_pkgs （ROS 1）。
我已经使用主 Airsim 存储库以及 Colosseum 存储库（使用 UE 5.1）进行了测试。它们都与 ros 包一起工作得很好。我可以让airsim_node 工作并且可以获取图像。但只有在使用remove-simmode 分支时才会出现此问题。如果我通过将airsim_node.launch 文件中的“update_airsim_img_response_every_n_sec”参数设置为较高值来隔离图像，则来自airsim_node 的其他主题会得到很好的接收。所以我认为这只是图像请求的问题。
有人可以帮我调试这个问题吗，因为我想要为我的项目提供此功能（无人机和汽车在同一环境中）？
提前致谢

你解决了这个问题吗？我想知道是什么原因导致了错误？ 🥺

我也遇到了这个问题，请问你解决了吗？

[JeroZe]

这是来自QQ邮箱的假期自动回复邮件。   您好，我最近正在休假中，无法亲自回复您的邮件。我将在假期结束后，尽快给您回复。

[TaoYuanZhongRen]

I encountered some problems. When I opened the block environment after running build.cmd, an error occurred: Severity Code Description Item File Line Suppressed Status Details Error Unable to instantiate module 'AirSim': System.IO.DirectoryNotFoundException: Part of the path 'D:\Unreal Projects\Test\Plugins\AirSim\Source\AirLib\deps\MotionCore' could not be found. Test D:\Unreal Projects\Test\Intermediate\ProjectFiles\UnrealBuildTool 1 What should I do to solve this problem? Thanks for the answer!

hello，I met the same thing, Have you solved this problem？

[JeroZe]

这是来自QQ邮箱的假期自动回复邮件。   您好，我最近正在休假中，无法亲自回复您的邮件。我将在假期结束后，尽快给您回复。