Get actors information from Unreal Engine 4.21

[aprilaugust]

Hello,

Many thanks for such great Airsim!

I want to use UAV to collect data of pedestrians, for example: how many pedestrians in a scene? or segmentation of pedestrians. I successfully added 3D pedestrians into LanscapeMap in Unreal Engine (UE) 4.21, but I can't get any information about pedestrians except scene/image itself.

I tried simSetSegmentationID to set new IDs for pedestrians & these new IDs are sucessfully updated in UE 4.21, but when I use simGetSegmentationID on them, it returns -1 in Airsim

I also tried simListSceneObjects & get this error

My settings:
"SimMode": "Multirotor"
My pedestrians type in UE 4: SkeletalMeshActor

I highly appreciate any help or suggestions!

Many thanks.
ps: I'm new to both Airsim & UE 4

[msb336]

the simListSceneObjects error you have looks like it is because you are not importing the most recent python module from git, but instead using your global python from pip install airsim. Try adding the path to your local airsim module before importing airsim:

import sys
sys.path.insert(0, "{Your path to airsim}/Airsim/PythonClient")
import airsim

As for you simGetSegmentationID problems, can you please copy and paste the relevant code from your script?

[aprilaugust]

Thank you so much for your prompt support.

The simListSceneObjects problem is solved perfectly.

For simGetSegmentationID problem, following is my script:

import sys
sys.path.insert(0, r"C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\AirSim\PythonClient")
import airsim
import cv2
import numpy as np
import os
import pprint
import setup_path 
import tempfile
client = airsim.MultirotorClient()
client.confirmConnection()

anim = client.simListSceneObjects('1AnimBlueprint[\w]*') # work perfectly
print("\nsim List Scene Object", anim)

client.enableApiControl(True, "Drone1")
client.enableApiControl(True, "Drone2")
client.armDisarm(True, "Drone1")
client.armDisarm(True, "Drone2")

found = client.simSetSegmentationObjectID("1AnimBlueprint[\w]*", 101, True) # return **True**
print("Done: %r" % (found)) 

idfound2 = client.simGetSegmentationObjectID("1AnimBlueprint[\w]*") # return **-1**
print("id found2: %r" %(idfound2))

After I use simSetSegmentationObjectID to set my objects to 101, the object IDs will be 101 in UE as image below

but when I use simGetSegmentationObjectID on the same objects above, it returns -1 in Airsim as below

Actually, my final goal is that I want to know how many pedestrians in each image captured by UAV. I work around with segmentation since I think it will lead me to my goal. But if you know the faster & more direct solution, please suggest to me.

Thanks a lot in advance.

[msb336]

@aprilaugust this is an interesting problem. After trying it myself, it appears that calling simSetSegmentationObjectID on an object does change the segmentation color correctly in the segmentation image (press 2 on your keyboard to view the segmentation image or call simGetImages to view the segmentation image), but it does not actually change the return value of simGetSegmentationObjectID. This will require some looking into. For now however, using the image return from simGetImages should suit your purposes, just make sure to track the segmentation ID you set your objects to in your client script.

@robertLiuLinFeng I have hidden your post in this thread because it is an unrelated issue. please create a new issue for your problem.

[aprilaugust]

Thank you very much for your kind reply, @msb336.

I checked simSetSegmentationObjectID & it does change segmentation color of SM_PineTree object, but not working on my 3D pedestrians/human due to not being segmented. For more detail, my 3D human type is SkeletalMeshActor & SM_PineTree type Is StaticMeshActor.

So currently, I still encounter two problems:

1. My 3D pedestrians/human are not being segmented (please see Fig 1)

2. I want to know how many 3D pedestrians/human (in the red box in Fig 1) in each & every image captured by UAV. Could you please kindly suggest me some helpful functions?

Fig 1. Scene Image & Segmentation Image by UAV

Fig 2. Here is a scene with my 3D human & SM_PineTree

Here is my full script:

import sys
sys.path.insert(0, r"C:\Program Files (x86)\Microsoft Visual Studio\2017\Community\AirSim\PythonClient")
import airsim
import cv2
import numpy as np
import os
import pprint
import setup_path 
import tempfile

client = airsim.MultirotorClient()
client.confirmConnection()

anim = client.simListSceneObjects('1AnimBlueprint[\w]*')
print("\nsim List Scene Object", anim)

client.enableApiControl(True, "Drone1")
client.enableApiControl(True, "Drone2")
client.armDisarm(True, "Drone1")
client.armDisarm(True, "Drone2")

found = client.simSetSegmentationObjectID("1AnimBlueprint[\w]*", 101, True)
print("Done: %r" % (found))

idfound2 = client.simGetSegmentationObjectID("1AnimBlueprint[\w]*")
print("id found2: %r" %(idfound2))

airsim.wait_key('Press any key to takeoff')
f1 = client.takeoffAsync(vehicle_name="Drone1")
f2 = client.takeoffAsync(vehicle_name="Drone2")
f1.join()
f2.join()

state1 = client.getMultirotorState(vehicle_name="Drone1")
s = pprint.pformat(state1)
print("state: %s" % s)
state2 = client.getMultirotorState(vehicle_name="Drone2")
s = pprint.pformat(state2)
print("state: %s" % s)

airsim.wait_key('Press any key to take images')
responses1 = client.simGetImages([
       airsim.ImageRequest("3", airsim.ImageType.Segmentation, False, False)], vehicle_name="Drone1")  
print('Drone1: Retrieved images: %d' % len(responses1))

responses2 = client.simGetImages([
    airsim.ImageRequest("3", airsim.ImageType.Scene),  
    airsim.ImageRequest("3", airsim.ImageType.Segmentation, False, False)], vehicle_name="Drone2")  
print('Drone2: Retrieved images: %d' % len(responses2))

img1d = np.fromstring(responses1[0].image_data_uint8, dtype=np.uint8) #get numpy array
img_rgb = img1d.reshape(responses1[0].height, responses1[0].width, 3) #reshape array to 3 channel image array H X W X 3
img_rgb = np.flipud(img_rgb) #original image is fliped vertically

tmp_dir = os.path.join(tempfile.gettempdir(), "airsim_drone")
print ("Saving images to %s" % tmp_dir)
try:
    os.makedirs(tmp_dir)
except OSError:
    if not os.path.isdir(tmp_dir):
        raise

for idx, response in enumerate(responses1 + responses2):

    filename = os.path.join(tmp_dir, str(idx))

    if response.pixels_as_float:
        print("Type %d, size %d" % (response.image_type, len(response.image_data_float)))
        airsim.write_pfm(os.path.normpath(filename + '.pfm'), airsim.get_pfm_array(response))
    elif response.compress: #png format
        print("Type %d, size %d" % (response.image_type, len(response.image_data_uint8)))
        airsim.write_file(os.path.normpath(filename + '.png'), response.image_data_uint8)
    else: #uncompressed array
        print("Type %d, size %d" % (response.image_type, len(response.image_data_uint8)))
        img1d = np.fromstring(response.image_data_uint8, dtype=np.uint8) #get numpy array
        img_rgb = img1d.reshape(response.height, response.width, 3) #reshape array to 3 channel image array H X W X 3
        cv2.imwrite(os.path.normpath(filename + '.png'), img_rgb) # write to png

airsim.wait_key('Press any key to reset to original state')

client.armDisarm(False, "Drone1")
client.armDisarm(False, "Drone2")
client.reset()

client.enableApiControl(False, "Drone1")
client.enableApiControl(False, "Drone2")

Thank you very much for your time.

[msb336]

Hmm, it looks like SkeletalMeshActor types are not getting searched for in the segmentation script. I'll look into what it would take to mend that problem.

[aprilaugust]

Thank you a lot in advance.

[PatrickUtz]

I am getting the same issue yet with another Drone (PlayerStart) object I instantiated. I posted the following issue: #2077

[madratman]

fixed by #2855