Skip to content

simGetCameraInfo.proj_mat return an unexpected form of matrix #5044

Open

Description

Bug Report

  • AirSim Version/#commit: 1.8.1
  • UE/Unity version: 4.27.2
  • Autopilot version: N/A
  • OS Version: Windows 11

What's the issue you encountered?

The projection matrix (proj_mat) returned by the simGetCameraInfo API does not match the expected structure of a standard perspective projection matrix.

The returned projection matrix is as follows:

<ProjectionMatrix> {   
  'matrix': [   
    [0.0, 1.0016834735870361, 0.0, 0.0],
    [0.0, 0.0, -1.7807706594467163, 0.0],
    [0.0, 0.0, 0.0, 10.0],
    [-1.0, 0.0, 0.0, 0.0]
  ]
}

This matrix does not conform to the standard pinhole camera projection matrix structure (which typically includes parameters such as focal lengths fx/fy and principal points cx/cy), making it unusable for common computer vision tasks such as 3D-to-2D coordinate transformation.

Settings

Custom settings.json configuration is used as follows:

{
    "SettingsVersion": 1.2,
    "SimMode": "",
    "CameraDefaults": {
        "CaptureSettings": [
            {
                "ImageType": 0,
                "Width": 1920,
                "Height": 1080
            },
            {
                "ImageType": 1,
                "Width": 1920,
                "Height": 1080
            },
            {
                "ImageType": 3,
                "Width": 1920,
                "Height": 1080
            },
            {
                "ImageType": 5,
                "Width": 1920,
                "Height": 1080
            }
        ]
    },
    "ClockSpeed": 1.0,
    "ViewMode": "",
    "Vehicles": {
        "test1_car": {
            "VehicleType": "PhysXCar",
            "X": 0,
            "Y": 0,
            "Z": 0,
            "ApiServerPort": 41461,
            "Cameras": {
                "front_center": {
                    "X": 0.0, "Y": 0.0, "Z": 0.0,
                    "Pitch": 0, "Roll": 0, "Yaw":0
                }
            }
        },
        "test2_drone": {
            "VehicleType": "SimpleFlight",
            "X": -5,
            "Y": 0,
            "Z": 0,
            "ApiServerPort": 41471,
            "Cameras": {
                "bottom_center": {
                    "X": 0.0, "Y": 0.0, "Z": 0.0,
                    "Pitch": -90, "Roll": 0, "Yaw": 0
                }
            }
        }
    },
    "WindowWidth": 1920,
    "WindowHeight": 1080,
    "ClockType": "ScalableClock"
}

How can the issue be reproduced?

  1. Launch the AirSim simulation environment based on UE 4.27.2 on Windows 11 with the above settings.json configuration.
  2. Run the following Python code to connect to the vehicle and retrieve camera information:
    import airsim
    car_client = airsim.CarClient(port=41461)
    car_client.confirmConnection()
    camera_info = car_client.simGetCameraInfo("0", vehicle_name="test1_car")
    print(camera_info.proj_mat)
  3. Observe the structure of the returned proj_mat.matrix in the output.

Include full error message in text form

No explicit error message is displayed, but the returned projection matrix has an abnormal structure. The specific content is as follows:

ProjectionMatrix(matrix=[[0.0, 1.0016834735870361, 0.0, 0.0], [0.0, 0.0, -1.7807706594467163, 0.0], [0.0, 0.0, 0.0, 10.0], [-1.0, 0.0, 0.0, 0.0]])

Activity

  1. zhicongsun commented on Aug 16, 2025

    @zhicongsun

    omg I meet the same issue. any update?

  2. Joyymoonh commented on Aug 30, 2025

    @Joyymoonh
    Author

    天哪,我遇到了同样的问题。有什么更新吗?

    I didn't solve it, pity

  3. deansereigns commented on Jan 28, 2026

    @deansereigns

    Camera Model and Projection in AirSim

    AirSim uses a pinhole camera model, implemented through Unreal Engine (UE4), to render images.

    Key Points

    • The camera model is a standard pinhole camera.
    • Camera intrinsics are not directly exposed by the API.
    • Intrinsic parameters must be computed from:
      • image resolution
      • camera horizontal field of view (FOV)

    About CameraInfo.proj_mat

    • CameraInfo.proj_mat is not a pinhole projection matrix of the form:

      P = K [R | t]

    • It represents Unreal Engine’s clip-space projection matrix, intended for rendering.

    • It includes internal engine transformations (coordinate conversions, perspective divide, near/far planes).

    • Because of this, intrinsic parameters cannot be extracted or decomposed from proj_mat.

    Correct Workflow

    • Treat the AirSim camera as a pinhole camera

    • Compute the intrinsic matrix K analytically from FOV and image resolution

    • Compute extrinsics R, t from CameraInfo.pose

    • Construct the full projection explicitly as:

      P = K [R | t]

    Important Note

    The camera pose is different from the vehicle/drone pose.
    Camera offsets and rotations relative to the vehicle must be taken into account when computing transformations.

    Reference

  4. somveeryada289-lgtm commented on Jun 28, 2026

    @somveeryada289-lgtm

    Somveer yadav

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions