[.NET 6][Uno Platform][Ubuntu 21.10] Using Leap Motion

Intro

In this time, I will try using Leap Motion Controller from the Uno Platform application on Ubuntu.

• Ultraleap for Developers

I use this project to add LeapSDK.

• [C#][Uno Platform] Create desktop applications on Ubuntu 1

Environments

• Ubuntu 21.10
• .NET 6.0.101
• LeapDeveloperKit ver.2.3.1

Using Leap Motion Controller on Ubuntu

To use Leap Motion Controller on Ubuntu I have to install tracking software.

• Download Ultraleap's Hand Tracking Software - Ultraleap for Developers

After unpacking the tgz file, install a deb file.

sudo dpkg --install Leap-2.3.1+31549-x64.deb

And I also install Mesa OpenGL.

sudo apt-get install libgl1-mesa-dri libgl1-mesa-glx

After installing, I start the daemon of Leap Motion.

sudo leapd

I can open a control panel.

LeapControlPanel &

To open the control panel, I have to execute the command twice.
I don't know why :(

• Linux Installation - Ultraleap Tracking Support

Controller was not working

Though I didn't get any errors, my machine hasn't recognized the controller yet.

According to this post, I could know I had to add a config file(leapd.service).

• 忙中偸閑=망중투한: [Note] Leap Motion on Ubuntu 20.04

Now I can use the Leap Motion Controller on the control panel.

Add a LeapSDK library file in .NET 6 projects

In the tgz file, I can also get library files to control the Leap Motion Controller.
So I add a dll file for .NET Framework 4.0.

UnoSample.Skia.Gtk.csproj

<Project Sdk="Microsoft.NET.Sdk">
  <PropertyGroup>
    <OutputType Condition="'$(Configuration)'=='Release'">WinExe</OutputType>
    <OutputType Condition="'$(Configuration)'=='Debug'">Exe</OutputType>
    <TargetFramework>net6.0</TargetFramework>
    <LangVersion>latest</LangVersion>
    <Nullable>enable</Nullable>
    <ImplicitUsings>enable</ImplicitUsings>
  </PropertyGroup>
  <ItemGroup Condition="exists('..\UnoSample.UWP')">
    <EmbeddedResource Include="..\UnoSample.UWP\Package.appxmanifest" LogicalName="Package.appxmanifest" />
    <Content Include="..\UnoSample.UWP\Assets\StoreLogo.png" Link="Assets\StoreLogo.png" />
    <Content Include="Assets\Fonts\uno-fluentui-assets.ttf" />
  </ItemGroup>
  <ItemGroup>
    <UpToDateCheckInput Include="..\UnoSample.Shared\**\*.xaml" />
  </ItemGroup>
  <ItemGroup>
    <Reference Include="Leap">
      <HintPath>References/LeapSdk/LeapCSharp.NET4.0.dll</HintPath>
    </Reference>
  </ItemGroup>
  <ItemGroup>
    <PackageReference Include="Microsoft.Extensions.Logging" Version="5.0.0" />
    <PackageReference Include="Microsoft.Extensions.Logging.Console" Version="5.0.0" />
    <PackageReference Include="Uno.UI.Skia.Gtk" Version="4.0.11" />
    <PackageReference Include="Uno.UI.RemoteControl" Version="4.0.11" Condition="'$(Configuration)'=='Debug'" />
    <PackageReference Include="Uno.UI.Adapter.Microsoft.Extensions.Logging" Version="4.0.11" />
  </ItemGroup>
  <Import Project="..\UnoSample.Shared\UnoSample.Shared.projitems" Label="Shared" />
</Project>

• 【.NET】Add files into projects

libLeapCSharp can't be found

Though I don't get any compile errors and I can get static value like "Gesture.GestureType.TYPE_CIRCLE",
when I add classes for controlling the Leap Motion Controller, I get a runtime exception.

LeapCtrlListener.cs

using Leap;

namespace UnoSample.Skia.Gtk.MotionControllers
{
    public class LeapCtrlListener: Listener
    {
        public LeapCtrlListener()
        {
            Console.WriteLine("LeapCtrlListener");
        }
    }
}

Program.cs

using GLib;
using Uno.UI.Runtime.Skia;
using Leap;
using UnoSample.Skia.Gtk.MotionControllers;

namespace UnoSample.Skia.Gtk
{
    class Program
    {
        static void Main(string[] args)
        {
            // OK. "Gesture: TYPE_CIRCLE"
            Console.WriteLine("Gesture: " + Gesture.GestureType.TYPE_CIRCLE);

            // Error
            var leapCtrlListener = new LeapCtrlListener();

            ExceptionManager.UnhandledException += delegate (UnhandledExceptionArgs expArgs)
            {
                Console.WriteLine("GLIB UNHANDLED EXCEPTION" + expArgs.ExceptionObject.ToString());
                expArgs.ExitApplication = true;
            };

            var host = new GtkHost(() => new App(), args);

            host.Run();
        }
    }
}

Error

Unhandled exception. System.TypeInitializationException: The type initializer for 'Leap.LeapPINVOKE' threw an exception.
 ---> System.TypeInitializationException: The type initializer for 'SWIGExceptionHelper' threw an exception.
 ---> System.DllNotFoundException: Unable to load shared library 'LeapCSharp' or one of its dependencies. In order to help diagnose loading problems, consider setting the LD_DEBUG environment variable: libLeapCSharp: cannot open shared object file: No such file or directory
   at Leap.LeapPINVOKE.SWIGExceptionHelper.SWIGRegisterExceptionCallbacks_Leap(ExceptionDelegate applicationDelegate, ExceptionDelegate arithmeticDelegate, ExceptionDelegate divideByZeroDelegate, ExceptionDelegate indexOutOfRangeDelegate, ExceptionDelegate invalidCastDelegate, ExceptionDelegate invalidOperationDelegate, ExceptionDelegate ioDelegate, ExceptionDelegate nullReferenceDelegate, ExceptionDelegate outOfMemoryDelegate, ExceptionDelegate overflowDelegate, ExceptionDelegate systemExceptionDelegate)
   at Leap.LeapPINVOKE.SWIGExceptionHelper..cctor()
   --- End of inner exception stack trace ---
   at Leap.LeapPINVOKE.SWIGExceptionHelper..ctor()
   at Leap.LeapPINVOKE..cctor()
   --- End of inner exception stack trace ---
   at Leap.LeapPINVOKE.new_Listener()
   at Leap.Listener..ctor()
   at UnoSample.Skia.Gtk.LeapMotions.LeapController..ctor() in /home/example/Documents/workspace/UnoSample/UnoSample.Skia.Gtk/LeapMotions/LeapController.cs:line 7
   at UnoSample.Skia.Gtk.Program.Main(String[] args) in /home/example/Documents/workspace/UnoSample/UnoSample.Skia.Gtk/Program.cs:line 13

Add link to shared object file

To use the library file, I have to add link to "libLeapCSharp.so" on my machine.

I can execute this command before run the project.

export LD_LIBRARY_PATH="$Home/home/example/Documents/leap_developer_kit"

(In "leap_developer_kit" folder, there are "libLeapCSharp.so", "libLeap.so", and so on)

But because I don't want to do this everytime.

So I add "leap_developer_kit.conf" into "/etc/ld.so.conf.d".

leap_developer_kit.conf

/home/example/Documents/leap_developer_kit

After I execute "sudo ldconfig", I can watch the link of the files by "ldconfig -p".

...
  libLeapCSharp.so (libc6,x86-64) => /home/example/Documents/leap_developer_kit/libLeapCSharp.so
    libLeap.so (libc6,x86-64) => /home/example/Documents/leap_developer_kit/libLeap.so
...

• EnvironmentVariables - Community Help Wiki
• (6) ldconfigでライブラリパスを更新 - Linuxやってみる！

(Sample) Getting hand count

According to the sample code in the tgz file, I add classes and get tracking hand count.

Program.cs

using GLib;
using Uno.UI.Runtime.Skia;
using UnoSample.Skia.Gtk.MotionControllers;

namespace UnoSample.Skia.Gtk
{
    class Program
    {
        static void Main(string[] args)
        {
            using(var motionController = new MotionController())
            {
                ExceptionManager.UnhandledException += delegate (UnhandledExceptionArgs expArgs)
                {
                    Console.WriteLine("GLIB UNHANDLED EXCEPTION" + expArgs.ExceptionObject.ToString());
                    expArgs.ExitApplication = true;
                };
                var host = new GtkHost(() => new App(), args);
                host.Run();
            }
        }
    }
}

MotionController.cs

using Leap;

namespace UnoSample.Skia.Gtk.MotionControllers;
public class MotionController: IDisposable
{
    private LeapCtrlListener leapCtrlListener;
    private Controller controller;
    public MotionController()
    {
        this.leapCtrlListener = new LeapCtrlListener();
        this.controller = new Controller ();
        this.controller.AddListener(this.leapCtrlListener);
    }
    public void Dispose()
    {
        this.controller.RemoveListener(this.leapCtrlListener);
        this.controller.Dispose();
    }
}

LeapCtrlListener.cs

using Leap;

namespace UnoSample.Skia.Gtk.MotionControllers;
public class LeapCtrlListener : Listener
{
    public LeapCtrlListener()
    {
        Console.WriteLine("LeapCtrlListener");
    }
    public override void OnInit(Controller controller)
    {
        Console.WriteLine("Initialized");
    }
    public override void OnConnect(Controller controller)
    {
        Console.WriteLine("Connected");
        controller.EnableGesture(Gesture.GestureType.TYPE_CIRCLE);
        controller.EnableGesture(Gesture.GestureType.TYPE_KEY_TAP);
        controller.EnableGesture(Gesture.GestureType.TYPE_SCREEN_TAP);
        controller.EnableGesture(Gesture.GestureType.TYPE_SWIPE);
    }
    public override void OnDisconnect(Controller controller)
    {
        //Note: not dispatched when running in a debugger.
        Console.WriteLine("Disconnected");
    }
    public override void OnExit(Controller controller)
    {
        Console.WriteLine("Exited");
    }
    public override void OnFrame(Controller controller)
    {
        var leapFrame = controller.Frame();
        var handList = leapFrame.Hands;
        Console.WriteLine($"OnFrame Hand: {handList.Count}");
    }
}