Detecting screen touch

How do we detect a finger on the screen, or multiple fingers, for taps/swipes on clickable objects we show on Jibo’s face? Should we use jQuery to access the DOM directly, or is that possible?



Hi @michael,

Great question!

This is one of those instances where I find it useful to remind myself that Jibo and the SDK are built on electron and, with this in mind, Jibo’s face is basically displaying a web browser and you should be able to treat the elements of his screen accordingly in most all instances.

That means that standard mouse events (supported by Chromium) should be considered when simulating touch in the SDK’s simulator and for the actual robot screen touches will work the same as standard touch events that are supported by Chrome on mobile/tablet devices. In any case, you should be able to develop the code for your skill accordingly.

I hope that is helpful but absolutely let me know if I can provide further input!

John W

Thanks, I guess I jumped the gun a bit on the question. I was trying to access click events with jQuery, but since it’s loaded in via Node, it doesn’t appear to have a persistent DOM connection and thus a $("#elem").on(“click”) call doesn’t work from within an ExecuteScript behavior. I’ll keep playing with it and find a better way around that limitation. I’m guessing it’s probably adding jQuery.js via the index.html file as well.