🔮 AR Touch

Obtain the power of touchless interaction! Being able to effortlessly navigate and interact with various screen displays without any physical contact, is what this project brings you.

While touchless technology exists, it often remains out of reach for many due to its limited availability and high costs. Our mission with AR Touch is to bridge this gap and democratize touchless interaction for everyone. By implementing this cutting-edge technology only in a user-friendly Android app, we enable you to interact with any screen, regardless of whether it supports touch input or its dimensions.

Features

• It is independent of the target operating system, that is, there is no need to install a special interface software (I already installed it for you!) and it has been tested on Windows and Android operating systems.
• All the processes are done in the device itself, so there is no need to spend extra money to buy an additional tool.

The magic behind it

When we place our fingers on the touchscreen, occurred an interrupt to handle touch, which the operating system must handle.

Since we need to have the touch event first for the OS to handle it, we must first determine the touchable area. For this purpose, I have utilized ArUco Markers to identify the edges of the screen frame.

Additionally, I have used hand gestures to generate events, and for their respective computation, I have employed on-device ML-powered solutions available within the MediaPipe framework.

To deliver the touch event independently of the OS, I have utilized BLE technology and the HID protocol.

🔍 Prerequisites

1. A device which Android 9 was init version
   • with mid-range processor e.g. Snapdragon 730
2. Four ArUco markers, can be downloaded here

🚫 Limitation

• Only one hand can be placed in the frame of the screen, so currently zooming is not available as usual.
• Currently, the touch event is generated in line with the content displayed on the phone screen.

🎬 Demo on Youtube

🛠️ Technology Stack

• Kotlin
• C/CPP
• View System
• Jetpack Compose
• CameraX
• NDK and JNI
• OpenCV
• MediaPipe
• Coroutines
• Flow
• Bluetooth Low Energy
• Koin

🏗️ Architecture

Uses modularization approach and follows the official architecture guidance.

🧪 Testing & Quality Assurance

Testing has been performed in this project without using common frameworks for mocking, relying solely on dummy, stub, shadow, and fake test doubles.

Robolectric, a powerful simulator, has been utilized to take advantage of its benefits in executing UI and Integration tests.

To execute Isolated Instrumentation tests, the Orchestrator tool has been enabled in Gradle.

To enhance the components of the producer module, Performance tests have been considered using the Microbenchmark tool. Additionally, to prevent code leaks in production, a separate module has been designated for them.

You can see the list of tested classes here.

👷‍♂️ Steps to Build

1. Download NDK version 25.2.9519653 from the SDK manager.
2. Download and extract the Android OpenCV SDK version 4.10.
   1. Create the opencv.properties file in the root project.
   2. Set OpenCV SDK directory to sdk.dir property.
3. Build the project as an Android Developer.

📚 More about first release

If you want to know about the process of evolution and other aspects of this project, you can read the project report. This project was for my undergraduate degree at the University of Tabriz.

• English version
• Persian version

🤝 Contributation

I welcome and appreciate contributions from the community to improve the AR Touch project. If you have any ideas, suggestions, or feedback to enhance the project, feel free to share them in the Discussions section.

You can actively participate by:

• Opening discussions to propose new features or improvements.
• Creating new issue to report the bugs you encounter during testing.
• Submitting pull requests to introduce code enhancements or optimizations.

💖 Sponsors and Supporters

Thank you for considering supporting the AR Touch project! Your donation, no matter the amount, makes a significant impact and inspires us to push the boundaries of what's possible. By clicking on the Sponsor button right now, you can become a valuable partner in the project's evolution.

Let's make magic together! 🌟

📜 License

AR Touch is distributed under the terms of the Apache License (Version 2.0). See the license for more information.