Achievements and evaluation of the project.

Evaluation

Accomplishments
The following table shows which requirements we were able to meet during development over the past few months.

ID

Requirement

Priority

Status

Contributors

Touchless Kiosk Controls

F1.1
The system must allow controlling web-based kiosk interfaces using hand gestures such as swipes, finger pinches etc.
MUST
Aaryaman, Aryan
F1.2
The system must be able to track the relevant hand and ignore the disturbance from the environment
MUST
Aaryaman, Aryan
F1.3
The gesture controls must be relatively performant and run at at least 20fps on a mid-powered kiosk computer (i5 8th gen, 4GB RAM)
MUST
Aaryaman, Aryan
F1.4
The system should allow adjusting parameters such as gesture sensitivity, detection thresholds, and deciding between using the right or left hand.
SHOULD
Aaryaman, Aryan

End user frontend (Web Extension)

F2.1
The extension must be able to hook into any web-based interface without modifying its source code.
MUST
Aaryaman
F2.2
The extension must be able to display a live preview of the user's hands so that they can more reliably control interfaces via gestures.
MUST
Aaryaman
F2.3
To respect the user's privacy and avoid embarrassment, the live preview should display an abstracted version of the user's hands.
SHOULD
Aaryaman
F2.4
The extension should allow changing the sensitivity of the gestures in real-time based on user preference.
SHOULD
Aaryaman
F2.5
The extension should have a status box indicating if the backend software is active.
COULD
Aaryaman
F2.6
To improve resiliency, the extension should be able to relaunch the backend if it stops functioning.
SHOULD
Aaryaman

Communication Protocol

F3.1
The communication protocol must support duplex communication between the front and the back end.
MUST
Aaryaman
F3.2
The communication protocol must support the manipulation of config settings and allow tweaking parameters such as sensitivity in real time.
MUST
Aaryaman
F3.3
The communication protocol must not use any remote endpoints and work without network access to maintain security.
MUST
Aaryaman
F3.4
The communication protocol should provide an API for external use and use a consistent JSON schema.
SHOULD
Aaryaman
F3.5
The communication protocol should work even in the event of MotionInput crashing and should be able to relaunch it to improve resiliency.
SHOULD
Aaryaman

Source Compilation

F4.1
There must be an easy-to-use compilation script that can compile the backend into an executable that doesn't require additional dependencies.
MUST
Aaryaman, Aryan
F4.2
The executable must target the Windows operating system from Windows 10 and above.
MUST
Aaryaman, Aryan

Touchless Kiosk Launcher

F5.1
The launcher must not display the MotionInput live preview screen or open the windows console.
MUST
Aryan
F5.2
The launcher should have a single launch button which starts the server and MotionInput.
SHOULD
Aryan
F5.3
The launcher must allow the administrator to set the desired sensitivity of the swipe gesture.
MUST
Aryan
F5.4
The launcher must allow the administrator to set the desired hand (right or left).
MUST
Aryan

Key Functionalities (Must Have & Should Have)

100 % Completed

Optional Functionalities (Could Have)

100 % Completed

ID

Requirement

Priority

Status

Contributors

Touches Kiosk Controls

NF1
The code must be open-source and easily understandable so that future developers can access the API and backend, enabling them to create their own extensions.
MUST
Aaryaman, Aryan
NF2
The code should be well-maintained, with bugs regularly addressed, and updates made available for future compatibility.
SHOULD
Aaryaman, Aryan
NF3
The system's structure should be straightforward enough for individuals without extensive technical knowledge to deploy it easily.
SHOULD
Aaryaman, Aryan
NF4
Writing documentation could increase convenience for users and developers, both for utilizing the system and for further development.
COULD
Aaryaman, Aryan

Key Functionalities (Must Have & Should Have)

100 % Completed

Optional Functionalities (Could Have)

100 % Completed

Known BugsThese are the bugs we were aware of by the end of our development .

ID

BUG Description

Priority

B1
The detection of swipes can sometimes be unreliable, especially when the sensitivity is set low and the hand is positioned close to the edge of the camera frame.
LOW
B2
When the sensitivity is changed through the MFC, the sensitivity slider UI element is not updated.
LOW

Individual ContributionThe following table displays each team member's contribution to the project.

Task

Aaryaman

Aryan

Yidan

Client Liason
40
60
0
Liason with other Teams
50
50
0
Requirement Analysis
40
30
30
Pitch Presentations
30
20
50
UI Design
33
33
33
System Design
50
50
0
Coding
60
40
0
Testing
50
25
25
Report Website
15
70
15

Overall Contribution

40%
40%
20%

Critical EvaluationThis section talks about our thoughts on how well we implemented our solution.

A. Control Method:
Functionality
Overall, our design of gestures that fully support most existing touch kiosks can be considered a success. The gestures are intuitive enough to avoid causing much frustration, and once a user adapts to them, they are nearly as efficient as touch controls. Additionally, the movements are smooth enough to meet accessibility demands.

However, there is still room for improvement. New users may feel overwhelmed without any instructions, which can lead to confusion. Nevertheless, this is an expected outcome when introducing a new control mode that replaces a popular one.

One limitation of our design is that we did not have enough time to implement a head gesture mode. This limitation arguably reduced the accessibility of our project.
B. User Interface
Our user interface has gone through several iterations and has proven to be successful in achieving our initial purpose. We made minimal changes to the website's original structure but added a live preview box that reminds users of the position of their hand relative to the MotionInput checking zone. This addition has effectively reduced confusion by providing users with feedback.

However, there is still room for improvement. We could add text notes to provide better user instructions. The lack of such notes caused confusion, as mentioned in the control method section.
C. Web Extension
Stability & Compatibilty
One crucial aspect of the web extension was the system's robustness, making it easier to restart after a crash. As a result, it has fully accomplished its goal. It can connect to a website without requiring modifications to the source code, and its sensitivity can be adjusted in real-time without restarting.

However, there are limitations with the MotionInput backend. Currently, changing between left and right hands in real-time is not possible. To do this, a total restart of the MotionInput backend is required, which could be an area for future improvement.
D. Crowd Detection
Functionality & Efficiency
Our crowd detection function uses Mediapipe to find the best hand detected in the video feed and tracks it. This ensures that the system responds only to the best-fitted hand and rejects undesirable gesture inputs, such as those from the environment.

The result of this feature is positive, as it works well to reject interference captured by the webcam and to switch to another person's hand when needed.

We did notice some minor frame loss due to the calculation required for this function. However, it did not cause any inconvenience in using the system.
E. Internal Communication
Reliability
We designed the communication protocol between the frontend and backend to support the web extension functions. Utilizing ZeroMQ, we ensured smooth and efficient communication between the Motioninput API, the intermediary server, and the frontend. This design fulfilled all the requirements we listed before and is considered to be without much room for improvement.
F. Launcher
Functionality
The Motioninput launcher is built upon the MFC library and provides a user-friendly interface for adjusting settings. This is especially helpful for users without programming experience. Although it was a minor function added late in the project, it worked well and was developed in no more than two weeks. We did not discover any major issues in this area.

Future WorkThis section talks about the future work that we would have carried out if we had more time.

A. Support for other Gesture Modes
Currently, our system only supports hand gestures, which could be an accessibility problem for those who are unable to use their hands. To address this, we need to make changes to the backend logic and modify the web extension feedback and launcher to support other gesture modes.
C. Optimising Performance
In some cases, the system experiences minor frame loss when tested on low-performance machines. While this doesn't cause significant issues and is not expected on common kiosk machine configurations, we could still optimize the backend to ensure consistent performance in different environments.
B. Improving User Feedback
Our current user feedback has caused slight confusion, especially during initial encounters. To provide clearer guidance, we need to incorporate text notifications or other direct instructions. This addition may require a redesign of the frontend UI.