INTEGRASI VOICE AI DAN FACE TRACKING UNTUK PENINGKATAN REALISME KARAKTER VIRTUAL INTERAKTIF MENGGUNAKAN RESEARCH AND DESIGN

Interaction with realistic virtual characters has become a key focus in the
development of modern digital applications, ranging from games to artificial
intelligence. However, the lack of synchronization between voice responses and
visual expressions can often diminish users’ perception of realism and immersion.
This research aims to develop and evaluate a system integrating Voice AI and
realtime facial animation in interactive visual characters to enhance the quality of
interaction. Adopting a Research and Design (R&D) methodology, the system was
built through the stages of analysis, design, development, and evaluation. User
voice input is converted to text by Whisper AI, which is then processed by the Large
Language Model Ollama 3.2B to generate a response. These responses are
synthesized into natural speech via Text-to-Speech LMNT, while lip-sync and
realtime facial expressions are generated using Reallusion AccuFACE on iClone,
then integrated and visualized within the Unity environment. This system is
designed to create “relatable and multidimensional” conversational characters, with
the specific aim of “enhancing the realism of interactions with ICT systems”
through “tangible interactions.” The evaluation results are expected to demonstrate
that the integration of high-quality voice and synchronized facial animations can
significantly enhance users' perceptions of realism, naturalness, and emotional
engagement. This research contributes to the development of more lifelike and
interactive virtual characters, opening up broader application potential in the future,
in line with the importance of “real-time lip-sync” that can “guide user
interactions.”