The Emoji Bar
CPSC 581 Fall 2021 - Project 4
CPSC 581 Fall 2021 - Project 4
Created By: Abraham Beauferris, Marcello Di Benedetto, Christopher Rodriguez, Rory Skipper
This project seeks to create a novel interface to augment in-person and/or remote communication and collaboration. As an answer to this, we created a novel interface capable of allowing users to remotely communicate through emojis that correspond to the user's expression and pose. Users can emote in real life and the corresponding caricature will be shared, facilitating communication seamlessly and without the use of one's direct likeness.
This system can be used to facilitate the private communication of emotions and gestures without the need to have a webcam on throughout periods of collaboration. This system is also a more lightweight design compared to video conferencing software, and could be implemented as an “overlay” over full screen applications. It allows remote collaborators to get a sense of your emotions or gestures without exposing your face.
We used a collaborative take on the 10 + 10 ideation method for this project to decide on a final design with useful and intuitive features and use cases.
Intial ideation sketches for a prototype to "augment in-person and/or remote communication and collaboration"
This design describes an interface in which users’ real time expressions are visually represented into corresponding emojis to facilitate remote communication. This idea seemed like a novel augmentation of remote communication with a reasonable scope and so we Ultimately decided to pursue it.
This design describes an interface for users to converse remotely in a virtual reality environment, having functions such as suggesting and watching YouTube videos while sharing the virtual environment together. We ultimately decided this would be difficult to prototype due to the amount of headsets needed.
This design describes an augmentation of a video-calling interface, in which the person that the user is visually focused on is maximized. While novel, we decided such an interface would be technically difficult to implement.
This design describes an app prototype that facilitates remote collaboration by providing an interface for group study sessions. Ultimately we decided the interface did not augment the communication or collaboration as much as we desired.
This design outlines a physical touch like interface that would show the user's inputs on an accompanying desktop application. We did not pursue this as it seemed to similar to common digital drawing pads.
This design describes a virtual checkers interface that is updated based on a corresponding physical checkers board. While novel we ultimately decided the concept did not satisfy the design goals of augmenting communication and collaboration strongly enough.
This design describes a prototype interface where one could act as a remote director through virtual reality. While novel we ultimately decided against pursuing it due to its high complexity required to create a minimum viable prototype.
This design describes an interface that augments a cooking experience. It would allow a user to remotely cook through some sort of virtual interface. Unfortunately, while the concept is novel, it seemed difficult to realize due to safety concerns.
This design describes a system where someone could be served a meal by a remotely controlled trolley. While novel, we felt it wasn’t a direct fit for our design goal of a novel way to communicate or collaborate, as the interaction would be limited to serving a meal.
This design describes another interface prototype, this time describing a method for multiple users to collaborate remotely. It entailed superimposing user’s likeness from a video-chat feed into an augmented reality environment, enhancing the realism of the communication. We ultimately thought this idea was novel but would be difficult to implement in a convincing manner given the time restraints.
Moving forward from the ideation phase, our group chose to iterate on concept #1, “The Emoji Overlay”. We landed on this concept as this system aims to solve a well defined problem that is especially relevant to our position as students working remotely in collaborative environments. This relatively simple design would provide a great benefit to the user, while also allowing us to become more acquainted with interesting libraries that identify changes in the user’s facial expressions and gestures.
The following are some details and variations of our chosen concept, the “The Emoji Overlay”
This detail outlines the system’s support for multiple concurrent users. Each user will be displayed on the overlay, excluding the user who is running the program. Emojis will be labelled with the individual’s username to allow for quick differentiation. This was implemented in the final product as it showed a scalable solution to our problem.
This variation includes a color coded display for classes of emojis. “Positive” emotions would be green, “Negative” are red, “Neutral” are yellow and “utility” emotes would be blue. This would allow the user to at a glance, get a ballpark notion of how another person is feeling. This was not implemented in the final design as we thought giving each emotion a unique color was more aesthetically pleasing and was still intuitive enough to support this “at a glance” functionality.
This variation would parse the user’s numerical hand gestures into the numbers they represent. This was not implemented in the final design as the accuracy was greatly limited by the Google Teachable Machine.
This detail shows a “non-intrusive” design for the overlay. The window containing the emojis for each individual would be transparent, allowing the user to use full screen applications alongside this overlay. This was not implemented in the final design due to implementation limitations, but would not be difficult to implement in future iterations using more appropriate frameworks.
This variation sketch describes a version of the system that includes a physical counterpart to the system. Instead of displaying the other user's emojis on a UI overlay instead, the system will display the emojis on a physical counterpart on each user's desk. We did not end up using this variation due to its difficulty with upscaling to multiple users.
In this detail we describe a feature for the system that allows for the UI overlay to be resizable. This would allow for more flexibility and ease of use of our system. This was not implemented in the final design due to limitations in the GUI libraries used.
This detail describes another feature which would allow for the users of the system to change their display name for other users to see. This would help with the anonymity and privacy angle of our project. This was not added to the final design as it did not add much for a proof of concept.
This variation depicts a version of our system that upon interpreting certain gestures and facial expressions will relay different motions to a mobile robot in this case Toio. This variation would try to emulate human physicality using different gestures, but we did not implement this variation in the end as it would require every user to have a Toio robot.
This detail introduces using hand gestures in the system. In this picture we show examples of going forward or moving backwards gestures. In the final system we ended up using different hand gestures including a “peace” sign, and an “okay” sign.
This is another variation of the project that would introduce a queue like system for the emoji display. This describes a version that would show users which emojis they are sending to the chat and which emoji is currently being displayed, is next to be displayed, and what was previously displayed.
For the final design, we chose to implement refinement 9: “gesture based inputs” and refinement 4: “Color coding” to varying degrees. Gestures were included in a smaller scope, supporting only three gesture based inputs, and color coding was utilized on a per emoji basis to support “at a glance” usage. Refinement 1 was implemented with networking functionality, to reinforce this tools intent as a remote collaboration and communication tool. These refinements were chosen as they provided a reasonable breadth to the functionality and design for a prototype as a proof of concept for this system.
This project was implemented with Python 3.8.7 using the socket library to structure a client/server architecture to update all remote applications with each individual's changes in emotions. Facial expressions and emotions are detected through the user’s webcam, and run through the DeepFace library, which uses a trained neural network to identify emotions such as sadness, anger, happiness, surprise or fear. Gesture based inputs are identified with a Google Teachable Machine we trained ourselves, allowing the parsing of webcam data into gestures including “thinking”, “ok” and “peace”. These states are received by the client application, and emoji based representations are shown accordingly. The GUI was implemented with TKinter, which came with some limitations to the aesthetic design of the application as our imagined overlay-like display with transparency was not possible with this library. In further iterations, this system would be adapted to a desktop based Javascript application to better facilitate a responsive and aesthetically pleasing program layout.
My personal contribution to this project was the integration of libraries for the software, along with participating in pair programming for the program logic. All group members contributed to the ideation and refinment process of the 10+10 design process, along with the documentation of the portfolio.