Apr - Jun 2024

Tennis VR (developed in Unity)

Assignment and Skill

Create a virtual reality experience focused on developing a specific physical skills.

Skill: Tennis

We focused on the hand eye coordination of the virtual racket and the ball as well as quick responses of the body.
When the ball is bouncing back, there will be a guided line to show where the ball will bounce, helping the user get used to moving into optimal positions to strike the ball.

There is also an AI agent/opponent that provides specific feedback based on where the user hits the ball.

Pros and Cons of the AI Agent

Over the course of this project, we explored the benefits and disadvantages of an AI agent that provides feedback trained by the user's actions.

How can an AI-based VR system help in learning or practicing or enhancing this skill?
Users can test their reaction time and test their skills without much pressure and without worry of an injury (such as sunburn, strain from muscles, etc). Users would not have to worry about purchasing expensive equipment and finding a partner. They would also be able to learn the rules of the game using practice.
Without the weight of rackets and the pressure from hitting the ball, this would be a lot less strain on the body.

In what ways adding AI could be counterproductive or not helpful to the user?
Adding AI could be unhelpful to the user if there are some glitches in the movement, leading to the user being confused. Also, if the AI gives the wrong instructions to the user. The result of this would be the user learning the wrong skills and not being able to play the sport in real life properly. Furthermore, the experience of playing tennis virtually may have a different feel than in reality, which may make the skills difficult to transfer.

How can an AI-based VR system help in learning or practicing or enhancing this skill?
Users can test their reaction time and test their skills without much pressure and without worry of an injury (such as sunburn, strain from muscles, etc). Users would not have to worry about purchasing expensive equipment and finding a partner. They would also be able to learn the rules of the game using practice.

Without the weight of rackets and the pressure from hitting the ball, this would be a lot less strain on the body.

In what ways adding AI could be counterproductive or not helpful to the user? AI is not perfect. What are some possible challenges or negative outcomes that could come from adding AI to this specific skill (i.e., what would be the result of AI errors and unpredictability for the user trying to learn this skill)?

Adding AI could be unhelpful to the user if there are some glitches in the movement, leading to the user being confused. Also, if the AI gives the wrong instructions to the user. The result of this would be the user learning the wrong skills and not being able to play the sport in real life properly. Furthermore, the experience of playing tennis virtually may have a different feel than in reality, which may make the skills difficult to transfer.

Target Users

Target Audience: Beginner tennis players

User goal: Let people who are new to tennis practice hand eye coordination and learn the rules of the game.

In this project, the specific aspect of the skill we focused was the essential motions in tennis such as swings and serves.

How can an AI-based VR system help in learning or practicing or enhancing this skill?
Users can test their reaction time and test their skills without much pressure and without worry of an injury (such as sunburn, strain from muscles, etc). Users would not have to worry about purchasing expensive equipment and finding a partner. They would also be able to learn the rules of the game using practice.

Without the weight of rackets and the pressure from hitting the ball, this would be a lot less strain on the body.

Design and Function Requirements

System requirements and design parameters:

Environment: the court, net, how much space is around the user
The user’s movement and position
The user’s opponent (another user/ wall)
What the user is picking up (tennis racket and ball)

Interaction Modalities: interaction modalities that could and could not be used in our system:

Since both hands should grab one single racket, which is a single hand controller in this case, system input would only go into one controller. (User may choose dominant hand for L/R input)
Since the eyes would be focused on the ball, eye-tracking for options such as pausing training would not be ideal
Gesture recognition for options like pausing training isn’t ideal. Motions while waving around racket during gameplay may be misinterpreted

How can an AI-based VR system help in learning or practicing or enhancing this skill?
Users can test their reaction time and test their skills without much pressure and without worry of an injury (such as sunburn, strain from muscles, etc). Users would not have to worry about purchasing expensive equipment and finding a partner. They would also be able to learn the rules of the game using practice.

Without the weight of rackets and the pressure from hitting the ball, this would be a lot less strain on the body.

Workflows

We created two flowcharts that illustrate the steps that the users take by:

1) using the tools that we would provide for them in the scene,

2) and with the guidance of the AI agent:

Features & AI Agent

When creating our scene in Unity, we incorporated these main features:

Racket functionality
Scoreboard
AI Agent (audio instructions)
Trajectory line: guides user on where to hit the ball
Ball Respawn

The instructions we created for the AI Agent to say were catered towards humans, providing positive reinforcement even when the user does something wrong (such as "Nice job, but…"). This established a comfortable and non-judgmental setting for the user when learning a skill for the first time.

Gulfs of Execution/Evalution

After implementing basic functionality, we examined the gulfs of execution and evaluation that were present in our simulation, and provided solutions to bridge these gaps between the user, environment and the AI.

Some examples of these gulfs:

Gulf: User >> Environment [Gulf of Evaluation]

Users have difficulty predicting the trajectory of the ball after they hit it because the VR system does not provide visual or sensory cues. This results in the ball landing far from the court lines, which can hurt their learning process. The system did not provide immediate and clear feedback on the quality of the user's shots: the game does not clearly communicate the current score, the rules, or the implications of specific actions (e.g., double faults, let serves).

Solution: We implemented a trajectory line that anticipates the direction of the ball after the user hits it. Additionally, the AI provides immediate, detailed feedback every other shot, including information on ball landing accuracy.

Gulf: User >> System/AI [Gulf of Execution]

The user wants to serve the ball but doesn’t know how to, because the AI only says “Now serve the ball.” The AI might not correctly interpret the user’s intentions, such as the direction and strength of a swing. This results in the ball going in unintended directions or not responding as expected, making it hard for users to control their shots and gameplay.

Solution: The AI provides clearer instructions on how to serve the ball. For example, now, the agent says “Throw the ball upwards and hit the ball with your tennis racket.”