Mindfocus
Professional projects | | Links:
Overview
Type : Professional
Technology : Unity, Oculus Quest 2, Pico 4
Platforms : PC VR, Android VR
Introduction
MindFocus is a project developed by MindMaze in collaboration with the Centre Hospitalier Universitaire Vaudois (CHUV). It is an immersive virtual reality (VR) solution designed to improve the cognitive functions of brain-damaged patients.
MindFocus offers gamified activities adapted to each patient’s level, with personalized assessment and remote follow-up by therapists. Its aim is to optimize cognitive rehabilitation by specifically targeting certain cognitive functions, a frequent deficit in brain-damaged patients.
Mindfocus is part of the SWISSNEUROREHAB project, which has received an Innosuisse grant. A study has been funded to demonstrate that Mindfocus is more effective than conventional therapies. This study will involve immersive, personalised training to improve patient motivation and adherence. For more details please click here.
And if you’d like to see how Mindfocus is used at the CHUV, take a look at the short report by RTS.
Project
The first version of MindFocus was developed for virtual reality on the PC, using an HTC Vive Pro Eye headset (HMD). This choice made it possible to exploit the eye-tracking data provided by the headset, in addition to data on the patient’s progress.
The application consists of two parts: a session control interface used by the therapist on a computer, and a VR headset experience where the patient plays therapeutic games.
The application includes a main menu for selecting the different activities offered to the patient. Each activity has its own environment adapted to the different scenarios. The activities are divided into two categories: assessments and games.
Assessments are simplified, non-gamified cognitive tasks, similar to those performed outside VR. They are used to measure the patient’s cognitive state and check progress throughout MindFocus therapy.
Games are cognitive tasks that are gamified with a scenario and an environment. These are the games the patient will play to improve cognitive function.
Each game and assessment targets one or more cognitive functions, with an emphasis on attention and executive functions. Categories covered include :
- Alertness
- Spatial and selective attention
- Spatial attention and neglect
- Selective attention and inhibition
- Divided attention
- Cognitive flexibility
- Working memory
- Planning
For each category, one or two games have been developed. In cases where there are two different games for a cognitive function, the second game is essentially a variation of the first, with only the scenario and environment being changed. The aim of this reskin is to offer the patient more variety while maintaining the same core gameplay.
Each game consists of multiple levels with increasing difficulty. These levels are designed to progressively enhance cognitive function training for the patient and can be adapted to suit the individual needs of each patient.
This version is currently in use at CHUV and the Lavigny institute.
PC VR version menu
Following this initial release, a standalone VR version was developed for the Quest 2 and Pico 3/4 headsets. These headsets offer simplified use, as they operate autonomously, without the need for a PC or tracking stations like the Vive.
In future, this version will enable patients to use the headset at home, increasing the frequency of sessions without having to come to hospital.
A web-based platform has been created to compensate for the therapist’s inability to configure directly from the headset. It enables centralized management of patient data and improved monitoring thanks to in-depth analysis of the data collected. The therapist can also take control of the session remotely and launch games for the patient. A real-time streaming function has been added, enabling the therapist to visualize the patient’s activity in VR. Finally, the platform integrates a therapy planner, a calendar for planning sessions. This feature is particularly useful when the patient uses the headset at home, enabling him or her to follow a pre-defined therapeutic program.
Web app screen shot
Developement
I joined the project as a secondary developer. My two main missions were to integrate gamification elements between games to keep patients motivated and offer them a sense of progression, and to design a planning game in collaboration with a PhD Psychologist in neuropsychology at CHUV. In addition, I contributed to the overall improvement of MindFocus, by carrying out bug fixes and performance optimizations.
Gamification
The purpose of adding gamification and level progression to Mindfocus is mainly to keep the patient motivated and engaged in therapy. Especially for patients who will be doing therapy at home without the therapist’s intervention.
Medal Ranking system
Once the activity was completed, a score was calculated according to the objectives achieved/failed. This score was displayed to the patient as a percentage. Although functional, this score remains very formal and is not very motivating or exciting.
A reward system with stars had been set up initially, but it provided little information on the patient’s progress. We decided to add a ranking system with medals to the score displayed at the end of the game.
The advantage of medals is that they are a universally recognized system of reward, making them an effective way to acknowledge achievement and motivate patient. The other advantage is that it’s easy to self-assess your level: by comparing the number of medals and their rank, you can see whether you still have some progress to make, or whether you’ve already successfully completed all the activities.
To simplify the medal ranking and maintain consistency with the rest of Mindfocus, we use existing level recommendation system for define the diiferent medal rank. Each activity level as regression and progression threshold which suggests the next level to play to the patient based on the result of the game he has just finished.
We have defined the ranking and medals as follows :
- No medal for a score below the regression threshold
- Bronze medal for a score between the regression threshold and a intermediate threshold calculated between the regression threshold and progression.
- Silver medal for a score between the intermediary threshold and the progression threshold
- Gold medal for a score above the progression threshold
The patient can obtain only one medal per level per game, if he improves his score, the new medal replaces the previous one. Since medals are linked to the progression system, they also make it easy for patients to understand why they can or cannot progress to the next level.
- No medal: downgrade to lower level
- Bronze and silver medal: restart level
- Gold medal: upgrade to the next level
Progress visualization
To help patients visualize their progress during therapy, 2 new UI elements have been added to the lobby and activities.
In the lobby, a new panel on the left has been added. It displays the patient’s overall progress. Three metrics are presented to the patient : total medals for each rank playing time adherence to therapy (played activities / planned activities)
Lobby progress screen
In the activities, the display and selection of the level has been changed. Each level is represented by a hexagon-shaped button, arranged in rows where each row represents a category. Category names have been replaced by simple letters. This makes it easier for patients to read the progression of levels. All level buttons are placed in a scroll view, allowing fluid navigation between levels and categories, while displaying only a small portion of the game’s levels.
In addition to representing levels and categories, each button also represents the level status and current rank. The button is grey if the level is not unlocked, and blue when it can be played. A colored border (gold, silver, bronze) is added to show which medal the player has obtained. This level representation is used at the beginning and end of the activity.
Level selection
Planning game
As part of the development of this new game, we collaborated with Charlène Moser from the University Neurorehabilitation Service at CHUV. This project is part of her PhD work and also involves a partnership with MindMaze. The goal was for her to provide us with all the clinical requirements so that we could design a concept and prototype tailored to those needs.
The planning exercises traditionally used at CHUV mainly involve “agenda” tasks. Patients are given a series of activities to organize, each with specific times and dependencies, and must construct a coherent schedule. However, this type of exercise lacks real-world application—the planning remains abstract and disconnected from actual execution.
The concept behind the planning task in Mindfocus aims to bridge this gap by offering a game structured in two phases: a planning phase and an execution phase. During the planning phase, the patient must create a schedule while considering various constraints (temporal, spatial, financial, etc.) provided in the instructions. Once the planning is validated, the patient moves on to the execution phase.
In this second phase, the patient is immersed in a virtual environment designed to be as realistic (ecologically valid) as possible and must carry out the planned tasks while making decisions along the way. Unexpected events may arise, forcing the patient to adapt and modify their initial plan. The game is designed to include a wide variety of constraints and scenarios, allowing it to be repeated multiple times by the same patient without becoming repetitive.
The main design challenge was to create a scenario and gameplay that introduced a rich diversity of constraints while maintaining coherence between the planning and execution phases. Additionally, the game had to comply with the technical limitations of VR development and the physical constraints of the patients, particularly the seated position.
Space Cargo
Several scenario ideas were explored, such as planning a trip around the world, organizing activities in a house, or guiding a small remote-controlled robot through a maze.
The final concept that was selected is called “Space Cargo”. In this scenario, the patient is tasked with organizing and executing space cargo delivery missions while managing several constraints: delivery times, travel routes, fuel costs, and cargo capacity. The objective is to ensure the successful completion of the deliveries.
This scenario could have been set on Earth—for example, using a delivery truck—but the decision to place it in space was made for both technical and design reasons. A space environment is easier to create, reduces the risk of motion sickness in VR, and offers something unique and engaging for patients. In fact, the current favorite game among patients also takes place in space.
Here are the main constraints included in this scenario:
- Pickup and delivery times
- Cargo lifespan
- Cargo size
- Maximum transport capacity
- Fuel costs
- Unexpected events: space pirate attacks, asteroid fields, blocked routes, and urgent delivery requests
During the planning phase, the patient is placed in a “planning center” aboard the spaceship. They receive a list of deliveries and a map of the galaxy, and must organize their actions (pickups and deliveries) using a dedicated interface. Once the plan is validated, they can move on to the next phase.
They are then transported to the cockpit of the spaceship for the execution phase, where they follow their plan, select destinations, and confirm their routes through simple interactions—such as activating a light-speed lever or pressing buttons.
Space cargo prototype
Unfortunately, my involvement in the project ended at this point. We had only just begun setting up a prototype, which was not yet functional when my contract ended.
Despite that, I’m confident that the work accomplished during the design phase is important, as it lays the foundation for the next developer to create a fully-fledged and clinically relevant planning game.
Constraint Optimization
One of the solutions tested to solve the problem posed by the Space Cargo scenario is the use of a constraint solver. This is a set of algorithms for solving constrained optimization problems. These algorithms explore a space of solutions while respecting predefined rules (constraints), and use advanced methods such as linear programming and local search to find optimal solutions. Constraints can include resource limitations, time windows or relationships between variables.
To test this type of algorithm, I used an exercise we had built ourselves. This allows us to compare the solution found by the algorithm with our own solution. I chose to use OR-Tools, an open-source library developed by Google, to solve combinatorial optimization problems such as vehicle routing and constrained planning. This library is compatible with several languages (Python, C++, Java).
To solve a problem with a constraint solver, we first model the problem by defining different data structures. Constraints are then applied to this model before it is submitted to the solver, which attempts to find an optimal solution.
In the case of the Space Cargo scenario, the problem can be summarized as a vehicle routing problem with time windows (VRPTW) including pick-ups and deliveries. Here are the key elements of the model:
Problem modeling
- Time matrix: indicates travel times between points.
- Time windows: each location must be visited within a given interval.
- Pick-up/delivery relationships: each pair defines a pick-up and corresponding delivery.
- Vehicle capacity: limit of transported volume.
- Gains and penalties: each delivery earns a gain, and a penalty is applied if a point is ignored.
- Number of vehicles: here, a single vehicle is used.
Constraints applied to the model
- Respect time windows and vehicle capacities.
- Optimize gains by minimizing penalties.
- Management of waiting times between deliveries.
The algorithm generated an optimized route, but with a waiting time of 5 hours between certain deliveries, whereas our manual solution required only 3 hours. By adjusting certain model parameters, I managed to obtain the expected solution, but this requires precise finetuning.
These algorithms have great potential, however, a compatibility problem between the library and the Android platform prevented us from using it in our project.
Use at home
Another important task was to anticipate and test the use of the headset by the patient at home.
Indeed, to make home use as simple as possible, it is necessary to simplify the interactions required by the patient as much as possible, while thinking about possible blockages that could occur. One of the biggest problems lies in interaction with the helmet’s operating system. When the headset is switched on, you access the main menu, then select the MindFocus application to start the therapy. You can also access the headset’s various parameters, such as wifi or bluetooth. To solve this problem, I used the kiosk mode. This special mode on the Pico 4 locks the headset in single-use mode, restricting access to applications and settings to provide a controlled experience. With kiosk mode, you can start an application as soon as the headset starts up, modify shortcuts on the joysticks or change the settings available in the headset’s quick settings menu. All these features enabled me to create a simplified experience for the patient. The remaining problem was the initial installation of the headset in the patient’s home, as the application required Internet access to function. This meant that the therapist had to visit the patient’s home to help them enter the Wi-Fi password to connect the headset to the Internet.
In addition to the addition of the kiosk mode, I’ve also written a short guide providing the patient with all the necessary information in case of doubt about how the helmet works. It summarizes the procedure for turning on the helmet, the joysticks, the various controls and how to recharge the helmet. This little guide is included in the helmet box, which is given to the patient.
MindFocus Quick Start Guide - Pico 4
General improvements and bug fixes
On top of these additions, I also took part in the overall improvement of the project.
In particular, I worked on optimizing the streaming between the headset and the webapp. A memory leak was present, which could lead to crashes. I reworked the retrieval of the helmet view in a texture without memory overhead. In addition, I improved the streaming display in the webapp, enabling the window to be moved and resized for a more flexible experience.
I also helped optimizing the scenes of the games, reducing loading times and improving overall project performance. As some scenes were more resource-intensive than others, it was essential to standardize scene consumption to ensure a seamless experience.
I created a performance recorder that allows CDV files to be exported from the headset, providing detailed frame data metrics. A Google Sheet has been set up to compare and analyze these files, making it easy to interpret the results with graphs showing frametime, amount of memory used, etc.
Finally, I resolved a number of bugs to enhance the project’s stability and fluidity.