Phds in Perception-action in Bcis, Robotic

In this interdisciplinary project, we create models of the world, conduct visual data analysis to understand them better, and use machine learning models to yield optimized Brain-computer interfaces (BCI), visualization representations and interfaces, as well as robotic task planning.

This overarching research direction is split into two PhD project topics.

PhD Project 1: Context-specific Grasping Control and Adaptive Visual Interfaces
In robotic manipulation, human-robot, and human-computer interaction, recent research endeavors are concentrated on advancing context-specific grasping control and adaptive visual interfaces. These interfaces incorporate intricate feedback mechanisms utilizing 2D/3D visualizations on computer screens and augmented reality overlays on physical objects. A critical aspect of this research involves the incorporation of multimodal representations, spanning from neural to behavioral patterns associated with the object interaction task, and integrating visual, auditory, and tactile sensory inputs to construct a comprehensive model of the environment. Developing and analyzing such models further refine our understanding of interaction-specific properties between (i) the human or the robotic hand and (ii) the objects or a visualization, enabling nuanced control strategies.

Both in the context of brain-computer interfaces (BCI) and visualization, leveraging multimodal data and machine learning methods can be employed to enable real-time adaptation and optimization. This multidisciplinary project aims to propel the scientific frontier, fostering the integration of non-invasive BCI, computer vision, and machine learning for more sophisticated and context-aware human-machine collaborations and visualization interfaces.

The goal of this project is to address these questions:

- What is the impact of context on the neural correlates of movement? How can a computational model capture these context-related features?
- To what extent are multimodal (e.g., neural and behavioral) patterns resembling during context-specific tasks? How can data visualization techniques be used most effectively to analyze this?
- How can the obtained multimodal representations and the identification of intent enable the dynamic adaptation of a visualization? Can we go beyond adaptively showing controls that allow users to make certain changes or even carry them out automatically (e.g., for rotating a 3D representation)? Would it be beneficial to even switch between whole different visual representations (like switching from a 3D volumetric representation to a 2D slice-based view for medical data analysis)?
- Can adaptive visual interfaces modulate the motor decision-making processes in the presence of multiple competing, potential actions? What is the interplay between the perception of action opportunities and other processes such as action selection, planning, and execution?

PhD Project 2: Neural Task Planning for Optimizing Visualization and Robot Interaction

We will leverage the power of Large Multimodal Models (LMMs) for continual task planning/modeling and visualization design. In robotic domains and visualization alike, a task is often specified in various forms, such as language and visual instructions. We aim to develop a multimodal model that accommodates both textual and visual modalities, overcoming issues of conventional approaches: reliance on complex programming and data collection processes-coupled with limited adaptability and scalability and the involvement of domain experts to explicitly train the robot or adapt a visualization for specific tasks and integrate (rule-based) mechanisms for explanation. To achieve this, we employ LMMs to act as a continual task planner and visualization designer. Our model takes natural task descriptions and the current state as input and generates a hierarchical task plan or visualization as output.

The goal of the project is to address these questions:

- How can we design a process of prompt learning/engineering that allows LMMs to generate task plans that align with the robot’s physical capabilities and motion primitives?
- How can we employ LMMs to transform natural task descriptions into visualizations that enable humans to efficiently address the respective analysis task?
- How can the recursive decomposition of tasks into subtasks be optimized to minimize hallucinations in task planning by LMMs?
- In both simulated and real-robot environments, in what ways can virtual scenarios improve the adaptability and execution success of task plans generated by LMMs? What role does the iterative generation of synthetic images play in enhancing the accuracy and success rate of hierarchical task planning? How can we effectively integrate such a synthetic model into the task-planning process to improve the robot's predictive capabilities?

Organisation
Founded in 1614, the University of Groningen enjoys an international reputation as a dynamic and innovative inst