Clustering Based Kernel Reinforcement Learning for Neural Adaptation in Brain-Machine Interfaces.

Reinforcement learning (RL) interprets subject's movement intention in Brain Machine Interfaces (BMIs) through trial-and-error with the advantage that it does not need the real limb movements. When the subjects try to control the external devices purely using brain signals without actual movements (brain control), they adjust the neural firing patterns to adapt to device control, which expands the state-action space for the RL decoder to explore. The challenge is to quickly explore the new knowledge in the sizeable state-action space and maintain good performance. Recently quantized attention-gated kernel reinforcement learning (QAGKRL) was proposed to quickly explore the global optimum in Reproducing Kernel Hilbert Space (RKHS). However, its network size will grow large when the new input comes, which makes it computationally inefficient. In addition, the output is generated using the whole input structure without being sensitive to the new knowledge. In this paper, we propose a new kernel based reinforcement learning algorithm that utilizes the clustering technique in the input domain. The similar neural inputs are grouped, and a new input only activates its nearest cluster, which either utilizes an existing sub-network or forms a new one. In this way, we can build the sub-feature space instead of the global mapping to calculate the output, which transfers the old knowledge effectively and also consequently reduces the computational complexity. To evaluate our algorithm, we test on the synthetic spike data, where the subject's task mode switches between manual control and brain control. Compared with QAGKRL, the simulation results show that our algorithm can achieve a faster learning curve, less computational time, and more accuracy. This indicates our algorithm to be a promising method for the online implementation of BMIs.