5-HT 2A and 5-HT 2C receptor antagonism differentially modulate reinforcement learning and cognitive flexibility: behavioural and computational evidence.

RATIONALE: Cognitive flexibility, the ability to adapt behaviour in response to a changing environment, is disrupted in several neuropsychiatric disorders, including obsessive-compulsive disorder and major depressive disorder. Evidence suggests that flexibility, which can be operationalised using reversal learning tasks, is modulated by serotonergic transmission. However, how exactly flexible behaviour and associated reinforcement learning (RL) processes are modulated by 5-HT action on specific receptors is unknown.

OBJECTIVES: We investigated the effects of 5-HT2A receptor (5-HT2A R) and 5-HT2C receptor (5-HT2C R) antagonism on flexibility and underlying RL mechanisms.

METHODS: Thirty-six male Lister hooded rats were trained on a touchscreen visual discrimination and reversal task. We evaluated the effects of systemic treatments with the 5-HT2A R and 5-HT2C R antagonists M100907 and SB-242084, respectively, on reversal learning and performance on probe trials where correct and incorrect stimuli were presented with a third, probabilistically rewarded, stimulus. Computational models were fitted to task choice data to extract RL parameters, including a novel model designed specifically for this task.

RESULTS: 5-HT2A R antagonism impaired reversal learning only after an initial perseverative phase, during a period of random choice and then new learning. 5-HT2C R antagonism, on the other hand, impaired learning from positive feedback. RL models further differentiated these effects. 5-HT2A R antagonism decreased punishment learning rate (i.e. negative feedback) at high and low doses. The low dose also decreased reinforcement sensitivity (beta) and increased stimulus and side stickiness (i.e., the tendency to repeat a choice regardless of outcome). 5-HT2C R antagonism also decreased beta, but reduced side stickiness.

CONCLUSIONS: These data indicate that 5-HT2A and 5-HT2C Rs both modulate different aspects of flexibility, with 5-HT2A Rs modulating learning from negative feedback as measured using RL parameters and 5-HT2C Rs for learning from positive feedback assessed through conventional measures.