Read by QxMD icon Read

model based model free reinforcement learning

Jason M Scimeca, Perri L Katzman, David Badre
Adaptive memory requires context-dependent control over how information is retrieved, evaluated and used to guide action, yet the signals that drive adjustments to memory decisions remain unknown. Here we show that prediction errors (PEs) coded by the striatum support control over memory decisions. Human participants completed a recognition memory test that incorporated biased feedback to influence participants' recognition criterion. Using model-based fMRI, we find that PEs-the deviation between the outcome and expected value of a memory decision-correlate with striatal activity and predict individuals' final criterion...
October 7, 2016: Nature Communications
John P O'Doherty, Jeffrey Cockburn, Wolfgang M Pauli
In this review, we summarize findings supporting the existence of multiple behavioral strategies for controlling reward-related behavior, including a dichotomy between the goal-directed or model-based system and the habitual or model-free system in the domain of instrumental conditioning and a similar dichotomy in the realm of Pavlovian conditioning. We evaluate evidence from neuroscience supporting the existence of at least partly distinct neuronal substrates contributing to the key computations necessary for the function of these different control systems...
September 28, 2016: Annual Review of Psychology
Wouter Kool, Fiery A Cushman, Samuel J Gershman
Many accounts of decision making and reinforcement learning posit the existence of two distinct systems that control choice: a fast, automatic system and a slow, deliberative system. Recent research formalizes this distinction by mapping these systems to "model-free" and "model-based" strategies in reinforcement learning. Model-free strategies are computationally cheap, but sometimes inaccurate, because action values can be accessed by inspecting a look-up table constructed through trial-and-error. In contrast, model-based strategies compute action values through planning in a causal model of the environment, which is more accurate but also more cognitively demanding...
August 2016: PLoS Computational Biology
Arkady Konovalov, Ian Krajbich
Organisms appear to learn and make decisions using different strategies known as model-free and model-based learning; the former is mere reinforcement of previously rewarded actions and the latter is a forward-looking strategy that involves evaluation of action-state transition probabilities. Prior work has used neural data to argue that both model-based and model-free learners implement a value comparison process at trial onset, but model-based learners assign more weight to forward-looking computations. Here using eye-tracking, we report evidence for a different interpretation of prior results: model-based subjects make their choices prior to trial onset...
2016: Nature Communications
Gautam Reddy, Antonio Celani, Terrence J Sejnowski, Massimo Vergassola
Birds and gliders exploit warm, rising atmospheric currents (thermals) to reach heights comparable to low-lying clouds with a reduced expenditure of energy. This strategy of flight (thermal soaring) is frequently used by migratory birds. Soaring provides a remarkable instance of complex decision making in biology and requires a long-term strategy to effectively use the ascending thermals. Furthermore, the problem is technologically relevant to extend the flying range of autonomous gliders. Thermal soaring is commonly observed in the atmospheric convective boundary layer on warm, sunny days...
August 16, 2016: Proceedings of the National Academy of Sciences of the United States of America
Martin V Butz
This paper proposes how various disciplinary theories of cognition may be combined into a unifying, sub-symbolic, computational theory of cognition. The following theories are considered for integration: psychological theories, including the theory of event coding, event segmentation theory, the theory of anticipatory behavioral control, and concept development; artificial intelligence and machine learning theories, including reinforcement learning and generative artificial neural networks; and theories from theoretical and computational neuroscience, including predictive coding and free energy-based inference...
2016: Frontiers in Psychology
Elena Daskalaki, Peter Diem, Stavroula G Mougiakakou
Although reinforcement learning (RL) is suitable for highly uncertain systems, the applicability of this class of algorithms to medical treatment may be limited by the patient variability which dictates individualised tuning for their usually multiple algorithmic parameters. This study explores the feasibility of RL in the framework of artificial pancreas development for type 1 diabetes (T1D). In this approach, an Actor-Critic (AC) learning algorithm is designed and developed for the optimisation of insulin infusion for personalised glucose regulation...
2016: PloS One
Aida Farreny, Ángel Del Rey-Mejías, Gemma Escartin, Judith Usall, Núria Tous, Josep Maria Haro, Susana Ochoa
BACKGROUND: Schizophrenia involves marked motivational and learning deficits that may reflect abnormalities in reward processing. The purpose of this study was to examine positive and negative feedback sensitivity in schizophrenia using computational modeling derived from the Wisconsin Card Sorting Test (WCST). We also aimed to explore feedback sensitivity in a sample with bipolar disorder. METHODS: Eighty-three individuals with schizophrenia and 27 with bipolar disorder were included...
July 2016: Comprehensive Psychiatry
Adam J Culbreth, Andrew Westbrook, Nathaniel D Daw, Matthew Botvinick, Deanna M Barch
Individuals with schizophrenia have a diminished ability to use reward history to adaptively guide behavior. However, tasks traditionally used to assess such deficits often rely on multiple cognitive and neural processes, leaving etiology unresolved. In the current study, we adopted recent computational formalisms of reinforcement learning to distinguish between model-based and model-free decision-making in hopes of specifying mechanisms associated with reinforcement-learning dysfunction in schizophrenia. Under this framework, decision-making is model-free to the extent that it relies solely on prior reward history, and model-based if it relies on prospective information such as motivational state, future consequences, and the likelihood of obtaining various outcomes...
August 2016: Journal of Abnormal Psychology
Johannes H Decker, A Ross Otto, Nathaniel D Daw, Catherine A Hartley
Theoretical models distinguish two decision-making strategies that have been formalized in reinforcement-learning theory. A model-based strategy leverages a cognitive model of potential actions and their consequences to make goal-directed choices, whereas a model-free strategy evaluates actions based solely on their reward history. Research in adults has begun to elucidate the psychological mechanisms and neural substrates underlying these learning processes and factors that influence their relative recruitment...
June 2016: Psychological Science
Matthew Balcarras, Thilo Womelsdorf
Learning in a new environment is influenced by prior learning and experience. Correctly applying a rule that maps a context to stimuli, actions, and outcomes enables faster learning and better outcomes compared to relying on strategies for learning that are ignorant of task structure. However, it is often difficult to know when and how to apply learned rules in new contexts. In our study we explored how subjects employ different strategies for learning the relationship between stimulus features and positive outcomes in a probabilistic task context...
2016: Frontiers in Neuroscience
Simon Dunne, Arun D'Souza, John P O'Doherty
A major open question is whether computational strategies thought to be used during experiential learning, specifically model-based and model-free reinforcement learning, also support observational learning. Furthermore, the question of how observational learning occurs when observers must learn about the value of options from observing outcomes in the absence of choice has not been addressed. In the present study we used a multi-armed bandit task that encouraged human participants to employ both experiential and observational learning while they underwent functional magnetic resonance imaging (fMRI)...
June 1, 2016: Journal of Neurophysiology
Payam Piray, Ivan Toni, Roshan Cools
Two distinct systems, goal-directed and habitual, support decision making. It has recently been hypothesized that this distinction may arise from two computational mechanisms, model-based and model-free reinforcement learning, neuronally implemented in frontostriatal circuits involved in learning and behavioral control. Here, we test whether the relative strength of anatomical connectivity within frontostriatal circuits accounts for variation in human individuals' reliance on model-based and model-free control...
March 9, 2016: Journal of Neuroscience: the Official Journal of the Society for Neuroscience
Miriam Sebold, Daniel J Schad, Stephan Nebe, Maria Garbusow, Elisabeth Jünger, Nils B Kroemer, Norbert Kathmann, Ulrich S Zimmermann, Michael N Smolka, Michael A Rapp, Andreas Heinz, Quentin J M Huys
Behavioral choice can be characterized along two axes. One axis distinguishes reflexive, model-free systems that slowly accumulate values through experience and a model-based system that uses knowledge to reason prospectively. The second axis distinguishes Pavlovian valuation of stimuli from instrumental valuation of actions or stimulus-action pairs. This results in four values and many possible interactions between them, with important consequences for accounts of individual variation. We here explored whether individual variation along one axis was related to individual variation along the other...
July 2016: Journal of Cognitive Neuroscience
Alisabeth Ayars
Dual-systems frameworks propose that moral judgments are derived from both an immediate emotional response, and controlled/rational cognition. Recently Cushman (2013) proposed a new dual-system theory based on model-free and model-based reinforcement learning. Model-free learning attaches values to actions based on their history of reward and punishment, and explains some deontological, non-utilitarian judgments. Model-based learning involves the construction of a causal model of the world and allows for far-sighted planning; this form of learning fits well with utilitarian considerations that seek to maximize certain kinds of outcomes...
May 2016: Cognition
Hilary J Don, Micah B Goldwater, A Ross Otto, Evan J Livesey
Numerous tasks in learning and cognition have demonstrated differences in response patterns that may reflect the operation of two distinct systems. For example, causal and reinforcement learning tasks each show responding that considers abstract structure as well as responding based on simple associations. Nevertheless, there has been little attempt to verify whether these tasks are measuring related processes. The current study therefore investigated the relationship between rule- and feature-based generalization in a causal learning task, and model-based and model-free responding in a reinforcement learning task, including cognitive reflection as a predictor of individual tendencies to use controlled, deliberative processes in these tasks...
February 23, 2016: Psychonomic Bulletin & Review
Bradley B Doll, Kevin G Bath, Nathaniel D Daw, Michael J Frank
UNLABELLED: Considerable evidence suggests that multiple learning systems can drive behavior. Choice can proceed reflexively from previous actions and their associated outcomes, as captured by "model-free" learning algorithms, or flexibly from prospective consideration of outcomes that might occur, as captured by "model-based" learning algorithms. However, differential contributions of dopamine to these systems are poorly understood. Dopamine is widely thought to support model-free learning by modulating plasticity in striatum...
January 27, 2016: Journal of Neuroscience: the Official Journal of the Society for Neuroscience
Judit Zsuga, Klara Biro, Csaba Papp, Gabor Tajti, Rudolf Gesztelyi
Reinforcement learning (RL) is a powerful concept underlying forms of associative learning governed by the use of a scalar reward signal, with learning taking place if expectations are violated. RL may be assessed using model-based and model-free approaches. Model-based reinforcement learning involves the amygdala, the hippocampus, and the orbitofrontal cortex (OFC). The model-free system involves the pedunculopontine-tegmental nucleus (PPTgN), the ventral tegmental area (VTA) and the ventral striatum (VS)...
February 2016: Behavioral Neuroscience
Madeleine E Sharp, Karin Foerde, Nathaniel D Daw, Daphna Shohamy
Patients with loss of dopamine due to Parkinson's disease are impaired at learning from reward. However, it remains unknown precisely which aspect of learning is impaired. In particular, learning from reward, or reinforcement learning, can be driven by two distinct computational processes. One involves habitual stamping-in of stimulus-response associations, hypothesized to arise computationally from 'model-free' learning. The other, 'model-based' learning, involves learning a model of the world that is believed to support goal-directed behaviour...
February 2016: Brain: a Journal of Neurology
Thomas Akam, Rui Costa, Peter Dayan
The recently developed 'two-step' behavioural task promises to differentiate model-based from model-free reinforcement learning, while generating neurophysiologically-friendly decision datasets with parametric variation of decision variables. These desirable features have prompted its widespread adoption. Here, we analyse the interactions between a range of different strategies and the structure of transitions and outcomes in order to examine constraints on what can be learned from behavioural performance. The task involves a trade-off between the need for stochasticity, to allow strategies to be discriminated, and a need for determinism, so that it is worth subjects' investment of effort to exploit the contingencies optimally...
December 2015: PLoS Computational Biology
Fetch more papers »
Fetching more papers... Fetching...
Read by QxMD. Sign in or create an account to discover new knowledge that matter to you.
Remove bar
Read by QxMD icon Read

Search Tips

Use Boolean operators: AND/OR

diabetic AND foot
diabetes OR diabetic

Exclude a word using the 'minus' sign

Virchow -triad

Use Parentheses

water AND (cup OR glass)

Add an asterisk (*) at end of a word to include word stems

Neuro* will search for Neurology, Neuroscientist, Neurological, and so on

Use quotes to search for an exact phrase

"primary prevention of cancer"
(heart or cardiac or cardio*) AND arrest -"American Heart Association"