exploration exploitation tradeoff

Decision-making in noisy, changing, and partially observable environments entails a basic tradeoff between immediate reward and longer-term information gain, known as the exploration-exploitation dilemma. Computationally, an effective way to balance this tradeoff is by leveraging uncertainty to guide exploration. Yet, in humans, empirical findings are mixed, from suggesting uncertainty-seeking to indifference and avoidance. In a novel bandit task that better captures uncertainty-driven behavior, we find multiple roles for uncertainty in human choices...

Deciding whether to forego immediate rewards or explore new opportunities is a key component of flexible behavior and is critical for the survival of the species. Although previous studies have shown that different cortical and subcortical areas, including the amygdala and ventral striatum (VS), are implicated in representing the immediate (exploitative) and future (explorative) value of choices, the effect of the motor system used to make choices has not been examined. Here, we tested male rhesus macaques with amygdala or VS lesions on two versions of a three-arm bandit task where choices were registered with either a saccade or an arm movement...

During ecological decisions, such as when foraging for food or selecting a weekend activity, we often have to balance the costs and benefits of exploiting known options versus exploring novel ones. Here, we ask how individuals address such cost-benefit tradeoffs during tasks in which we can either explore by ourselves or seek external advice from an oracle (e.g., a domain expert or recommendation system). To answer this question, we designed two studies in which participants chose between inquiring (at a cost) for expert advice from an oracle, or to search for options without guidance, under manipulations affecting the optimal choice...

Adaptive information-sampling approaches enable efficient selection of mobile robots' waypoints through which the accurate sensing and mapping of a physical process, such as the radiation or field intensity, can be obtained. A key parameter in the informative sampling objective function could be optimized balance the need to explore new information where the uncertainty is very high and to exploit the data sampled so far, with which a great deal of the underlying spatial fields can be obtained, such as the source locations or modalities of the physical process...

This work proposed a new method to optimize the antenna S-parameter using a Golden Sine mechanism-based Honey Badger Algorithm that employs Tent chaos (GST-HBA). The Honey Badger Algorithm (HBA) is a promising optimization method that similar to other metaheuristic algorithms, is prone to premature convergence and lacks diversity in the population. The Honey Badger Algorithm is inspired by the behavior of honey badgers who use their sense of smell and honeyguide birds to move toward the honeycomb. Our proposed approach aims to improve the performance of HBA and enhance the accuracy of the optimization process for antenna S-parameter optimization...

We propose a novel master-slave architecture to solve the top- K combinatorial multiarmed bandits (CMABs) problem with nonlinear bandit feedback and diversity constraints, which, to the best of our knowledge, is the first combinatorial bandits setting considering diversity constraints under bandit feedback. Specifically, to efficiently explore the combinatorial and constrained action space, we introduce six slave models with distinguished merits to generate diversified samples well balancing rewards and constraints as well as efficiency...

Similar decision-making situations often arise repeatedly, presenting tradeoffs between (i) acquiring new information to facilitate future-related decisions (exploration) and (ii) using existing information to secure expected outcomes (exploitation). Exploration choices have been well characterized in nonsocial contexts, however, choices to explore (or not) in social environments are less well understood. Social environments are of particular interest because a key factor that increases exploration in nonsocial contexts is environmental uncertainty, and the social world is generally appreciated to be highly uncertain...

Childhood adversity can have wide-ranging and long-lasting effects on later life. But what are the mechanisms that are responsible for these effects? This article brings together the cognitive science literature on explore-exploit tradeoffs, the empirical literature on early adversity, and the literature in evolutionary biology on 'life history' to explain how early experience influences later life. We propose one potential mechanism: early experiences influence 'hyperparameters' that determine the balance between exploration and exploitation...

Cultures around the world show varying levels of conservatism. While maintaining traditional ideas prevents wrong ones from being embraced, it also slows or prevents adaptation to new times. Without exploration there can be no improvement, but often this effort is wasted as it fails to produce better results, making it better to exploit the best known option. This tension is known as the exploration/exploitation issue, and it occurs at the individual and group levels, whenever decisions are made. As such, it is has been investigated across many disciplines...

The basal ganglia (BG) contribute to reinforcement learning (RL) and decision making, but unlike artificial RL agents, it relies on complex circuitry and dynamic dopamine modulaton of opponent striatal pathways to do so. We develop the OpAL* model to assess the normative advantages of this circuitry. In OpAL*, learning induces opponent pathways to differentially emphasize the history of positive or negative outcomes for each action. Dynamic DA modulation then amplifies the pathway most tuned for the task environment...

Recent studies have shown the efficacy of mobile elements in optimizing the energy consumption of sensor nodes. Current data collection approaches for waste management applications focus on exploiting IoT-enabled technologies. However, these techniques are no longer sustainable in the context of smart city (SC) waste management applications due to the emergence of large-scale wireless sensor networks (LS-WSNs) in smart cities with sensor-based big data architectures. This paper proposes an energy-efficient swarm intelligence (SI) Internet of Vehicles (IoV)-based technique for opportunistic data collection and traffic engineering for SC waste management strategies...

Foraging is an essential behavior for animal survival and requires both learning and decision-making skills. However, despite its relevance and ubiquity, there is still no effective mathematical framework to adequately estimate foraging performance that also takes interindividual variability into account. In this work, foraging performance is evaluated in the context of multi-armed bandit (MAB) problems by means of a biological model and a machine learning algorithm. Siamese fighting fish ( Betta splendens ) were used as a biological model and their ability to forage was assessed in a four-arm cross-maze over 21 trials...

The catecholamines dopamine (DA) and norepinephrine (NE) have been repeatedly implicated in neuropsychiatric vulnerability, in part via their roles in mediating the decision making processes. Although the two neuromodulators share a synthesis pathway and are co-activated under states of arousal, they engage in distinct circuits and roles in modulating neural activity across the brain. However, in the computational neuroscience literature, they have been assigned similar roles in modulating the latent cognitive processes of decision making, in particular the exploration-exploitation tradeoff...

Humans and animals have to balance the need for exploring new options with exploiting known options that yield good outcomes. This tradeoff is known as the explore-exploit dilemma. To better understand the neural mechanisms underlying how humans and animals address the explore-exploit dilemma, a good animal behavioral model is critical. Most previous rodents explore-exploit studies used ethologically unrealistic operant boxes and reversal learning paradigms in which the decision to abandon a bad option is confounded by the need for exploring a novel option for information collection, making it difficult to separate different drives and heuristics for exploration...

Balancing the exploration of new options and the exploitation of known options is a fundamental challenge in decision-making, yet the mechanisms involved in this balance are not fully understood. Here, we aimed to elucidate the distinct roles of dopamine and noradrenaline in the exploration-exploitation tradeoff during human choice. To this end, we used a double-blind, placebo-controlled design in which participants received either a placebo, 400 mg of the D2/D3 receptor antagonist amisulpride, or 40 mg of the β-adrenergic receptor antagonist propranolol before they completed a virtual patch-foraging task probing exploration and exploitation...

Simultaneous localization and mapping (SLAM) represents a fundamental problem for autonomous embodied systems, for which the hippocampal/entorhinal system (H/E-S) has been optimized over the course of evolution. We have developed a biologically-inspired SLAM architecture based on latent variable generative modeling within the Free Energy Principle and Active Inference (FEP-AI) framework, which affords flexible navigation and planning in mobile robots. We have primarily focused on attempting to reverse engineer H/E-S "design" properties, but here we consider ways in which SLAM principles from robotics may help us better understand nervous systems and emergent minds...

Features play a critical role in computer vision tasks. Deep learning methods have resulted in significant breakthroughs in the field of object detection, but it is still an extremely challenging obstacle when an object is very small. In this work, we propose a feature-enhancement- and channel-attention-guided single-shot detector called the FCSSD with four modules to improve object detection performance. Specifically, inspired by the structure of atrous convolution, we built an efficient feature-extraction module (EFM) in order to explore contextual information along the spatial dimension, and then pyramidal aggregation module (PAM) is presented to explore the semantic features of deep layers, thus reducing the semantic gap between multi-scale features...

The level of detail (LOD) technique has been widely exploited as a key rendering optimization in many graphics applications. Numerous approaches have been proposed to automatically generate different kinds of LODs, such as geometric LOD or shader LOD. However, none of them have considered simplifying the geometry and shader at the same time. In this paper, we explore the observation that simplifications of geometric and shading details can be combined to provide a greater variety of tradeoffs between performance and quality...

Ancestors of macaques and humans separated into distinct lineages 25 million years ago. Despite this long separation, Hogeveen et al. (2022) show, in this issue of Neuron, that they mediate the explore-exploit tradeoff, which must be managed by any agent adapting to a dynamic environment, using similar computational and neural mechanisms.

Despite decades of effort, no earth-abundant homogeneous catalysts have been discovered that can selectively oxidize methane to methanol. We exploit active learning to simultaneously optimize methane activation and methanol release calculated with machine learning-accelerated density functional theory in a space of 16 M candidate catalysts including novel macrocycles. By constructing macrocycles from fragments inspired by synthesized compounds, we ensure synthetic realism in our computational search. Our large-scale search reveals that low-spin Fe(II) compounds paired with strong-field (e...