exploration exploitation tradeoff

Despite decades of experimental and clinical practice, the neuropsychological mechanisms underlying neurofeedback (NF) training remain obscure. NF is a unique form of reinforcement learning (RL) task, during which participants are provided with rewarding feedback regarding desired changes in neural patterns. However, key RL considerations - including choices during practice, prediction errors, credit-assignment problems, or the exploration-exploitation tradeoff - have infrequently been considered in the context of NF...

Flexible decision-making requires animals to forego immediate rewards (exploitation) and try novel choice options (exploration) to discover if they are preferable to familiar alternatives. Using the same task and a partially observable Markov decision process (POMDP) model to quantify the value of choices, we first determined that the computational basis for managing explore-exploit tradeoffs is conserved across monkeys and humans. We then used fMRI to identify where in the human brain the immediate value of exploitative choices and relative uncertainty about the value of exploratory choices were encoded...

Health monitoring is a prominent factor in a person's daily life. Healthcare for the elderly is becoming increasingly important as the population ages and grows. The health of an Elderly patient needs frequent examination because the health deteriorates with an increasing age profile. IoT is utilized everywhere in the health industry to identify and communicate with the patients by the professional. A cyber-physical system (CPS) is used to combine physical processes with communication and computation. CPS and IoT are both wirelessly connected via information and communication technologies...

Small hydropower (SHP) possesses significant economic, technical, and environmental advantages, and accounts for a large proportion of hydropower development in China. However, the concentrated, cascaded, and diversion-type development of SHP has resulted in long-distance dewatering of river sections, and inter-basin water transfers have led to severe exploitation of water resources and damage to river ecosystems. In this paper, the Datong River Basin, a secondary sub-basin of the Yellow River Basin in China, was selected as the illustrative case, which includes 22 hydropower projects (HPPs) and three inter-basin water diversion projects (WDPs)...

Sustainable human exploitation of living marine resources stems from a delicate balance between yield stability and population persistence to achieve socioeconomic and conservation goals. But our imperfect knowledge of how oceanic oscillations regulate temporal variation in an exploited species can obscure the risk of missing management targets. We illustrate how applying a management policy to suppress fluctuations in fishery yield in variable environments (prey density and regional climate) can present unintended outcomes in harvested predators and the sustainability of harvesting...

Contextual bandit is a popular sequential decision-making framework to balance the exploration and exploitation tradeoff in many applications such as recommender systems, search engines, etc. Motivated by two important factors in real-world applications: 1) latent contexts (or features) often exist and 2) feedbacks often have humans in the loop leading to human biases, we formulate a generalized contextual bandit framework with latent contexts. Our proposed framework includes a two-layer probabilistic interpretable model for the feedbacks from human with latent features...

Model-free reinforcement learning algorithms based on entropy regularized have achieved good performance in control tasks. Those algorithms consider using the entropy-regularized term for the policy to learn a stochastic policy. This work provides a new perspective that aims to explicitly learn a representation of intrinsic information in state transition to obtain a multimodal stochastic policy, for dealing with the tradeoff between exploration and exploitation. We study a class of Markov decision processes (MDPs) with divergence maximization, called divergence MDPs...

Obligate insect social parasites evolve traits to effectively locate and then exploit their hosts, whereas hosts have complex social behavioral repertoires, which include sensory recognition to reject potential conspecific intruders and heterospecific parasites. While social parasites and host behaviors have been studied extensively, less is known about how their sensory systems function to meet their specific selective pressures. Here, we compare investment in visual and olfactory brain regions in the paper wasp Polistes dominula, and its obligate social parasite P...

Even as vaccination for severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) expands in the United States, cases will linger among unvaccinated individuals for at least the next year, allowing the spread of the coronavirus to continue in communities across the country. Detecting these infections, particularly asymptomatic ones, is critical to stemming further transmission of the virus in the months ahead. This will require active surveillance efforts in which these undetected cases are proactively sought out rather than waiting for individuals to present to testing sites for diagnosis...

Automatic and accurate detection of anatomical landmarks is an essential operation in medical image analysis with a multitude of applications. Recent deep learning methods have improved results by directly encoding the appearance of the captured anatomy with the likelihood maps (i.e., heatmaps). However, most current solutions overlook another essence of heatmap regression, the objective metric for regressing target heatmaps and rely on hand-crafted heuristics to set the target precision, thus being usually cumbersome and task-specific...

The balance between exploration and exploitation is critical to the performance of a Meta-heuristic optimization method. At different stages, a proper tradeoff between exploration and exploitation can drive the search process towards better performance. This paper develops a multi-objective grasshopper optimization algorithm (MOGOA) with a new proposed framework called the Multi-group and Co-evolution Framework which can archive a fine balance between exploration and exploitation. For the purpose, a grouping mechanism and a co-evolution mechanism are designed and integrated into the framework for ameliorating the convergence and the diversity of multi-objective optimization solutions and keeping the exploration and exploitation of swarm intelligence algorithm in balance...

Finding the underlying principles of social attention in humans seems to be essential for the design of the interaction between natural and artificial agents. Here, we focus on the computational modeling of gaze dynamics as exhibited by humans when perceiving socially relevant multimodal information. The audio-visual landscape of social interactions is distilled into a number of multimodal patches that convey different social value, and we work under the general frame of foraging as a tradeoff between local patch exploitation and landscape exploration...

Growing evidence suggests that behavioral variability plays a critical role in how humans manage the tradeoff between exploration and exploitation. In these decisions a little variability can help us to overcome the desire to exploit known rewards by encouraging us to randomly explore something else. Here we investigate how such 'random exploration' could be controlled using a drift-diffusion model of the explore-exploit choice. In this model, variability is controlled by either the signal-to-noise ratio with which reward is encoded (the 'drift rate'), or the amount of information required before a decision is made (the 'threshold')...

In everyday life, risky decision-making relies on multiple cognitive processes including sensitivity to reinforcers, exploration, learning, and forgetting. Neuroimaging evidence suggests that the dorsolateral prefrontal cortex (DLPFC) is involved in exploration and risky decision-making, but the nature of its computations and its causal role remain uncertain. We provide evidence for the role of the DLPFC in value-independent, directed exploration on the Iowa Gambling Task (IGT) and we describe a new computational model to account for the competition of directed exploration and exploitation in guiding decisions...

Adversarial attack can be deemed as a necessary prerequisite evaluation procedure before the deployment of any reinforcement learning (RL) policy. Most existing approaches for generating adversarial attacks are gradient based and are extensive, viz., perturbing every pixel of every frame. In contrast, recent advances show that gradient-free selective perturbations (i.e., attacking only selected pixels and frames) could be a more realistic adversary. However, these attacks treat every frame in isolation, ignoring the relationship between neighboring states of a Markov decision process; thus resulting in high computational complexity that tends to limit their real-world plausibility due to the tight time constraint in RL...

As humanity grapples with the concept of autonomy for human-machine teams (A-HMTs), unresolved is the necessity for the control of autonomy that instills trust. For non-autonomous systems in states with a high degree of certainty, rational approaches exist to solve, model or control stable interactions; e.g., game theory, scale-free network theory, multi-agent systems, drone swarms. As an example, guided by artificial intelligence (AI, including machine learning, ML) or by human operators, swarms of drones have made spectacular gains in applications too numerous to list (e...

No abstract text is available yet for this article.

Reinforcement learning (RL) refers to the behavioral process of learning to obtain reward and avoid punishment. An important component of RL is managing explore-exploit tradeoffs, which refers to the problem of choosing between exploiting options with known values and exploring unfamiliar options. We examined correlates of this tradeoff, as well as other RL related variables, in orbitofrontal cortex (OFC) while three male monkeys performed a three-armed bandit learning task. During the task, novel choice options periodically replaced familiar options...

We study a minimal model for the growth of a phenotypically heterogeneous population of cells subject to a fluctuating environment in which they can replicate (by exploiting available resources) and modify their phenotype within a given landscape (thereby exploring novel configurations). The model displays an exploration-exploitation trade-off whose specifics depend on the statistics of the environment. Most notably, the phenotypic distribution corresponding to maximum population fitness (i.e., growth rate) requires a nonzero exploration rate when the magnitude of environmental fluctuations changes randomly over time, while a purely exploitative strategy turns out to be optimal in two-state environments, independently of the statistics of switching times...

Over the last few decades, the decomposition-based multiobjective evolutionary algorithms (DMOEAs) have became one of the mainstreams for multiobjective optimization. However, there is not too much research on applying DMOEAs to uncertain problems until now. Usually, the uncertainty is modeled as additive noise in the objective space, which is the case this paper concentrates on. This paper first carries out experiments to examine the impact of noisy environments on DMOEAs. Then, four noise-handling techniques based upon the analyses of empirical results are proposed...