model based model free reinforcement learning

We focus on learning the zero-constraint-violation safe policy in model-free reinforcement learning (RL). Existing model-free RL studies mostly use the posterior penalty to penalize dangerous actions, which means they must experience the danger to learn from the danger. Therefore, they cannot learn a zero-violation safe policy even after convergence. To handle this problem, we leverage the safety-oriented energy functions to learn zero-constraint-violation safe policies and propose the safe set actor-critic (SSAC) algorithm...

BACKGROUND: Quantification of aortic morphology plays an important role in the evaluation and follow-up assessment of patients with aortic diseases, but often requires labor-intensive and operator-dependent measurements. Automatic solutions would help enhance their quality and reproducibility. PURPOSE: To design a deep learning (DL)-based automated approach for aortic landmarks and lumen detection derived from three-dimensional (3D) MRI. STUDY TYPE: Retrospective...

In the field of quadruped robots, the most classic motion control algorithm is based on model prediction control (MPC). However, this method poses challenges as it necessitates the precise construction of the robot's dynamics model, making it difficult to achieve agile movements similar to those of a biological dog. Due to these limitations, researchers are increasingly turning to model-free learning methods, which significantly reduce the difficulty of modeling and engineering debugging and simultaneously reduce real-time optimization computational burden...

For completely unknown affine nonlinear systems, in this article, a synergetic learning algorithm (SLA) is deve-loped to learn an optimal control. Unlike the conventional Hamilton-Jacobi-Bellman equation (HJBE) with system dynamics, a model-free HJBE (MF-HJBE) is deduced by means of off-policy reinforcement learning (RL). Specifically, the equivalence between HJBE and MF-HJBE is first bridged from the perspective of the uniqueness of the solution of the HJBE. Furthermore, it is proven that once the solution of MF-HJBE exists, its corresponding control input renders the system asymptotically stable and optimizes the cost function...

Wireless communications systems are traditionally designed by independently optimising signal processing functions based on a mathematical model. Deep learning-enabled communications have demonstrated end-to-end design by jointly optimising all components with respect to the communications environment. In the end-to-end approach, an assumed channel model is necessary to support training of the transmitter and receiver. This limitation has motivated recent work on over-the-air training to explore disjoint training for the transmitter and receiver without an assumed channel...

Freeform nanostructures have the potential to support complex resonances and their interactions, which are crucial for achieving desired spectral responses. However, the design optimization of such structures is nontrivial and computationally intensive. Furthermore, the current "black box" design approaches for freeform nanostructures often neglect the underlying physics. Here, a hybrid data-efficient neural optimizer for resonant nanostructures by combining a reinforcement learning algorithm and Powell's local optimization technique is presented...

Research on predictive processing models has focused largely on two specific algorithmic theories: Predictive Coding for perception and Active Inference for decision-making. While these interconnected theories possess broad explanatory potential, they have only recently begun to receive direct empirical evaluation. Here, we review recent studies of Predictive Coding and Active Inference with a focus on evaluating the degree to which they are empirically supported. For Predictive Coding, we find that existing empirical evidence offers modest support...

A recently introduced bendable concrete having hundred times greater strain capacity provides promising results in repair of engineering structures, known as strain hardening cementitious composites (SHHCs). The current research creates new empirical prediction models to assess the mechanical properties of strain-hardening cementitious composites (SHCCs) i.e., compressive strength (CS), first crack tensile stress (TS), and first crack flexural stress (FS), using gene expression programming (GEP). Wide-ranging records were considered with twelve variables i...

Contemporary structure-based molecular generative methods have demonstrated their potential to model the geometric and energetic complementarity between ligands and receptors, thereby facilitating the design of molecules with favorable binding affinity and target specificity. Despite the introduction of deep generative models for molecular generation, the atom-wise generation paradigm that partially contradicts chemical intuition limits the validity and synthetic accessibility of the generated molecules. Additionally, the dependence of deep learning models on large-scale structural data has hindered their adaptability across different targets...

BACKGROUND AND OBJECTIVE: Advancement in the treatment of cancer, as a leading cause of death worldwide, has promoted several research activities in various related fields. The development of effective treatment regimens with optimal drug dose administration using a mathematical modeling framework has received extensive research attention during the last decades. However, most of the control techniques presented for cancer chemotherapy are mainly model-based approaches. The available model-free techniques based on Reinforcement Learning (RL), commonly discretize the problem states and variables, which other than demanding expert supervision, cannot model the real-world conditions accurately...

Realizing the full potential of quantum technologies requires precise real-time control on time scales much shorter than the coherence time. Model-free reinforcement learning promises to discover efficient feedback strategies from scratch without relying on a description of the quantum system. However, developing and training a reinforcement learning agent able to operate in real-time using feedback has been an open challenge. Here, we have implemented such an agent for a single qubit as a sub-microsecond-latency neural network on a field-programmable gate array (FPGA)...

This article proposes a data-efficient model-free reinforcement learning (RL) algorithm using Koopman operators for complex nonlinear systems. A high-dimensional data-driven optimal control of the nonlinear system is developed by lifting it into the linear system model. We use a data-driven model-based RL framework to derive an off-policy Bellman equation. Building upon this equation, we deduce the data-efficient RL algorithm, which does not need a Koopman-built linear system model. This algorithm preserves dynamic information while reducing the required data for optimal control learning...

With the explosive growth of micro-video applications, the transmission burden of fronthaul and backhaul links is increasing, and meanwhile, a lot of energy consumption is also generated. For reducing energy consumption and transmission delay burden, we propose a cell-free massive multiple-input multiple-output (CF-mMIMO) system in which the cache on the access point (AP) is used to reduce the load on the link. In this paper, a total energy efficiency (EE) model of a cache-assisted CF-mMIMO system is established...

Text generative models trained via maximum likelihood estimation (MLE) suffer from the notorious exposure bias problem, and generative adversarial networks (GANs) are shown to have potential to tackle this problem. The existing language GANs adopt estimators, such as REINFORCE or continuous relaxations to model word probabilities. The inherent limitations of such estimators lead current models to rely on pretraining techniques (MLE pretraining or pretrained embeddings). Representation modeling methods (RMMs), which are free from those limitations, however, are seldomly explored because of their poor performance in previous attempts...

PURPOSE OF REVIEW: Studies increasingly show the importance of reward processing in binge eating and provide evidence of associated changes in the neurobiological reward system. This review gives an up-to-date overview of the neurobiological substrates of reward processing subconstructs in binge eating. Neural findings are linked to different behavioral theories and the clinical relevance is discussed. RECENT FINDINGS: Increased neural responses in the orbitofrontal cortex, anterior cingulate cortex as well as striatum during anticipation and receipt of food rewards are found in association to binge eating...

For adaptive real-time behavior in real-world contexts, the brain needs to allow past information over multiple timescales to influence current processing for making choices that create the best outcome as a person goes about making choices in their everyday life. The neuroeconomics literature on value-based decision-making has formalized such choice through reinforcement learning models for two extreme strategies. These strategies are model-free (MF), which is an automatic, stimulus-response type of action, and model-based (MB), which bases choice on cognitive representations of the world and causal inference on environment-behavior structure...

Human behaviour may be thought of as supported by two different computational-learning mechanisms, model-free and model-based respectively. In model-free strategies, stimulus-response associations are strengthened when actions are followed by a reward and weakened otherwise. In model-based learning, previous to selecting an action, the current values of the different possible actions are computed based on a detailed model of the environment. Previous research with the two-stage task suggests that participants' behaviour usually shows a mixture of both strategies...

Magnetically manipulated medical robots are a promising alternative to current robotic platforms, allowing for miniaturization and tetherless actuation. Controlling such systems autonomously may enable safe, accurate operation. However, classical control methods require rigorous models of magnetic fields, robot dynamics, and robot environments, which can be difficult to generate. Model-free reinforcement learning (RL) offers an alternative that can bypass these requirements. We apply RL to a robotic magnetic needle manipulation system...

A quantum thermal machine is an open quantum system that enables the conversion between heat and work at the micro or nano-scale. Optimally controlling such out-of-equilibrium systems is a crucial yet challenging task with applications to quantum technologies and devices. We introduce a general model-free framework based on reinforcement learning to identify out-of-equilibrium thermodynamic cycles that are Pareto optimal tradeoffs between power and efficiency for quantum heat engines and refrigerators. The method does not require any knowledge of the quantum thermal machine, nor of the system model, nor of the quantum state...

Individuals with PTSD often exhibit deficits in executive functioning. An unexplored aspect of neurocognitive functions associated with PTSD is the type of learning system engaged in during decision-making. A model-free (MF) system is habitual in nature and involves trial-and-error learning that is often updated based on the most recent experience (e.g., repeat action if rewarded). A model-based (MB) system is goal-directed in nature and involves the development of an abstract representation of the environment to facilitate decisions (e...