Closed-loop Rescheduling using Deep Reinforcement Learning

JORGE A. PALOMBARINI; ERNESTO MARTÍNEZ

Florianópolis

Simposio; 12th IFAC Symposium on Dynamics and Control of Process Systems, including Biosystems DYCOPS 2019; 2019

IFAC- International Federation of Automatic Control

Modern socio-technical manufacturing systems are characterized by high levels of variability which gives rise to poor predictability of environmental conditions at the shop-floor. Therefore, a closed-loop rescheduling strategy for handling unforeseen events and unplanned disturbances has become a key element for any real-time disruption management strategy in order to guarantee highly efficient production in uncertain and dynamic conditions. In this work, a real-time rescheduling task is modeled as a closed-loop control problem in which an artificial intelligent agent implements control knowledge generated off-line using a schedule simulator to learn schedule repair policies directly from high-dimensional sensory inputs. The rescheduling control knowledge is stored in a deep Q-network, which is used closed-loop to select repair actions to achieve a small set of repaired goal states. The network is trained using the deep Q-learning algorithm with experience replay over a variety of simulated transitions between schedule states based on color-rich Gantt chart images and negligible prior knowledge as inputs. An industrial example is discussed to highlight that the proposed approach enables end-to-end learning of comprehensive rescheduling policies and encoding plant-specific knowledge that can be understood by human experts.