Closed-loop Rescheduling using Deep Reinforcement Learning

PALOMBARINI, JORGE A.; MARTÍNEZ, ERNESTO C.

IFAC-PapersOnLine

Elsevier B.V.

Lugar: Amsterdam; Año: 2019 vol. 52 p. 231 - 236

2405-8963

Modern socio-technical production and supply-chain systems are characterized by high levels of variability which give rise to poor predictability of environmental conditions at the shop-floor. Therefore, a closed-loop rescheduling strategy for handling unforeseen events and unplanned disturbances has become a key component of any real-time disruption management control system in order to guarantee highly efficient production in increasing dynamic conditions. In this work, a real-time rescheduling task is modeled as a closed-loop control problem in which an artificial agent implements a control policy generated off-line using on a schedule simulator to learn schedule repair policies directly from high-dimensional sensory inputs. The generated rescheduling knowledge is stored as a control policy encoded in a deep Q-network, which is used in a closed-loop strategy to select repair actions in order to achieve a small set of repaired goal states. The network is trained using the deep Q-learning algorithm with experience replay over a variety of simulated transitions between schedule states using color-rich Gantt chart images and negligible prior knowledge as inputs. An industrial example is discussed to highlight that the proposed approach enables end-to-end deep learning of successful rescheduling policies which encode task-specific knowledge that can be understood by human experts.