Manufacturing Scheduling Using Colored Petri Nets and Reinforcement Learning

Maria Drakaki, Panagiotis Tzionas
2017 Applied Sciences  
Agent-based intelligent manufacturing control systems are capable to efficiently respond and adapt to environmental changes. Manufacturing system adaptation and evolution can be addressed with learning mechanisms that increase the intelligence of agents. In this paper a manufacturing scheduling method is presented based on Timed Colored Petri Nets (CTPNs) and reinforcement learning (RL). CTPNs model the manufacturing system and implement the scheduling. In the search for an optimal solution a
more » ... heduling agent uses RL and in particular the Q-learning algorithm. A warehouse order-picking scheduling is presented as a case study to illustrate the method. The proposed scheduling method is compared to existing methods. Simulation and state space results are used to evaluate performance and identify system properties. task is to produce a scheduling action that will lead to minimize (or maximize) the related performance measure. A widely used RL algorithm is Q-learning [6, 7] . Petri Nets (PNs) and Colored Petri Nets (CPNs) are a discrete-event graphical and mathematical modeling tool applicable to systems characterized as being concurrent, asynchronous, distributed, parallel, nondeterministic, and/or stochastic [8, 9] . As such, they have been used extensively for modeling, scheduling, and control of Flexible Manufacturing Systems (FMS). Their main features for this purpose can be described as a powerful modeling ability to describe concurrent, synchronous, conflict, and casual behavior; logic properties (such as boundedness, liveness) and control logic code generated directly from PNs [10]; and the performance evaluation of the system [10, 11] , including the ability to represent many states concisely, as well as to model precedence relations, deadlocks, conflicts, and resource constraints [12, 13] . CPNs can represent parts with attributes as well as temporal activities (Timed Colored Petri Nets, CTPNs). CPN is a discrete-event modeling tool combining PNs with the functional programming language Standard ML [14] . As a graphical-oriented high-level language, it is used for the modeling and validation of systems in which concurrency, communication, and synchronization play a major role. This is why it finds many applications in the area of distributed artificial intelligence (AI) where agents come from. Combinations of PNs-and AI-search based techniques have found applications to manufacturing scheduling, where PNs model the system and a heuristics based search through the reachability graph finds an optimal or near-optimal solution. However, these methods are not as efficient for large, complex manufacturing systems in changing environments. Moreover, CPNs are more suitable for modeling complex manufacturing systems. In this paper a CTPN-based manufacturing scheduling method is presented. CTPNs model an agent-based system and the Q-learning RL algorithm is used by the scheduling agent as a guide to obtain an optimal solution. In order to evaluate the proposed method, it is compared to existing scheduling methods applied to known job shop benchmark examples. A warehouse order-picking scheduling is used as a case study to illustrate the applicability of the method. The rest of the paper is organized as follows. A literature review is given in Section 2. The scheduling method is presented in Section 3. It is illustrated with a case study in Section 4. In Section 5 the performance evaluation and verification of PN-related system properties are given using simulation and state space report results. Conclusions are presented in Section 6.
doi:10.3390/app7020136 fatcat:tuhfgaakojemdnwktx7enozvvq