How can I delegate AGV tasks using a parameter table

Question

question

Vothan Salomão asked Nov 28, '23 Jeanette F commented Dec 6, '23

How can I delegate AGV tasks using a parameter table

I would like to implement reinforcement learning in this simple model. For the agent's observations, I have already managed to collect LastCP, CurrentCP, Destination, and the distance to the destination CP. However, for the action, I am struggling to think of a way to use the job routing template so that the agent can delegate the Destinations of the AGVs. I tried using a custom task, but it was not successful. My idea would be to place a Parameter Table where the Agent would switch the Taskexecuters going to the tasks that appear in the AGV's Work List, but not simply follow a fifo rule

Software Version:

FlexSim 23.2.2

agv reinforcement learning parametertable

exercicio-workforwarding-teste-2.fsm (81.0 KiB)

· 3

Joerg Vogel commented · Nov 29, 2023 at 03:50 AM

@Vothan Salomão, what method should a RL algorithm learn? Does this algorithm should decide which vehicle is best suited to do a transport job? I mean this would implement to transfer transport jobs to immediately available taskexecuter or to wait for taskexecuter getting available in a reasonable time. Or does this algorithm must learn which route any chosen taskexecuter should take despite a shortest route is possible, but this could last longer because other transport jobs can delay an arrival to a destination? From what I understood from RL is, that your model and any decision in your model must be controlled by a logic that an algorithm can find a solution for. If routing is done by shortest distance but there doesn’t exist a measured cost equivalent, then which parameter should be evaluated to make a decision on. Do you record a travel time? Do you record a time of completion for a created transport tasksequence? Reinforced Learning must be able to find any kind of correlation in your collected data to find an optimum of input parameters to collected output data.

Next progress step would be to implement an AI to maximize a reward.

0 ·

Vothan Salomão Joerg Vogel commented · Nov 29, 2023 at 10:13 AM

I'm considering using reinforcement learning to make decisions in the following model, where there are multiple AGVs and consequently, queues are generated. I'd like to conduct tests on a simpler model before implementing it in a more complex one. Although the shortest route might seem ideal, factors like traffic, potential delays from other transport jobs, and varying speeds across different routes might influence the decision. The algorithm can learn to predict and mitigate potential delays by choosing a route that, while not the shortest, might be the most reliable or fastest in practice. Other inputs such as travel times, completion times for transport tasks, delays, and other logistical data will also be considered.
ReinforcementLearning_AGVsv17_autosave.fsm

0 ·

reinforcementlearning-agvsv17-autosave.fsm (14.4 MiB)

Jeanette F ♦♦ commented · Dec 06, 2023 at 06:40 PM

Hi @Vothan Salomão, was Jordan Johnson's answer helpful? If so, please click the "Accept" button at the bottom of their answer. Or if you still have questions, add a comment and we'll continue the conversation.

If we haven't heard back from you within 3 business days we'll auto-accept an answer, but you can always comment back to reopen your question.

0 ·

Answer 1 · 2023-12-01T16:48:36Z

Jordan Johnson answered Dec 1, '23

The answer depends on your goal.

Goal 1: Trying to Route AGVs

If you are trying to route AGVs in the best possible way for a real world application, I would strongly suggest using an approach like the one shown in this article:

https://answers.flexsim.com/articles/136679/optimized-agv-dispatching-with-google-or-tools.html

The Vehicle Routing Problem is common enough that there are many existing solutions. In the linked article, the model uses OR Tools from Google. Those tools let you specify whether you want to minimize wait time for the item, travel distance for the AGVs, or other goals.

So why use a solver instead of AI? In a way, a solver is just a slower AI agent that you don't have to train. A true AI agent will only take ~100ms or so to return an action given a state. A solver might take several seconds for large problems. So yes, it's slower, but you don't have to train an agent. Training is a difficult task that can be time consuming and computationally expensive. In addition, a trained agent might still make poor choices every now and then, where a solver will always give a good choice.

So in this case, since there is a solver designed for your exact problem, I would use that. I would only use Reinforcement Learning where no known solution exists, or where decisions in real life must be made in less than a second, and known methods are slower.

Goal 2: Exploring Reinforcement Learning with an Example

If you are just trying to learn about Reinforcement Learning and you have chosen AGV routing as an example, that is fine too. In fact, since a solver exists for your situation, you could compare the performance of AI vs the solver.

So how to do it? For your action space, I'd say start with an Option parameter with one option per AGV. Whenever you request a decision, the agent should choose which AGV to use.

The observation space is more tricky. You'd probably want to observe various distances to each AGV and also an estimated time that the AGV will be available, or something like that. If there are many AGVs, this would lead to many, many observations. Then you have to come up with a reward function. This is tricky, too, as it's really easy to reward invalid behavior accidentally. Observation space and reward shaping is a difficult task, and there is no formula or method for it. You have to think through for yourself what observations might be helpful and then try to create and agent. If it fails to train, or learns improper behavior, you will have to update your observation space or reward function. This process is very iterative and can take a long, long time to produce a helpful agent.

question