How can I delegate AGV tasks using a parameter table

Question

question

Vothan Salomão asked Nov 28 2023 at 9:40 PM Jeanette F commented Dec 06 2023 at 6:40 PM

How can I delegate AGV tasks using a parameter table

I would like to implement reinforcement learning in this simple model. For the agent's observations, I have already managed to collect LastCP, CurrentCP, Destination, and the distance to the destination CP. However, for the action, I am struggling to think of a way to use the job routing template so that the agent can delegate the Destinations of the AGVs. I tried using a custom task, but it was not successful. My idea would be to place a Parameter Table where the Agent would switch the Taskexecuters going to the tasks that appear in the AGV's Work List, but not simply follow a fifo rule

Software Version:

FlexSim 23.2.2

agv reinforcement learning parametertable

exercicio-workforwarding-teste-2.fsm (81.0 KiB)

· 3

Joerg Vogel commented · Nov 29 2023 at 3:50 AM

@Vothan Salomão, what method should a RL algorithm learn? Does this algorithm should decide which vehicle is best suited to do a transport job? I mean this would implement to transfer transport jobs to immediately available taskexecuter or to wait for taskexecuter getting available in a reasonable time. Or does this algorithm must learn which route any chosen taskexecuter should take despite a shortest route is possible, but this could last longer because other transport jobs can delay an arrival to a destination? From what I understood from RL is, that your model and any decision in your model must be controlled by a logic that an algorithm can find a solution for. If routing is done by shortest distance but there doesn’t exist a measured cost equivalent, then which parameter should be evaluated to make a decision on. Do you record a travel time? Do you record a time of completion for a created transport tasksequence? Reinforced Learning must be able to find any kind of correlation in your collected data to find an optimum of input parameters to collected output data.

Next progress step would be to implement an AI to maximize a reward.

0 ·

Vothan Salomão Joerg Vogel commented · Nov 29 2023 at 10:13 AM

I'm considering using reinforcement learning to make decisions in the following model, where there are multiple AGVs and consequently, queues are generated. I'd like to conduct tests on a simpler model before implementing it in a more complex one. Although the shortest route might seem ideal, factors like traffic, potential delays from other transport jobs, and varying speeds across different routes might influence the decision. The algorithm can learn to predict and mitigate potential delays by choosing a route that, while not the shortest, might be the most reliable or fastest in practice. Other inputs such as travel times, completion times for transport tasks, delays, and other logistical data will also be considered.
ReinforcementLearning_AGVsv17_autosave.fsm

0 ·

reinforcementlearning-agvsv17-autosave.fsm (14.4 MiB)

Jeanette F ♦♦ commented · Dec 06 2023 at 6:40 PM

Hi @Vothan Salomão, was Jordan Johnson's answer helpful? If so, please click the "Accept" button at the bottom of their answer. Or if you still have questions, add a comment and we'll continue the conversation.

If we haven't heard back from you within 3 business days we'll auto-accept an answer, but you can always comment back to reopen your question.

0 ·

______

Cookie preferences

Your privacy is important to us and so is an optimal experience. To help us customize information and build applications, we collect data about your use of this site.

May we collect and use your data?

Learn more about the Third Party Services we use and our Privacy Statement.

Strictly necessary – required for our site to work and to provide services to you

These cookies allow us to record your preferences or login information, respond to your requests or fulfill items in your shopping cart.

YES

Improve your experience – allows us to show you what is relevant to you

These cookies enable us to provide enhanced functionality and personalization. They may be set by us or by third party providers whose services we use to deliver information and experiences tailored to you. If you do not allow these cookies, some or all of these services may not be available for you.

YES

NO

Customize your advertising – permits us to offer targeted advertising to you

These cookies collect data about you based on your activities and interests in order to show you relevant ads and to track effectiveness. By collecting this data, the ads you see will be more tailored to your interests. If you do not allow these cookies, you will experience less targeted advertising.

YES

NO

Are you sure you want a less customized experience?

We can access your data only if you select "yes" for the categories on the previous screen. This lets us tailor our marketing so that it's more relevant for you. You can change your settings at any time by visiting our privacy statement

Your experience. Your choice.

We care about your privacy. The data we collect helps us understand how you use our products, what information you might be interested in, and what we can improve to make your engagement with Autodesk more rewarding.

May we collect and use your data to tailor your experience?

Explore the benefits of a customized experience by managing your privacy settings for this site or visit our Privacy Statement to learn more about your options.

Answer 1 · 2023-12-01T16:48:36Z

Jordan Johnson answered Dec 01 2023 at 4:48 PM

The answer depends on your goal.

Goal 1: Trying to Route AGVs

If you are trying to route AGVs in the best possible way for a real world application, I would strongly suggest using an approach like the one shown in this article:

https://answers.flexsim.com/articles/136679/optimized-agv-dispatching-with-google-or-tools.html

The Vehicle Routing Problem is common enough that there are many existing solutions. In the linked article, the model uses OR Tools from Google. Those tools let you specify whether you want to minimize wait time for the item, travel distance for the AGVs, or other goals.

So why use a solver instead of AI? In a way, a solver is just a slower AI agent that you don't have to train. A true AI agent will only take ~100ms or so to return an action given a state. A solver might take several seconds for large problems. So yes, it's slower, but you don't have to train an agent. Training is a difficult task that can be time consuming and computationally expensive. In addition, a trained agent might still make poor choices every now and then, where a solver will always give a good choice.

So in this case, since there is a solver designed for your exact problem, I would use that. I would only use Reinforcement Learning where no known solution exists, or where decisions in real life must be made in less than a second, and known methods are slower.

Goal 2: Exploring Reinforcement Learning with an Example

If you are just trying to learn about Reinforcement Learning and you have chosen AGV routing as an example, that is fine too. In fact, since a solver exists for your situation, you could compare the performance of AI vs the solver.

So how to do it? For your action space, I'd say start with an Option parameter with one option per AGV. Whenever you request a decision, the agent should choose which AGV to use.

The observation space is more tricky. You'd probably want to observe various distances to each AGV and also an estimated time that the AGV will be available, or something like that. If there are many AGVs, this would lead to many, many observations. Then you have to come up with a reward function. This is tricky, too, as it's really easy to reward invalid behavior accidentally. Observation space and reward shaping is a difficult task, and there is no formula or method for it. You have to think through for yourself what observations might be helpful and then try to create and agent. If it fails to train, or learns improper behavior, you will have to update your observation space or reward function. This process is very iterative and can take a long, long time to produce a helpful agent.

question