Reinforcement Learning to determine the optimal number of queues

Question

question

Elax Kot asked Mar 13 2025 at 5:42 PM Elax Kot commented Mar 20 2025 at 9:39 AM

Reinforcement Learning to determine the optimal number of queues

Hello,

I am trying to make a reinforcement learning process in Flexsim. In our system, the processor 1 and 2 is a lot faster than processor 3. When starting processor 1 or 2, the system ensures that at least one queue is empty. If all of the queues are occupied, then the item will be held before processor 1/2. I want to determine if there is a way for me to find out what is the required number of queues. This is a simplification of the process that I am trying to optimize. Preferably, I want to use reinforcement learning.

In addition, since each queue has a specific logic (allowing only certain kinds of items to enter, with a certain "weight limit"), is there a way to duplicate the queues without copying the flexscript over and over again? Thanks!

Software Version:

FlexSim 23.2.0

queue reinforcement learning constraints

1741887559827.png (177.6 KiB)

· 2

______

Cookie preferences

Your privacy is important to us and so is an optimal experience. To help us customize information and build applications, we collect data about your use of this site.

May we collect and use your data?

Learn more about the Third Party Services we use and our Privacy Statement.

Strictly necessary – required for our site to work and to provide services to you

These cookies allow us to record your preferences or login information, respond to your requests or fulfill items in your shopping cart.

YES

Improve your experience – allows us to show you what is relevant to you

These cookies enable us to provide enhanced functionality and personalization. They may be set by us or by third party providers whose services we use to deliver information and experiences tailored to you. If you do not allow these cookies, some or all of these services may not be available for you.

YES

NO

Customize your advertising – permits us to offer targeted advertising to you

These cookies collect data about you based on your activities and interests in order to show you relevant ads and to track effectiveness. By collecting this data, the ads you see will be more tailored to your interests. If you do not allow these cookies, you will experience less targeted advertising.

YES

NO

Are you sure you want a less customized experience?

We can access your data only if you select "yes" for the categories on the previous screen. This lets us tailor our marketing so that it's more relevant for you. You can change your settings at any time by visiting our privacy statement

Your experience. Your choice.

We care about your privacy. The data we collect helps us understand how you use our products, what information you might be interested in, and what we can improve to make your engagement with Autodesk more rewarding.

May we collect and use your data to tailor your experience?

Explore the benefits of a customized experience by managing your privacy settings for this site or visit our Privacy Statement to learn more about your options.

Answer 1 · 2025-03-14T08:05:47Z

Felix Möhlmann answered Mar 14 2025 at 8:05 AM Elax Kot commented Mar 20 2025 at 9:39 AM

You can add (equal) queue to a group and link that group to a model parameter. The option "Delete and Copy Group Members" will create as many copies of the first object in the group as the parameter states.

You can then run an experiment to see how many queues you need (for the upstream processors to never be blocked I assume).

Here's a basic example where the number of processors needed to not get any backlog in the queue is determined that way.

experiment-example.fsm

Another approach would be to not change the number of queues and instead assume an unlimited capacity. By measuring the maximum content you can then calculate how many queues would have been needed to fit all material.

While you say that this is a simplification and I am by no means an expert when it comes to Reinforcement Learning, I don't see how it could be applied here. An RL agent is meant to make decisions during the model run based on the state of the model. Varying the starting conditions is what the experimenter/optimizer is for.

experiment-example.fsm (93.2 KiB)

· 3

Elax Kot commented · Mar 14 2025 at 8:17 AM

Thanks! Using experimenter with range is indeed quite useful and would probably work. An additional constraints that I have would be having to accommodate for the custom code that I have added in the individual queue. Each duplication of the queue will have a slight modifications with regards to the tagging when I use it to interact with a global table that tracks the capacity of the queues.

1. Items that enters a queue will trigger an update to a global table.

2. Items that leaves a queue will also trigger an update.

This is doable when I'm manually copying and editing the custom code in each queue. However, I have no idea how to do it when I'm modifying using the experimenter. Is there a way for me to modify the custom code depending on the queue index?

This would be something that I'm looking to solve as well.

I agree with your point about not using RL in this usecase. It does sound like an overcomplication as opposed to just using experimenter. Many thanks for your response again.

0 ·

Felix Möhlmann Elax Kot commented · Mar 14 2025 at 8:37 AM

If possible, I would make the differences in the code depend on labels on the object. Those can be set in the OnSet code of the parameter.

experiment-example_1.fsm

1 ·

capture1.png (10.0 KiB)

experiment-example-1.fsm (30.1 KiB)

Elax Kot Felix Möhlmann commented · Mar 20 2025 at 9:39 AM

That'll work wonderfully. Thank you so much!

0 ·

question