question

Maryam H2 avatar image
0 Likes"
Maryam H2 asked Maryam H2 edited

RL: action space

Hello -

If my observation space consists of the contents of several queues (multi-discrete) in the RL tool, is it correct to set the observation parameters as follows?

Model.parameters["Queue1_Content"].value = Model.find("Queue1").subnodes.length;
Model.parameters["Queue2_Content"].value = Model.find("Queue2").subnodes.length;
..

The action space is updating the content coming onto the queue, so I'm wondering what should be in Decision Events (Event 1 for Queue 1, Event 2 for Queue 2, etc.). Is "On Content Change" for the object Queue correct?

@Felix Möhlmann can you take a look at this one?

FlexSim 24.1.0
reinforcement learningobservationdecision event
· 3
5 |100000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Felix Möhlmann avatar image Felix Möhlmann commented ·

I'm not really understanding what role the RL agent is supposed to fill, so I can only give general advice.

A decision event, as the name suggests, should fire when a decision from the RL agent is required on how to proceed. Meaning before something happens in the model that depends on one or multiple action parameters.


You can set the observations parameters like this in the "On Observation" trigger, or you continually update them while the model is running. The latter would be slightly faster if the content changes less often than a decision is made.

0 Likes 0 ·
Maryam H2 avatar image Maryam H2 Felix Möhlmann commented ·

Hi @Felix Möhlmann @Logan Gold

The role of the RL agent is to configure the quantity of items allocated to multiple queues (see #1 below) by observing several factors, such as the stay time on the inspection queues (2) and the stay time or quantity of items in racks (3).

The decision on the quantity of items (how many to order?) is made once per day at the start of each shift. Given this setup, it is still feasible to continually update the observation parameters while the model is running to ensure that the most up-to-date information is available for the next decision point.

Also, my decision space is multi-discrete: there are four queues for four types of items, each with a value range between a minimum and maximum. However,when running flexsim_env.py, I am facing the following error: 

ValueError: too many values to unpack (expected 4).

Do you know how to resolve this?


1725552169611.png

0 Likes 0 ·
1725552169611.png (375.8 KiB)
Logan Gold avatar image Logan Gold ♦♦ commented ·

Hi @Maryam H2,

We haven't heard back from you. Were you able to solve your problem? If so, please add and accept an answer to let others know the solution. Or please respond to the previous comment so that we can continue to help you.

If we don't hear back in the next 3 business days, we'll assume you were able to solve your problem and we'll close this case in our tracker. You can always comment back at any time to reopen your question, or you can contact your local FlexSim distributor for phone or email help.

0 Likes 0 ·

0 Answers