AGVs and ReinforcementLearning

Question

question

Vothan Salomão asked Oct 9, '23 Felix Möhlmann commented Oct 10, '23

AGVs and ReinforcementLearning

I'm trying to apply reinforcement learning in my FlexSim model using AGVs. The idea is for them to learn how to assign AGVs to tasks more effectively. I'd like to know if anyone has any tips on what I can do in this case. I've built the model, and I want to use a globalTable as an observation for reinforcement learning, which contains the following information about the AGVs: current control point (Cp), last Cp, destination Cp, and the distance between the current AGV and the destination Cp. However, I noticed that FlexSim's Reinforcement Learning module is not able to read my ParameterTables. I've made various modifications, and as a last resort, I converted everything to integers, but even then, Reinforcement Learning can't read these ParameterTables. I'd like to know if someone could help me, whether it's possible to achieve this learning, and what might be a better approach to this problem. Thank you for your attention.

My model: ReinforcementLearning_AGVsv11.fsm

My GlobalTable :
ReinforcementLearning:

Software Version:

FlexSim 23.2.1

agv reinforcement learning global tables parameters table

1696884951224.png (38.1 KiB)

1696885017399.png (282.3 KiB)

reinforcementlearning-agvsv11.fsm (14.4 MiB)

5 |100000

Attachments: Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Answer 1 · 2023-10-10T06:59:36Z

Felix Möhlmann answered Oct 10, '23 Felix Möhlmann commented Oct 10, '23

The parameter name must be a string (text) value, not a pointer. It also has to be unique across all parameter tables. So you could name them CP_CAR2, LastCP_CAR2, Destination_CAR2 and so on.

· 2

5 |100000

Attachments: Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Vothan Salomão commented · Oct 10, 2023 at 12:02 PM

Hello @Felix Möhlmann , how are you? Even after making these changes, I still can't understand why my parameters table is not showing up in the reinforcement learning module, even when I input integers as requested in the documentation
ReinforcementLearning_AGVsv12.fsm

0 ·

reinforcementlearning-agvsv12.fsm (14.4 MiB)

Felix Möhlmann Vothan Salomão commented · Oct 10, 2023 at 01:55 PM

Thanks, I'm doing fine. I hope so are you.

Sorry that I missed the time I checked your model. Your parameters are set to "Passthrough". The Reinforcement Learning tool requires one of the types listed under observation space.

The boundaries should also be properly set, so the RL algorithm knows how many possible values to expect for that parameter.

This does also mean that you have to copy the table values to the parameters. Depending on what happens more often - the table values changing or the RL algorithm making a decision - you can either update the parameter whenever the table is updated. Or you copy the entire table every time an observation is made (in the On Observation trigger).

That would look something like below.

Table agvTable = Table("AGVs");
for(int row = 1; row <= agvTable.numRows; row++)
{
    string agvName = agvTable[row][1];
    Model.parameters["CP_" + agvName].value = agvTable[row][2];
    Model.parameters["LastCP_" + agvName].value = agvTable[row][3];
    Model.parameters["Destination_" + agvName].value = agvTable[row][4];
    Model.parameters["Distance_" + agvName].value = agvTable[row][5];
}

0 ·

capture1.png (16.0 KiB)

capture2.png (2.4 KiB)

question