question

Arthur Ml avatar image
0 Likes"
Arthur Ml asked Arthur Ml commented

Reinforcement Learning: Received action is not used during inference

Hi everyone,

I am following your tutorial for setting up an Reinforcement Learning pipeline with the "ChangeoverTimesRL" model. Everything works fine except that the action sent from the model and received from FlexSim is not in control of the process. The "ItemType" parameter has a constant value of 2. First, I assumed that my model is trained poorly and just gives all the time 2 as output.

Therefore, I added a print in the python code as well as in the "On Request Action - Query a server for a predicted action from a trained model" script to check the action values. The values match and change over time. I.e. communication between the model (python) and the flexsim simulation model is not the problem, the actions are received correctly. Somehow the received action is not passed to ItemType and therefore the RL agent is not in control of the process.

Can someone please help?

Thanks in advance!
Arthur


P.S.
Since I am not able to upload my ChangeoverTimesRL.fsm model here, I put it on google drive:
https://drive.google.com/file/d/1QK0hiBy-tF4BssXmvWCRlhVAWeTuKBzs/view?usp=sharing

FlexSim 22.2.1
reinforcement learning
· 2
5 |100000

Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Jordan Johnson avatar image
2 Likes"
Jordan Johnson answered Arthur Ml commented

This is a bug introduced in version 22.2. The issue is a json parsing issue, where we incorrectly parse single values. You can work around the issue by modifying flexsim_env.py, in the _take_action method:

Original, around line 110:

  1. return state, reward, done
  2. def _take_action(self, action):
  3. actionStr = json.dumps(action, cls=NumpyEncoder)
  4. if self.verbose:
  5. print("Sending Action message: " + actionStr)
  6. actionMessage = "TakeAction:" + actionStr + "?"
  7. self._socket_send(actionMessage.encode())
  8.  
  9.  
  10. def _socket_init(self, host, port):
  11. if self.verbose:

The fix is to add two lines of code, after the def _take_action line, and before the actionStr = json.dumps() line.

Code to insert:

  1. if not hasattr(action, "__len__"):
  2. action = [action]

Fixed code:

  1. return state, reward, done
  2. def _take_action(self, action):
  3. if not hasattr(action, "__len__"):
  4. action = [action]
  5. actionStr = json.dumps(action, cls=NumpyEncoder)
  6. if self.verbose:
  7. print("Sending Action message: " + actionStr)
  8. actionMessage = "TakeAction:" + actionStr + "?"
  9. self._socket_send(actionMessage.encode())
  10.  
  11.  
  12. def _socket_init(self, host, port):
  13. if self.verbose:

NOTE: in python, indentation is critical. The code must be indented as above. Furthermore, be sure to use the same kind of whitespace to indent. If the other lines use spaces to indent, use spaces on the new lines. If the other lines use tabs, then use tabs.

The basic idea of this fix is that, since the bug is that, since single numbers don't parse correctly, you can check if the value is a single value, and if it is, put it in an array. Arrays do parse correctly.

· 1
5 |100000

Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Jeanette F avatar image
0 Likes"
Jeanette F answered Jeanette F edited

Hello @Arthur Ml,

I did the tutorial is FlexSim 22.0 and 22.2.

I am running into the same problem you are in 22.2 but it is working correctly in 22.0. I am sending this in to the Development team as a bug.

· 2
5 |100000

Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.