Setting a parameter when resetting the environment in Reinforcement Learning

Question

question

Arthur Ml asked Mar 16, '23 Jeanette F commented Mar 25, '23

Setting a parameter when resetting the environment in Reinforcement Learning

Hi everyone,

I am doing Meta Reinforcement Learning for production scheduling. Therefore, I predefined several job lists (n=50) that should be scheduled optimally with RL. In MetaRL, the algorithm regards each list as "task" and the algorithm should be able to select a specific task (i.e. 1 list of the 50 lists) when resetting the environment. I want realize that, by changing the value of a global parameter, which then leads to the usage of the specific list. I.e.: episode is finished -> MetaRL selects. e.g. task=14 for the next episode -> parameter is set to 14 -> flexsim is resetted and job list 14 is used in the model now.

Currently, my code for resetting flexsim looks like this:

def _reset_flexsim(self):
    if self.verbose:
        print("Sending Reset message")
    resetString = "Reset?"
    if hasattr(self, "seedNum"):
        resetString = "Reset:" + str(self.seedNum) + "?"
    self._socket_send(resetString.encode())

Can anyone help me please how I can set the parameter value when resetting flexsim?

Thanks in advance!

Software Version:

FlexSim 22.2.1

reinforcement learning

· 1

Jeanette F ♦♦ commented · Mar 25, 2023 at 05:28 AM

Hi @Arthur Ml, was Kavika F's answer helpful? If so, please click the "Accept" button at the bottom of their answer. Or if you still have questions, add a comment and we'll continue the conversation.

If we haven't heard back from you within 3 business days we'll auto-accept an answer, but you can always unaccept and comment back to reopen your question.

0 ·

Answer 1 · 2023-03-16T21:48:57Z

Kavika F answered Mar 16, '23 Arthur Ml commented Mar 21, '23

Hey @Arthur Ml, one way you can set the parameter's value on model reset is using the "OnModelReset" trigger found in the Toolbox:

This gives you a code editor that will execute when the model is about to Reset. Here you can grab a reference to the Parameter's value cell and set it with the task for the next run. In my example I simply set it with the output count of a Processor.

set-param-on-reset.fsm

Hope this helps!

1679003018177.png (51.2 KiB)

change-param-on-reset.gif (87.0 KiB)

set-param-on-reset.fsm (150.8 KiB)

· 5

Arthur Ml commented · Mar 17, 2023 at 07:29 AM

Dear @Kavika F ,

thanks for the quick response.

Sorry, my question was not specific enough. How can I set a parameter value from Python, when resetting the model/ RL environment? I use a socket connection for connecting Python with Flexsim as presented in the documentation (https://docs.flexsim.com/en/22.1/ModelLogic/ReinforcementLearning/KeyConcepts/KeyConcepts.html). So far, with this socket connection, I can launch, reset, and close Flexsim, and I can send actions and receive observation from the model. But how can I set a specific parameter value when resetting the model?

Thanks and best regards,
Arthur

0 ·

Kavika F ♦ Arthur Ml commented · Mar 18, 2023 at 12:02 AM

Oh okay. Once you reset your model, you can use the following command to set a parameter:

controller.setParameter("ParameterName", value)

If you look at the FlexSimPy.cpp, you can see a list of commands you can use from the controller (and within the FlexSim):

0 ·

1679097754270.png (196.9 KiB)

Arthur Ml Kavika F ♦ commented · Mar 20, 2023 at 06:48 AM

Is there a way to set a Parameter in Flexsim with the help of the "flexsim_env.py" script you provided in the documentation (https://docs.flexsim.com/en/22.1/ModelLogic/ReinforcementLearning/KeyConcepts/KeyConcepts.html)? Because I am using the scripts from the documentation to connect to Flexsim and do the training.

0 ·

Show more comments

question