Error when local server is created for Reinforced Learning

Question

question

Felipe Capalbo asked Dec 1, '23 Jason Lightfoot commented Feb 7, '24

Error when local server is created for Reinforced Learning

I tried to create a reinforcement learning model to optimize a layout by changing the object's position on the X and Y axes. The observation field references the object's position in the tree, as does the action field, but this can change its value.

When I try to implement the trained model, this error shows up:

"Exception occurred during processing of request from ('127.0.0.1', 50682)
Traceback (most recent call last):
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\socketserver.py", line 317, in _handle_request_noblock
    self.process_request(request, client_address)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\socketserver.py", line 348, in process_request
    self.finish_request(request, client_address)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\socketserver.py", line 361, in finish_request
    self.RequestHandlerClass(request, client_address, self)
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\socketserver.py", line 755, in __init__
    self.handle()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\http\server.py", line 436, in handle
    self.handle_one_request()
  File "C:\Program Files\WindowsApps\PythonSoftwareFoundation.Python.3.11_3.11.1776.0_x64__qbz5n2kfra8p0\Lib\http\server.py", line 424, in handle_one_request
    method()
  File "c:\Users\GAIVOTA_FLEXSIM\Documents\FLEXSIM FELIPE CAPALBO\ESTUDOS ML\Exercicio ML Layout\ArquivosPY\flexsim_inference.py", line 11, in do_GET
    self._handle_reply(params)
  File "c:\Users\GAIVOTA_FLEXSIM\Documents\FLEXSIM FELIPE CAPALBO\ESTUDOS ML\Exercicio ML Layout\ArquivosPY\flexsim_inference.py", line 30, in _handle_reply
    action, _states = FlexSimInferenceServer.model.predict(observation)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\GAIVOTA_FLEXSIM\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\stable_baselines3\common\base_class.py", line 553, in predict
    return self.policy.predict(observation, state, episode_start, deterministic)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\GAIVOTA_FLEXSIM\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\stable_baselines3\common\policies.py", line 363, in predict
    obs_tensor, vectorized_env = self.obs_to_tensor(observation)
                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\GAIVOTA_FLEXSIM\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\stable_baselines3\common\policies.py", line 270, in obs_to_tensor
    vectorized_env = is_vectorized_observation(observation, self.observation_space)
                     ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\GAIVOTA_FLEXSIM\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\stable_baselines3\common\utils.py", line 399, in is_vectorized_observation
    return is_vec_obs_func(observation, observation_space)  # type: ignore[operator]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\GAIVOTA_FLEXSIM\AppData\Local\Packages\PythonSoftwareFoundation.Python.3.11_qbz5n2kfra8p0\LocalCache\local-packages\Python311\site-packages\stable_baselines3\common\utils.py", line 307, in is_vectorized_multidiscrete_observation
    raise ValueError(
ValueError: Error: Unexpected observation shape () for MultiDiscrete environment, please use (22,) or (n_env, 22) for the observation shape."

TreinamentoExercicioLayout.zip

Software Version:

FlexSim 23.2.2

reinforcement learning localhost

exerciciolayout.fsm (88.9 KiB)

flexsim-env.py (7.6 KiB)

flexsim-inference.py (2.5 KiB)

flexsim-training.py (1.6 KiB)

treinamentoexerciciolayout.zip (428.9 KiB)

· 16

Kavika F ♦ commented · Dec 01, 2023 at 07:54 PM

Hey @Felipe Capalbo, the error hints that the observation you pass into

action, _states = FlexSimInferenceServer.model.predict(observation)

isn't presented in the right format. The observation space is requesting a (22,) or (n_env, 22) shape. Can you see what kind of shape is getting passed in before the error?

0 ·

Felipe Capalbo Kavika F ♦ commented · Dec 02, 2023 at 05:58 PM

What does this "shape" means? The observations has 22 parameter, as does the actions.

All of them are discrete values.
In the Reinforcement Learning window it is inputed as multidiscrete, because of the 22 parameters.

0 ·

1701539978661.png (15.0 KiB)

1701540086017.png (37.4 KiB)

Kavika F ♦ Felipe Capalbo commented · Dec 04, 2023 at 04:43 PM

@Felipe Capalbo, here's documentation on the numpy.ndarray.shape property. The example in the documentation shows the following:

>>> x = np.array([1, 2, 3, 4])
>>> x.shape
(4,)

With an array size of 4, x.shape returns (4,). This format matches what the above error is looking for - a shape of (22,). What it says you're providing is (), which seems to be an empty array or just no array at all.

I'm able to train your model, but I'm not sure how you've setup the flexsim-interface.py > main() to work. I can't help you further unless you give me detailed steps on how to send input or results from the RL model to the server it sets up.

0 ·

Show more comments

Jason Lightfoot ♦ commented · Dec 14, 2023 at 04:45 PM

Hi @Felipe Capalbo,

We haven't heard back from you. Were you able to solve your problem? If so, please add and accept an answer to let others know the solution. Or please respond to the previous comment so that we can continue to help you.

If we don't hear back in the next 3 business days, we'll assume you were able to solve your problem and we'll close this case in our tracker. You can always comment back at any time to reopen your question, or you can contact your local FlexSim distributor for phone or email help.

0 ·

Felipe Capalbo Jason Lightfoot ♦ commented · Dec 15, 2023 at 12:04 AM

Couldn't figure out yet, but I thing the concept used to define the observations could be wrong.

0 ·

Phil BoBo ♦♦ Felipe Capalbo commented · Dec 15, 2023 at 10:16 PM

The example is training an AI that takes 1 discrete observation value and makes 1 discrete action value. You've changed your model to take multiple values for the observation, not just 1 value.

So passing just a 3 as an observation doesn't make sense. You need to pass 22 numbers.

That step in the tutorial expects that you were following the rest of the tutorial up to that point, not changing it. Since you are doing something different, the tutorial isn't going to match up exactly the same. Just skip that step (which is just testing if your web service is responding to queries) and move on to the next step.

0 ·

Show more comments

Jason Lightfoot ♦ commented · Feb 07 at 02:08 PM

Hi @Felipe Capalbo,

We haven't heard back from you. Were you able to solve your problem? If so, please add and accept an answer to let others know the solution. Or please respond to the previous comment so that we can continue to help you.

If we don't hear back in the next 3 business days, we'll assume you were able to solve your problem and we'll close this case in our tracker. You can always comment back at any time to reopen your question, or you can contact your local FlexSim distributor for phone or email help.

0 ·

question