Issue with observation space in Reinforcement Learning

Question

question

Felix Möhlmann asked Apr 24 2023 at 2:15 PM mark zhen commented Aug 22 2023 at 5:17 PM

Issue with observation space in Reinforcement Learning

I recently installed a new python version. When I tested a modified version of the Reinforcement Learning demo model I encountered a problem when trying to use MultiDiscrete or Box observation/action spaces.

When I try to run the flexsim_training.py I get the following error when using MultiDiscrete as the observation space.

And similarly for a Box space

Something seems to be going wrong with the value conversion. When I set Visual Studio Code to use the old inpreter (3.7.8 instead of 3.11.3) everything works.

Does anyone have an idea what I might have forgotten/done wrong when setting up the new version that could lead to this. (I just installed gym and stable_baselines3 again. Since the Discrete space worked I assumed it was successful)

The python scripts are the unaltered (apart from the file paths) versions available from the link in the tutorial.

RL_Demo_2.fsm

Software Version:

FlexSim 23.1.1

reinforcement learning python observation space

capture1.png (32.4 KiB)

capture2.png (1.5 KiB)

1682345360799.png (8.5 KiB)

rl-demo-2.fsm (67.8 KiB)

______

Cookie preferences

Your privacy is important to us and so is an optimal experience. To help us customize information and build applications, we collect data about your use of this site.

May we collect and use your data?

Learn more about the Third Party Services we use and our Privacy Statement.

Strictly necessary – required for our site to work and to provide services to you

These cookies allow us to record your preferences or login information, respond to your requests or fulfill items in your shopping cart.

YES

Improve your experience – allows us to show you what is relevant to you

These cookies enable us to provide enhanced functionality and personalization. They may be set by us or by third party providers whose services we use to deliver information and experiences tailored to you. If you do not allow these cookies, some or all of these services may not be available for you.

YES

NO

Customize your advertising – permits us to offer targeted advertising to you

These cookies collect data about you based on your activities and interests in order to show you relevant ads and to track effectiveness. By collecting this data, the ads you see will be more tailored to your interests. If you do not allow these cookies, you will experience less targeted advertising.

YES

NO

Are you sure you want a less customized experience?

We can access your data only if you select "yes" for the categories on the previous screen. This lets us tailor our marketing so that it's more relevant for you. You can change your settings at any time by visiting our privacy statement

Your experience. Your choice.

We care about your privacy. The data we collect helps us understand how you use our products, what information you might be interested in, and what we can improve to make your engagement with Autodesk more rewarding.

May we collect and use your data to tailor your experience?

Explore the benefits of a customized experience by managing your privacy settings for this site or visit our Privacy Statement to learn more about your options.

Answer 1 · 2023-05-02T13:49:58Z

Phil BoBo answered May 02 2023 at 1:49 PM Abhishek K commented May 17 2023 at 5:34 PM

The provided python scripts are example code to demonstrate how to communicate with FlexSim from a reinforcement learning algorithm. If they don't work perfectly with a particular version of python or another language, library, package, or implementation, then you should customize them according to the needs of your project. As explained in the first paragraph of Getting Started with OpenAI Gym (flexsim.com).

The observation space and the observation value are both set within the example code. If there's a mismatch, then change the code so that it isn't mismatched.

The function _convert_to_gym_space defined on line 178 creates the observation space. The function _convert_to_observation defined on line 196 returns the observation. (https://github.com/flexsim/FlexSimAI/blob/main/gym/flexsim_env.py)

Based on your error message, in the version of python and gym that you are using, you are creating an int64 observation space and recording an int32 observation. Adjust one or the other so that the observed value's type matches what the specified observation space expects.

Alter the example python script to work if it isn't working. That's why the script is provided.

· 4

Felix Möhlmann commented · May 02 2023 at 2:19 PM

Thank you for the pointers.

In case it might be useful to anyone in the future: Specifying the datatype of the MultiDiscrete space in _convert_to_gym_space of flexsim_env worked for me.

                    return
                     
                    gym
                    .spaces.MultiDiscrete(
                    params
                    , 
                    dtype
                     = 
                    np
                    .
                    int32
                    )
                   

3 ·

Abhishek K Felix Möhlmann commented · May 12 2023 at 10:56 PM

Thanks a lot for posting this, that worked for training the model.

Did you have issues using flexsim_inference.py as well ?
I am getting this error message when running flexsim_inference.py

ValueError: Error: Unexpected observation shape () for MultiDiscrete environment, please use (6,) or (n_env, 6) for the observation shape.

Please let me know if you ran into the same issue and how it could be fixed. I tried editing the python script flexsim_inference.py but couldn't get success and I'm not very familiar with coding in python. Basically the observation space has 6 entries which is the last type, followed by 5 discrete values showing how many items of each type are in the queue (same example problem from tutorial), thus making it a MultiDiscrete one. I'm wondering how the flexsim_inference.py needs to be modified to use that observation space to define action for the processor as to which type it should pull next (action)

1 ·

Felix Möhlmann Abhishek K commented · May 14 2023 at 10:45 AM

No, sorry, the inference script worked without problems for me. My only guess would be that you maybe forgot to adjust the file path and are trying to run an agent that was trained on a different set of observations?

0 ·

Show more comments

Answer 2 · 2023-05-01T17:26:42Z

Jeanette F answered May 01 2023 at 5:26 PM mark zhen commented Aug 22 2023 at 5:17 PM

Hello @Felix Möhlmann,

FlexSim does not support python version 3.11. You can try and build for a different python version as directed below. You can find this information in the manual as well.

1682960820395.png (90.3 KiB)

· 3

Felix Möhlmann commented · May 02 2023 at 6:26 AM

Thank you for your answer. I did forget to set the version in the preferences and was not aware that Python 3.11 is not supported.

However I still get the same error message when using version 3.10. Do you have an idea what else I might try to get it working?

0 ·

1683008732957.png (9.6 KiB)

1683008743386.png (6.5 KiB)

1683008753530.png (7.1 KiB)

mark zhen Felix Möhlmann commented · Aug 22 2023 at 5:17 PM

Have you considered downgrading the stable baseline version

0 ·

Phil BoBo ♦♦ commented · May 02 2023 at 1:53 PM

@Jeanette F The Reinforcement Learning example scripts do not use FlexSim's external code feature. They communicate with FlexSim via sockets. FlexSim's internal support for particular python versions is irrelevant to this question.

0 ·

question