Reinforcement Learning and Parameter Table Bound

Question

question

Steven Chen asked Apr 14, '22 Phil BoBo edited Apr 15, '22

Reinforcement Learning and Parameter Table Bound

Hello,

The observation variables in parameter table are not really bounded in the case I working on.

It's fine to set a large value as bound but I always wonder why parameter table must be strict? If unbounded variables need to save in global table, isn't it against the purpose to expose variables, because those bounded and unbounded variables were suppose to be on same table.

I think it's better to make optional lower bound and upper bound. What do you think?

By the way it will be nice to have these features in future, is this correct space to propose request?

Add "copy parameter" button to parameter table.

Listen to group instead object in Decision Events of Reinforcement Learning tool.

Software Version:

FlexSim 22.1.0

reinforcement learning parameters table

· 2

Iago MF commented · Apr 14, 2022 at 08:53 AM

Hi Steven,

If you need a solution right now, you can get the maximun allowed range (limited by the number of bits for each type) writting -1/0 for lower bound and 1/0 for upper bound.

I hope it helps you.

1 ·

Matthew Gillespie ♦♦ commented · Apr 14, 2022 at 07:52 PM

@Steven Chen For submitting feature requests you should create an idea on this site so that other users can vote and discuss the idea. You can also send the request to [email protected]. See How do I submit feature requests or bug reports?

0 ·

Answer 1 · 2022-04-14T17:00:40Z

Phil BoBo answered Apr 14, '22 Phil BoBo edited Apr 15, '22

You should normalize the observation space parameters when training RL algorithms. See Reinforcement Learning Tips and Tricks — Stable Baselines documentation (stable-baselines.readthedocs.io)

Machine Learning algorithms will learn better and faster if your observation space variables are normalized between [0,1] or [-1,1]. Using unbounded variables or variables with large, unknown ranges will cause the training to not work very well.

Machine learning isn't magic; it's math. Parameter tables are strict with bounds set on the variables so that optimization algorithms can work well.

You can copy/paste with Ctrl+C/Ctrl+V in a Parameters table to copy parameters.

I'll add a case to the dev list with the suggestion for listening to Groups.

· 2

Steven Chen commented · Apr 15, 2022 at 09:31 AM

How to use normalized value in parameter table?

I use continuous type but it is invisible in Reinforcement Learning tool.

0 ·

Phil BoBo ♦♦ Steven Chen commented · Apr 15, 2022 at 02:21 PM

If you want to use a continuous parameter, then you need to select a space type that supports continuous parameters. Use a Box space type. See The Reinforcement Learning Tool (flexsim.com)

To normalize a value, divide the difference between the value and the minimum by the difference of the maximum and the minimum:

double value = 17;
double min = 10;
double max = 20;
double normalized_0_1 = ((value - min) / (max - min)); // range [0, 1]
double normalized_neg1_1 = ((value - min) / (max - min)) * 2 - 1; // range [-1, 1]

See Feature scaling - Wikipedia

0 ·

question