Inconsistent Experimenter Results With High Scenario Count

Question

question

Leo L6 asked Feb 13, '23 Andrew O commented Feb 17, '23

Inconsistent Experimenter Results With High Scenario Count

Hi,

I have an issue with the experimenter when I create a high number of scenarios in models with high complexity (high number of objects and process flow blocks) in one experiment I get very weird results. Basically what happens is that I see unreasonably high variation in the metric I'm trying to measure. For example if I'm measuring output, the results vary from 1000 all the way down to 0. The number of scenarios in which this becomes a problem varies, but generally I seem to see it happen beyond ~60 scenarios, but sometimes as low as 15. The run time and replication count are over 1 month and over 20 respectively for each scenario.

I know this is an issue unrelated to how I set up the model because when I run each scenario one at a time (with exact same parameters, replications, run length, etc), the issue disappears and results are in line with expectations.

I am unable to share the models where I see this issue occur, but I am wondering if this is a known issue and if so, what steps can I take to resolve.

Another note, this seems to happen more frequently in newer versions (2020+)

Thanks in advance

Leo

Software Version:

FlexSim 22.0.12

experimenter bug experiment run

· 4

Jason Lightfoot ♦ commented · Feb 13, 2023 at 09:32 PM

Do you run your tests with all GUIs/views closed? Do you run the test replications back to back? Are you using the same stream with Repeat Streams turned on? Do you have any stochastic samplers that do not use a stream value (and will default to zero?) Have you checked the Model repeatability notes?

0 ·

Leo L6 Jason Lightfoot ♦ commented · Feb 13, 2023 at 11:30 PM

Thanks for the reply.

No I don't run with GUI/view closed, but I run tests through experimenter, so it shouldn't be a factor, no?

Yes they are back to back

Yes to repeat repeat streams

I don't believe I use any stochastic samplers.

I will look through the model repeatability notes. The event logger seems to be appropriate.

0 ·

Jason Lightfoot ♦ Leo L6 commented · Feb 14, 2023 at 03:00 PM

How are you running the scenarios - do you have a single webserver or multiple webservers, or are you just running them on your local machine? For each machine involved. how much memory does the machine have and how many child processes run simultaneously and how much memory does each use?

Whenever I see a user that is creating a large number of threads in parallel I'm reminded of the non-interactive desktop heap limit that is set in the registry and sometimes needs to be changed.

1 ·

Andrew O commented · Feb 17, 2023 at 07:03 PM

Hi @Leo L6,

We haven't heard back from you. Were you able to solve your problem? If so, please add and accept an answer to let others know the solution. Or please respond to the previous comment so that we can continue to help you.

If we don't hear back in the next 3 business days, we'll assume you were able to solve your problem and we'll close this case in our tracker. You can always comment back at any time to reopen your question, or you can contact your local FlexSim distributor for phone or email help.

0 ·

question