Experimenter Scenario Replication Error?

Question

question

Ankur A3 asked Nov 01 2022 at 3:15 PM Andrew O commented Nov 07 2022 at 6:19 PM

Experimenter Scenario Replication Error?

Hi Team,

While running multiple replications using experimenter, some replications are throwing errors which can be seen in console output tab. Still run is completed and results are generated for the scenario.

I have 2 questions:

1. Is it okay to believe on results in this case?

2. How to deep dive on errors for particular replication?

Thank you!

Software Version:

FlexSim 21.1.5

experimenter replications issue

· 2

______

Cookie preferences

Your privacy is important to us and so is an optimal experience. To help us customize information and build applications, we collect data about your use of this site.

May we collect and use your data?

Learn more about the Third Party Services we use and our Privacy Statement.

Strictly necessary – required for our site to work and to provide services to you

These cookies allow us to record your preferences or login information, respond to your requests or fulfill items in your shopping cart.

YES

Improve your experience – allows us to show you what is relevant to you

These cookies enable us to provide enhanced functionality and personalization. They may be set by us or by third party providers whose services we use to deliver information and experiences tailored to you. If you do not allow these cookies, some or all of these services may not be available for you.

YES

NO

Customize your advertising – permits us to offer targeted advertising to you

These cookies collect data about you based on your activities and interests in order to show you relevant ads and to track effectiveness. By collecting this data, the ads you see will be more tailored to your interests. If you do not allow these cookies, you will experience less targeted advertising.

YES

NO

Are you sure you want a less customized experience?

We can access your data only if you select "yes" for the categories on the previous screen. This lets us tailor our marketing so that it's more relevant for you. You can change your settings at any time by visiting our privacy statement

Your experience. Your choice.

We care about your privacy. The data we collect helps us understand how you use our products, what information you might be interested in, and what we can improve to make your engagement with Autodesk more rewarding.

May we collect and use your data to tailor your experience?

Explore the benefits of a customized experience by managing your privacy settings for this site or visit our Privacy Statement to learn more about your options.

Answer 1 · 2022-11-02T18:58:05Z

Jordan Johnson answered Nov 02 2022 at 6:58 PM

1) If some replications have exceptions and others don't, I wouldn't trust the results from the replications that had exceptions. However, you can always run the replication interactively.

2) To run a specific replication of a scenario interactively, hover over that replication on the chart, and click on the test tube button:

That button applies the parameter values, warmup time, stop time, and replication number to the model, and it resets the model. You should be able to run the model and get the exact same behavior interactively as you got in the experiment run.

1667415429668.png (42.7 KiB)

Answer 2 · 2022-11-01T15:20:54Z

Jason Lightfoot answered Nov 01 2022 at 3:20 PM Jason Lightfoot edited Nov 02 2022 at 7:06 PM

1. It Depends. If it's less than the required confidence level for the runs and/or you assume they are not biased then it might be okay. (I meant it might be okay to beleive the remaining results if you have enough confidence remaining - the error replications I would discard)

2. Set the "Repeat streams of Replication" number to one of the troublesome replications and run the model to see the problem unfold. Store the state of the model and load that if you think it will help you understand how the problem occurred by looking at the end state.

· 2

Ankur A3 commented · Nov 01 2022 at 6:20 PM

Hi @Jason Lightfoot,

I am not clear on my 2nd question. Let me ask with some more clarity.

I am running 10 replications for scenario1 using experimenter and scenario 1 is throwing error in replication #5. How to deep dive in replication 5?

The reason of asking this question is that we don't know which seed value has created that particular replication so how to run that replication in 3D environment to check the errors?

Can you add some images or model for easy explanation?

Thank you!

0 ·

Jason Lightfoot ♦♦ Ankur A3 commented · Nov 02 2022 at 7:29 AM

Under the scenario tab set the model to scenario 1 by selecting the column and pressing the button to the right.

Then under the Advanced tab set the replication to repeat to 5:

Now run the model - you should see the same results as the experiment - if not check for repeatability.

0 ·

1667374039587.png (10.2 KiB)

1667374106815.png (4.9 KiB)

question