Running Multiple Replications: Run until confidence interval is satisfied (v16.0.5)

Question

question

Valerie Marcard asked Sep 20, '16 Jouky D commented Oct 21, '20

Running Multiple Replications: Run until confidence interval is satisfied (v16.0.5)

Hi there,

I am about to start my "real" optimizations in FlexSim v16.0.5. In my testings I encountered a few things on terminating replication runs and OptQuest termination reasons. I hope somebody can help me out with my questions or/and has hints on what to try or change? That'd be great!!

I would like to let the optimizer “decide”, how many replications are "necessary” to run: It should at least run 4 replications per solution candidate and then go on running replications until a certain confidence interval and error percent is satisfied. Once, it is reached the optimizer should terminate the replication running for this solution candidate and move on to the next.

Therefore, I checked “Run multiple replications per solution” in the Optimizer Run tab and put the following in “Show advanced options”:

When I run the optimization, the optimizer terminates once the minimum number of replications is reached. I ran the Experimenter Simulation to check if the minimum number already satisfies the confidence level by chance. It doesn’t. I re-ran the Optimizer with a different value for minimum replications, and it stopped after this new minimum amount of replications. -> The option “run until confidence interval is satisfied” doesn’t seem to have any influence on the amount of replication runs.

My Run-Replications-until Questions: Am I using the advanced options for multiple replications in the right way? What should I change? Where would I see the setting in OptQuest/settings in the model tree (see second screenshot)? There was a bug on multiple replications in Version 16.0.1 – might my problem be somehow related to that bug?

On a similar topic: in the results (.csv and model tree), the “termination reason” is given. But regardless of if it was “stopped by the user” or if “all solutions have been generated” that reason is being set to “1”. According to the explanation in the model tree, “1” refers to “the optimization is still running”, though.

My Termination-Reason-Information Question: Is this (nice to have!) info supposed to be working correctly? What would I need to change to make use of it?

Thank you very much in advance for your support!!!

Best wishes Valerie

Software Version:

FlexSim 16.0.5

multiple replications confidence interval termination reason

optimizersettings.jpg (32.0 KiB)

model-tree.jpg (64.5 KiB)

Answer 1 · 2016-09-22T21:24:08Z

Jordan Johnson answered Sep 22, '16 Jouky D commented Oct 21, '20

To fix the replications issue, download this zip folder. Inside is a single folder. Extract that folder, and copy it into FlexSim's modules directory (Program Files\FlexSim 2016\modules). The folder should be with the other module folders, like AGV and AStar. This requires administrative permission.

This patch should be compatible with versions 16.0.5, 16.1.2, and 16.2.0.

IMPORTANT: This patch should not be used with later versions of FlexSim. If you upgrade, you will need to delete the patch by deleting the folder in the modules directory.

Also, I was wrong about the termination reason. The Termination Reason column is actually the reason that the interface stopped running replications for a given scenario. The reasons are explained below:

1 = REPLTERM_MAXREPLICATIONS = maximum replications were run
2 = REPLTERM_METCONFIDENCE = the confidence interval was met
3 = REPLTERM_NOTNEARBEST = the objective value was not near the best objective

The third reason means that the optimizer became confident that the given solution was not one of the best solutions, so it stopped spending replication time on that solution. This can happen for any 'Run replications until' option except the first, which always runs the minimum number of replications.

If you want to run replications until you get within the desired confidence, use -1 for your Max Replications value.

optquestpatch160916.zip

optquestpatch160916.zip (3.6 KiB)

· 5

Valerie Marcard commented · Sep 26, 2016 at 08:23 AM

Thanks a lot, Jordan!! Looking good, so far :-)

0 ·

Jouky D commented · Oct 20, 2020 at 01:26 AM

Hello,
I have the same problem as @Valerie Marcard but using the 20.0.2 version. Do I have to extract the zip file as Valerie? Or do I have another problem in the code?

0 ·

Joerg Vogel Jouky D commented · Oct 20, 2020 at 03:51 AM

@Jouky D, you won‘t get an answer fast, because this answers site works a bit different then you might custom to. An accepted answer is answered. You might consider to ask a new question and put a reference to this thread hrre. And perhaps you address the question to an expert.

2 ·

Jouky D Joerg Vogel commented · Oct 21, 2020 at 05:17 PM

Don't worry! Finally it works correctly, my fault. I put the minimum replications in 6, but the interval of confidence was satisfied with 2 replications and always I get 6 replications for each scenario. Thank you!

1 ·

Jouky D Jouky D commented · Oct 20, 2020 at 01:47 AM

Even though I put the file on modules, it doesn't work (the optimizer does all the minimum replications). I don't know how to solve it.

0 ·

Answer 2 · 2016-09-22T20:11:35Z

Jordan Johnson answered Sep 22, '16

The replications issue is a bug. I am working on a way to fix it now. For the time being, I would just use a high number of max replications. However, I am working on a fix for this problem, and I might be able to get you something by the end of the day, or tomorrow.

The termination reason is only used by the window that pops up when the optimization stops. The results bundle will have a 1 on every row in the termination reason column.

question