question

Valerie Marcard avatar image
0 Likes"
Valerie Marcard asked Jouky D commented

Running Multiple Replications: Run until confidence interval is satisfied (v16.0.5)

Hi there,

I am about to start my "real" optimizations in FlexSim v16.0.5. In my testings I encountered a few things on terminating replication runs and OptQuest termination reasons. I hope somebody can help me out with my questions or/and has hints on what to try or change? That'd be great!!

I would like to let the optimizer “decide”, how many replications are "necessary” to run: It should at least run 4 replications per solution candidate and then go on running replications until a certain confidence interval and error percent is satisfied. Once, it is reached the optimizer should terminate the replication running for this solution candidate and move on to the next.

Therefore, I checked “Run multiple replications per solution” in the Optimizer Run tab and put the following in “Show advanced options”:

When I run the optimization, the optimizer terminates once the minimum number of replications is reached. I ran the Experimenter Simulation to check if the minimum number already satisfies the confidence level by chance. It doesn’t. I re-ran the Optimizer with a different value for minimum replications, and it stopped after this new minimum amount of replications. -> The option “run until confidence interval is satisfied” doesn’t seem to have any influence on the amount of replication runs.

My Run-Replications-until Questions: Am I using the advanced options for multiple replications in the right way? What should I change? Where would I see the setting in OptQuest/settings in the model tree (see second screenshot)? There was a bug on multiple replications in Version 16.0.1 – might my problem be somehow related to that bug?

On a similar topic: in the results (.csv and model tree), the “termination reason” is given. But regardless of if it was “stopped by the user” or if “all solutions have been generated” that reason is being set to “1”. According to the explanation in the model tree, “1” refers to “the optimization is still running”, though.

My Termination-Reason-Information Question: Is this (nice to have!) info supposed to be working correctly? What would I need to change to make use of it?

Thank you very much in advance for your support!!!

Best wishes Valerie

FlexSim 16.0.5
multiple replicationsconfidence intervaltermination reason
optimizersettings.jpg (32.0 KiB)
model-tree.jpg (64.5 KiB)
5 |100000
Toggle Comment visibility. Current Visibility: Viewable by all users

Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

2 Answers

Jordan Johnson avatar image
1 Like"
Jordan Johnson answered Jouky D commented

To fix the replications issue, download this zip folder. Inside is a single folder. Extract that folder, and copy it into FlexSim's modules directory (Program Files\FlexSim 2016\modules). The folder should be with the other module folders, like AGV and AStar. This requires administrative permission.

This patch should be compatible with versions 16.0.5, 16.1.2, and 16.2.0.

IMPORTANT: This patch should not be used with later versions of FlexSim. If you upgrade, you will need to delete the patch by deleting the folder in the modules directory.

Also, I was wrong about the termination reason. The Termination Reason column is actually the reason that the interface stopped running replications for a given scenario. The reasons are explained below:

  • 1 = REPLTERM_MAXREPLICATIONS = maximum replications were run
  • 2 = REPLTERM_METCONFIDENCE = the confidence interval was met
  • 3 = REPLTERM_NOTNEARBEST = the objective value was not near the best objective

The third reason means that the optimizer became confident that the given solution was not one of the best solutions, so it stopped spending replication time on that solution. This can happen for any 'Run replications until' option except the first, which always runs the minimum number of replications.

If you want to run replications until you get within the desired confidence, use -1 for your Max Replications value.

optquestpatch160916.zip


    optquestpatch160916.zip (3.6 KiB)
    · 5
    5 |100000
    Toggle Comment visibility. Current Visibility: Viewable by all users

    Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

    Valerie Marcard avatar image Valerie Marcard commented ·

    Thanks a lot, Jordan!! Looking good, so far :-)

    0 Likes 0 ·
    Jouky D avatar image Jouky D commented ·

    Hello,
    I have the same problem as @Valerie Marcard but using the 20.0.2 version. Do I have to extract the zip file as Valerie? Or do I have another problem in the code?

    0 Likes 0 ·
    Joerg Vogel avatar image Joerg Vogel Jouky D commented ·

    @Jouky D, you won‘t get an answer fast, because this answers site works a bit different then you might custom to. An accepted answer is answered. You might consider to ask a new question and put a reference to this thread hrre. And perhaps you address the question to an expert.

    2 Likes 2 ·
    Jouky D avatar image Jouky D Joerg Vogel commented ·

    Don't worry! Finally it works correctly, my fault. I put the minimum replications in 6, but the interval of confidence was satisfied with 2 replications and always I get 6 replications for each scenario. Thank you!

    1 Like 1 ·
    Jouky D avatar image Jouky D Jouky D commented ·

    Even though I put the file on modules, it doesn't work (the optimizer does all the minimum replications). I don't know how to solve it.

    0 Likes 0 ·
    Jordan Johnson avatar image
    0 Likes"
    Jordan Johnson answered

    The replications issue is a bug. I am working on a way to fix it now. For the time being, I would just use a high number of max replications. However, I am working on a fix for this problem, and I might be able to get you something by the end of the day, or tomorrow.

    The termination reason is only used by the window that pops up when the optimization stops. The results bundle will have a 1 on every row in the termination reason column.

    5 |100000
    Toggle Comment visibility. Current Visibility: Viewable by all users

    Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.