question

If your issue is cost, it would probably be cheaper (in terms of time) to just buy an OptQuest license.

If your issue is that you want to control the metaheuristic, instead of using OptQuest's, then you could perhaps use the dll approach. However, if I were attempting it, I would write a C++ program that does the following:

1. Writes a file (maybe a .csv file) that has the variables for FlexSim
2. Opens a FlexSim model that has the following triggers:
   1. OnModelOpen - reads the input file, configures the model, and calls run(). You may also want to use the commandlineparam() command in that trigger, so that you can open the model manually and not have it start running immediately. You could also use that command to determine which random stream to use (if you are running replications).
   2. OnModelStop - writes an output file with the performance measures, and then calls exit(). This trigger should also probably check the command line parameter.
3. The C++ program waits for the output file to appears
4. Once the output file appears, the C++ program reads that file, and uses those results to run the metaheuristic.
5. Using the results of the metaheuristic, the C++ program writes another file.

In this way, you can write your own optimization algorithm. If you are running replications, you could launch many flexsims at once, all opening the same model, and wait for all replication results to appear.