question

Igor van der Weide avatar image
0 Likes"
Igor van der Weide asked Igor van der Weide answered

Write to (and export) global table not working during experiment?

During modeling I ran into a problem regarding the use of a global table combined with the use of the experimenter. I attached a simplified model that consists of a arrival- queuing- and exit-object. Each time a patient runs through the model the onEntry and the onExit trigger of the queuing object and the onEntry trigger of the exitobject set some value in the global table.

While running the model the above works as specified, however when using the experimenter the global table seems to remain empty...

Ideally, after each experiment I would like to store a csv file with the recorded data in the global table. For this matter I added some code in the endOfReplication trigger to save a csv file (learned from: https://www.flexsim.com/community/forum/showthread.php?t=2774). Also this file remains empty/zero. However, it seems to me that problem has more to do with the onEntry and onExit trigger than the csv solution.

Does anyone have an idea about how to solve this? Did I make wrong assumptions about the use of global tables?

testgtoutput.fsm

Flexsim HC 5.1

Choose One
global tableexperimenterhctrigger
testgtoutput.fsm (76.2 KiB)
5 |100000

Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Cliff King avatar image
1 Like"
Cliff King answered Cliff King commented

I'm not sure all that you've tried, but typically the problems associated with recording experiment data to a global table arises from the fact that each replication of an experiment is sent off to one of multiple processing cores on your computer.

I have had success recording information written to a global table during a model run out to an external csv file at the end of each replication of the experiment. In my example, the global table is named "Trace".

I put the following code in the "Start of Experiment" field of the Experiment Control Window's Advanced tab:

/**Save off a reference to the model directory*/
sets(assertsubnode(node("Tools", model), "ModelDir", DATATYPE_STRING), modeldir());

and I put the following code in the "End of Replication" field:

//////////////////////////////////// CODE HEADER //////////////////////////////////////////////////////////////////////
double replication = parval(1);
double scenario = parval(2);
treenode childexpfolder = parnode(3);
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////

{ //************* PickOption Start *************\\


/**Write the current rep's Trace table data to a csv file*/
// only do this on the child process
if (!objectexists(childexpfolder))
{
	
/**\n\nFile name: */


string modelDir = gets(node("Tools/ModelDir", model));
string filename = /**/concat(modelDir, "S", numtostring(scenario), "R", numtostring(replication), "_PatientTrace.csv")/**/;


//delimeter character used to separate data into columns
int comma = 44;
int tab = 9;
int semicolon = 59;
int CR = 13;


/**\nData delimeter: */
int delimeter = /**/comma/**list:comma~tab~semicolon*/;


//create csv file and write column headers
int fileHandle = fileopen(filename, "w");


// Add scenario and replication column headers first
fpt("Scenario, Replication, ");
int rows = nrows(Trace);
int cols = ncols(Trace);
for(int column=1; column<=cols; column++)
{
	fpt(gettableheader(Trace,2,column));
	if(column < cols)
	{
		fpc(delimeter);
		fpt(" ");
	}
	else
		fpc(CR);
}


// Loop through the Trace table and append data to the file one row at a time
for(int row = 1; row <= rows; row++)
{
	// Record the current scenario and replication as the first two columns of data
	fpd(scenario);
	fpc(delimeter);
	fpt(" ");
	fpd(replication);
	fpc(delimeter);
	fpt(" ");
	for(int col = 1; col <= cols-1; col++)
	{
		if(col == TRACE_COL_PatientID
		 || col == TRACE_COL_SlotIsAvailable
		  || col == TRACE_COL_InitialDose
		   || col == TRACE_COL_ScreenFail
		    || col == TRACE_COL_ActualDose)
			fpd(gettablenum(Trace, row, col));
		else if(col == TRACE_COL_FinalDisposition)
			fpt(gettablestr(Trace, row, col));
		else
			fpf(gettablenum(Trace, row, col));
		fpc(delimeter);
		fpt(" ");
	}
	fpt(gettablestr(Trace, row, cols));
	fpc(CR);
}


fileclose();


} // if (!objectexists(childexpfolder))


/** \n\n*/


} //******* PickOption End *******\\


· 2
5 |100000

Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Igor van der Weide avatar image Igor van der Weide commented ·

Thank you so much for sharing this code. Indeed, it works as described and even provides a solution for multiple scenario’s and replications. I managed to extend my example model with this code and attached it to this message, for those who find it useful to have a working example.

Maybe for future some version of HC it would be a great idea to include this (global table output) functionality in the experimenter and/or the output tab. The ease with which a global table allows a user to record (aggregate) model output makes it the perfect way (at least for me) to select and combine model data for further analysis. testgtoutput-v2.fsm

0 Likes 0 ·
testgtoutput-v2.fsm (65.4 KiB)
Cliff King avatar image Cliff King ♦ Igor van der Weide commented ·

Good suggestion. I'll make sure this code gets added as a standard pick list option for the next release of the software.

0 Likes 0 ·
Cliff King avatar image
0 Likes"
Cliff King answered

Per Igor's suggestion, we will be adding the option to "Write Global Table Data To An External CSV File" as a standard pick list option for the "End of Replication" field of the Simulation Experiment Control window. In the mean time, here is the code for the new pick list option. It is more generic than the previously submitted code and will automatically determine whether the data being copied is numeric or string data and handle it accordingly. You just need to specify the global table name, the directory path where the csv files will be saved, and the type of delimeter you want to use for separating the data values within the csv files.

//////////////////////////////////// CODE HEADER //////////////////////////////////////////////////////////////////////
double replication = parval(1);
double scenario = parval(2);
treenode childexpfolder = parnode(3);
///////////////////////////////////////////////////////////////////////////////////////////////////////////////////////
{ //************* PickOption Start *************\\
/**Write Global Table Data To An External CSV File*/
// Only write data if the current process is a the child process
if (!objectexists(childexpfolder))
{
	/**\n\nGlobal Table: */
	treenode globalTable = /**/MyTable/**/;
	
	/**\n\nDirectory Path: */
	string modelDir = /**/"D:/cliff.king/Documents/FlexSimHC 5 Projects/"/**/;
	string fileName = concat(modelDir, "S", numtostring(scenario), "R", numtostring(replication), getname(ownerobject(globalTable)), ".csv");
	// The possible delimeter characters used to separate data into columns
	#define COMMA_DELIMETER 44
	#define TAB_DELIMETER 9
	#define SEMICOLON_DELIMETER 59
	#define CARRIAGE_RETURN_DELIMETER 13
	/**\nData Delimeter: */
	int delimeter = /**/COMMA_DELIMETER/**list:COMMA_DELIMETER~TAB_DELIMETER~SEMICOLON_DELIMETER~CARRIAGE_RETURN_DELIMETER*/;
	/**\n\nNote: The CSV files will be saved in the directory specified above, and the names of the files will have the following format: S#R#_tablename. This format will clarify the scenario number and replication number associated with the tabular data saved in each new file created at the end of each replication.*/
	// Create and open a new csv file for writing
	int fileHandle = fileopen(fileName, "w");
	// Add scenario and replication column headers first
	fpt("Scenario, Replication, ");
	int rows = nrows(globalTable);
	int cols = ncols(globalTable);
	// Now add the other column headers associated with the global table columns
	for(int column=1; column<=cols; column++)
	{
		fpt(gettableheader(globalTable,2,column));
		if(column < cols)
		{
			fpc(delimeter);
			fpt(" ");
		}
		else
			fpt("\n"); //add a new line
	}
	// Loop through the global table and append data to the file one row at a time
	for(int row = 1; row <= rows; row++)
	{
		// Record the current scenario and replication as the first two columns of data
		fpd(scenario);
		fpc(delimeter);
		fpt(" ");
		fpd(replication);
		fpc(delimeter);
		fpt(" ");
		// Record the data (numbers or text) contained in each column of the table
		for(int col = 1; col <= cols; col++)
		{
			if(getdatatype(gettablecell(globalTable,row,col)) == DATATYPE_NUMBER)
				fpd(gettablenum(globalTable, row, col));
			else
				fpt(gettablestr(globalTable, row, col));
			if(col < cols)
			{
				fpc(delimeter);
				fpt(" ");
			}
		}
		fpt("\n"); //add a new line
	}
	fileclose();
}
} //******* PickOption End *******\\

5 |100000

Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Igor van der Weide avatar image
0 Likes"
Igor van der Weide answered

Strange that the website handles your post different than mine. I guess without a proper reproduction of the error it is impossible to find a solution.

I included a screenshot of the model tree node in my model, so you can see that it doesn't always includes an ending slash. Maybe you have to include a conditional slash or something in the code.

Thanks again for all your answers and willingness to improve flexsim!

outputdirectory.png


outputdirectory.png (17.0 KiB)
5 |100000

Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Cliff King avatar image
0 Likes"
Cliff King answered

Igor, I just posted the above post as if I were you (a tricky little feature allowed by the Answers Hub software!), and it did not give me the error message you were getting. The verbiage and suggested code in the post I copied directly from the reply.pdf file you sent a few posts back.

I understand what you're referring to by the "tiny problem" with the ending slash character, but my experience is a little different. The directory path string stored in the model tree node you're referencing (i.e. MODEL:/Tools/Output>variables/OutputDirectory), actually includes an ending slash, so in my case it is not necessary to concatenate an ending slash to the directory path string.

For now, I'd say to just go with what works for you, and hopefully my new changes in the official pick list option will resolve all these issues.

5 |100000

Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Igor van der Weide avatar image
0 Likes"
Igor van der Weide answered Cliff King edited

Great, thanks again! - In flexsim it has to be line 57... :-)

if(getdatatype(gettablecell(globalTable,row,col)) == DATATYPE_NUMBER)
     fpf(gettablenum(globalTable, row, col));
else
     fpt(gettablestr(globalTable, row, col)); 

Maybe some blank lines get lost in transferring the code through this website or something. Also the string statement of modelDir is on a different line (15) than you stated (18). - A window interface for selecting the directory sounds even better! I think the output directory of the output data setting would be a logical default. - The replacement code you provided for the modelDir introduced a tiny problem which I couldn't get my head around at first. The node you referred to leads to a directory address without an ending slash and as a consequence, the csv file gets saved one folder up and the foldername gets added to the csv filename... :). Adding a slash in the code does the trick. This is what I got, also including a suggestion for the prefix we discussed:

/**\n\nDirectory Path: */

string modelDir =

/**/concat(gets(node("MODEL:/Tools/Output>variables/OutputDirectory")), "/")/**/;

/**\n\nOutput File Name Tag: */

string fileName = concat(modelDir, /**/"MyRun"/**/, "_S", numtostring(scenario), "R",

numtostring(replication), getname(ownerobject(globalTable)), ".csv");
5 |100000

Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Igor van der Weide avatar image
0 Likes"
Igor van der Weide answered

I tried using code blocks and I also tried it again a few minutes ago. My IP does not seem to get blocked although the error message suggests that. I can immediately post something else, as long as it doesn't include code... Maybe you could also try to repost my reply and see if that works.

Ontopic: hope that @Cliff King gets to see my reply anyway :)

5 |100000

Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Igor van der Weide avatar image
0 Likes"
Igor van der Weide answered Ralf Gruber edited

I try to post a reply (with a shorte code statement) ,but now my IP is blocked...

Oh, this still seems to work, let me add the error message that I got: blocked.png

And if this is working, then here is my reply: reply.pdf


blocked.png (45.6 KiB)
reply.pdf (41.6 KiB)
· 2
5 |100000

Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Igor van der Weide avatar image Igor van der Weide commented ·

Cloudflare Ray ID: 381ca74a1f601497

0 Likes 0 ·
Ralf Gruber avatar image Ralf Gruber ♦ commented ·

CloudFlare is a service that acts to speed up and protect websites. Answers.flexsim.com is a Q&A site hosted on a platform called AnswerHub. Answerhub subscribes to CloudFlare's service to speed up and protect Q&A sites like ours.

This is definitely a good thing, but we have seen it occasionally block code submissions.

As part of the CloudFlare service, the admin (that would be the AnswerHub people) can set different rules for CloudFlare to prevent against certain types of attacks. One of those would be code-injection attacks against a site.

CloudFlare monitors form submissions, and if it determines that something being submitted could compromise the server, it blocks the submission, and sometimes blocks the user for a time period of a few hours or a day. This is probably what happened to you. You submitted some code that CloudFlare determined looked suspicious, and so CloudFlare blocked the code, as well as your IP for a short time.

I wonder, was the code you submitted within a code block? Similar to how Cliff has used code blocks above? Like this:

I don't know if that makes a difference, but it could be worth a try.

Another workaround would be to attach a .txt file containing the code (or a pdf, as you've already done).

0 Likes 0 ·
image.png (11.3 KiB)
Igor van der Weide avatar image
0 Likes"
Igor van der Weide answered Cliff King commented

Wow, very nice indeed!

A few comments:

  • Could you write numeric values as floating point instead of integer? Otherwise, I miss the decimal values. Line 57: fpd=fpf?
  • The format in which the directory path has to be defined is with the use of “/”. While I intuitively tried to use the windows standard “\”, I got a syntax error. Would it be possible to also include the windows standard “\”? In my case we use network paths and the most logical way to select it would be to copy/paste it from the file navigator, but then the “\” error appears…

Also may I suggest a few (minor) improvements?

  • Could you change the pre-coded directory path to the path of the simulation model? Or add this location as a picklist option?
    Or use the same path that is defined in the ‘output data settings tab’? And/or add this location as a picklist option.

    We use a company network drive and the actual path to our folders is long and tedious to enter manually.

  • Could you provide a possibility to enter a prefix for the CSV file name? I would suggest to make it consistent with the use of the ‘file name tag’ (default MyRun) in the ‘output data settings tab’. This would make it easy to make distinctions (e.g. between models with identical global table names).
  • And to maximize the consistency with the ‘output data’ functionality, maybe we could also have different runs saved as different versions with a trailing ‘_v#’. Or add this as a built-in picklist option ‘overwrite csv in directory during reruns y/n’?

Hope you appreciate these comments. Please note that I'm already grateful with even the simplest form of this functionality!

· 3
5 |100000

Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Cliff King avatar image Cliff King ♦ commented ·

Igor,

These are all good suggestions. I'll see what I can implement in a timely manner...

and yes, you can use fpf instead of fpd to capture floating point numbers. That should be the default, so I'll be sure to make that change as well. (You will want to do this on line 67, not 57, by the way).

0 Likes 0 ·
Cliff King avatar image Cliff King ♦ commented ·

Oh, and with regards to / versus \, that will not be an issue with the new pick option because I plan on using a window interface that will allow browsing for the directory rather than require the user to type in the string. The problem with using \ is that the backslash is the universal escape character and causes issues with interpretation by the code syntax checker. You can use the backslash instead of the forward slash, but then you would need to be sure and use double backslash marks. The first backslash escapes and the second one is then treated as a true backslash.

0 Likes 0 ·
Cliff King avatar image Cliff King ♦ commented ·

Oh, and if you want to use the directory path you've already specified in the Output Data Settings window, you can use the following replacement for line 18:

string modelDir = /**/gets(node("MODEL:/Tools/Output>variables/OutputDirectory"))/**/;
0 Likes 0 ·

Write an Answer

Hint: Notify or tag a user in this post by typing @username.

Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.