Why do different graphs based on the same data show different results?

Question

question

Lou Keller asked Jan 10, '17 Cliff King commented Jan 16, '17

Why do different graphs based on the same data show different results?

Submitted on behalf of Jim Montgomery, Carilion Clinic. Regarding the attached model, why would the “N” of my data result graphs be so different when comparing Milestone-minus-Milestone-Time graphs with Data-Collector-User-Defined graphs? The two graphs should demonstrate the same results and have the same sample size.

The Milestone-Milestone data points are created when the time the patient enters the exam room is subtracted from the time the patient concludes triage.

The Data-Collector data points are created each time the global variable “v_TriageEnd_to_Roomed” is updated.

Here’s the problem: The “N” of the data-collector graphs is 4 times larger than the milestone-milestone graphs, and the milestone-milestone graph’s N is closer to what actual patient throughput should be. Why are so many data points being collected for the data-collector graphs when the global variable on which they are based is only supposed to be reset when the patient enters the room?

ed-ver-23.fsm

Software Version:

FlexSim HC 5.1.0

data collector statistics and reports custom dashboard firedelegate user-defined graph

ed-ver-23.fsm (10.2 MiB)

5 |100000

Attachments: Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Answer 1 · 2017-01-10T18:31:57Z

Matthew Gillespie answered Jan 10, '17 Cliff King commented Jan 11, '17

It looks like you're over-recording data with the data collector. You have lots of duplicate values recorded:

You have the data collectors set up to record on the Custom recording event and to always record. It looks like you have FireDelegate(DELEGATE_CUSTOM); code sprinkled throughout your tracks. Whenever you call this it's recording a new value. I think you need to set some criteria so that it's not recording a new value everytime.

data.png (34.2 KiB)

· 3

5 |100000

Attachments: Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Lou Keller commented · Jan 10, 2017 at 08:33 PM

Thanks Matthew. As usual, you're right on top of things!

0 ·

Jim Montgomery commented · Jan 11, 2017 at 02:12 AM

Thank you Matthew for analyzing the problem. I do not know how to fix it. Could you please provide a step-by step guide (for us slow learners) on how to use data collectors with global variables? I have several global variables in the model that change values with patient activities, and I need to be able to graph these variable values. Thus, I have created several data collectors which, I thought were capturing a new data point each time the global variable changed values by using the command, “FireDelegate” I’m sorry, but as a former MedModel user, the concept of adding the “data collector” step in order to let me graph a variable is foreign to me. MedModel had no extra step; instead the values of all user-created variables could be graphed outright.

Thanks,

Jim

@lou.keller

0 ·

Cliff King commented · Jan 11, 2017 at 11:15 PM

You're right Jim, there are too many steps required for graphing a user-defined variable. In the next software release we let the user create variables called "tracked variables" which have nice automated reporting features.

I've noticed that our ex-MedModel users gravitate to the creation of their own user-defined variables rather than use standard system variables that are already defined in the software. As you've learned, our existing dashboard widgets are designed to display system variables, with the exception of the "User-Defined" widget that displays data captured by a user-defined "Data Collector." This is not as limiting as it sounds, because the Milestone and Milestone-To-Milestone widgets display information regarding milestone events that are entirely user-definable through the track activities. Please note that the milestone event times can be recorded either at the beginning or end of an activity (e.g. before or after the process time period of an activity), and if that's not enough resolution, you can always create another activity with the sole purpose of identifying a particular milestone event.

I will take a look at your model (ed-ver-23.fsm) and give you some specific feedback on how to use Data Collectors with Global Variables.

0 ·

Answer 2 · 2017-01-13T23:25:31Z

Cliff King answered Jan 13, '17 Cliff King commented Jan 16, '17

Incidentally, I learned more reasons for the difference in N (i.e. the number of data points used to calculate the box plot) between the Data Collector graph and the Milestone-to-Milestone graph.

So anyways, I dove into the model and tried to do some comparisons on a patient-by-patient basis. What I discovered is that the big difference between the two graphs is not due to the difference in timing between when the two graphs determine that the patient is roomed, but it is mostly due to the timing of when the two graphs have access to the data collected. The Data Collector graph gets the data immediately after the FireDelegate command is executed, but the Milestone-to-Milestone graph doesn't have access to the milestone data for a patient until after a patient leaves the model. Milestone times are recorded on the patient as they occur, but are not written to the global history database until the patient finishes their track and leaves the model.

If you remove the warmup period and run your model for just 1 day, you will be able to see that the two graphs are almost in sync except for one or two of the 24 periods of the 1st day. Unfortunately, the two charts will soon get out of sync as the model rolls into the second day, but things appear to be almost be in sync at this time. The only way to force the two graphs to perfectly sync, is to run a terminating model that flushes out all of the patients at some point in time. As a matter of note, you can move the FireDelegate command from the activity started trigger of the Nurse_Assess_WI where it is now, to the Process Time field, and this will eliminate any differences due to the reason I gave earlier. Then truly the only difference will be due to the lag between when the Data Collector graph receives the data point for a given patient and when the Milestone-to-Milestone graph does.

· 1

5 |100000

Attachments: Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Cliff King commented · Jan 16, 2017 at 06:27 PM

I should have identified warmup period as being another reason for discrepancy, because now that I think about it, there's no way that a user-defined graph would automatically know how to correct for a warmup period without having any knowledge of the type of metric being recorded by the user. A different correction scheme would be required for time based metrics, than for cumulative count type metrics, right? I'm curious how we are correcting for the warmup time in the Milestone-to-Milestone graph currently. I'll need to spend some time looking into the code behind the scenes to figure it out. I wonder what we do with the milestone-to-milestone time recorded for a patient whose cycle spans across the warmup time??

I avoid using warmup times in simulation projects I'm involved in for reasons such as this. I know as a programmer that warmup handling is tricky business, plus there are a number of controversial issues on the theoretical side of things as well. In an ER model, I usually don't like to look at overall end-of-run point statistics anyways, because they ignore important variations that occur across the hours of a day in an ER. I prefer to look at stats within a given hour. I also do not like to include the variations across the days of the week within the same ER model for similar reasons. I will run each day of the week as a separate model. Usually there is not interest in every day of the week, so I typically run just the busiest day of the week. The point is that each day of the model run represents arrivals for the same 24 hour period. Of course I use the variance in patient arrivals and model parameters for that day, so that each simulated day is not exactly the same.

0 ·

Answer 3 · 2017-01-12T04:04:30Z

Cliff King answered Jan 12, '17 Cliff King commented Jan 13, '17

In my attached model (ed-ver-23-ck.fsm) I made changes to the FireDelegate call in your model. I used the optional parameters of the FireDelegate command to pass in a parameter I could use in the Data Collectors to distinguish which of the three calls was being made. You call the FireDelegate command in the following three places for each of the tracks:

After changing the v_Triage_TAT global variable in the Activity Started Trigger of activity 46_Conduct_Triage.
After changing the v_RegToTriageStart_TAT global variable in the Activity Started Trigger of activity 46_Conduct_Triage.
After changing the v_TriageEnd_to_Roomed global variable in the Activity Finished Trigger of activity 59_Nurse_Assess_WI.

I passed in the values 1,2 and 3 respectively to identify each of these three calls it was. Incidentally, I also passed in a reference to the patient for the involved object so that I could associate the involved patient's PCI with the call as well.

Please take a look at what changes I made, and let me know if you have any questions.

I should warn you that you still won't be able to exactly correlate the results of the user-defined chart and the standard milestone-milestone chart. This is because of a difference in when the patient roomed time is recorded. The "WI Patient Roomed" milestone is being recorded "Before Process Time" as defined by the Milestone Recording Event option you chose. The "Before Process Time" event occurs after the nurse is allocated and travels to the patient for the assessment. On the other hand, the custom event that triggers the data collector recording occurs when the FireDelegate command is called in the Activity Started Trigger of the assessment activity. The Activity Started Trigger fires immediately after the activity gets kicked off, before the nurse even gets allocated!

Here's a few bullet points that may help you better understand Data Collectors and the user-defined dashboard widgets:

The sole purpose of a Data Collector is to record data whenever a specific event occurs in the model.
The "Recording Event" field on the Data Collector's properties window has a list of options to choose from. You can choose to specify a regular time interval for making recordings, or choose to piggy-back on top of a standard model event (there are 11 to choose from) to make a recording, or choose the "Custom" option which expects you to use the FireDelegate() command in your model code somewhere to force a recording event.
Whenever the Data Collector's recording event occurs, the "Data Recording Criteria" is evaluated to determine whether any data should actually be recorded or not. I used this field to distinguish between the three different calls I mentioned earlier for the FireDelegate command in your model.
When the recording event passes the recording criteria, all the columns of data you've defined for the data collector will be recorded. The reason we use the term "Columns" to refer to the data values that get recorded, is because the data is stored in a 2-dimensional table for each Data Collector, where each row represents a single occurrence of an event (that passes the recording criteria anyways), and each column of the table represents a separate metric that gets recorded.
The User Defined dashboard widget is used to display the data collected by a single Data Collector. The widget can display one or more of the columns of data recorded by the Data Collector. You do not need (or usually want) to display all the columns of data in the same graph. It's common to use one or more of the data columns for the sole purpose of defining filters when creating the "Sample Sets" for the graph.

ed-ver-23-ck.fsm (9.2 MiB)

· 2

5 |100000

Attachments: Up to 12 attachments (including images) can be used with a maximum of 23.8 MiB each and 47.7 MiB total.

Jim Montgomery commented · Jan 12, 2017 at 06:47 PM

Thank you Cliff. This is great information that I needed to finish up my model validation. So would it be possible to additionally create a 3rd column in my Data Collector, "DC_TriageEnd_to_Roomed_TAT" entitled, "Patient Zone", that would allocate data points between the 6 categories of my user-defined label, "a_My_Zone" ? It doesn't appear that I can, as I seem to be limited to Patient Attributes: PCI, Patient ID and Acuity. Ultimately, I would like to be able to view results after removing 2 patient areas (zones) from consideration. Perhaps an IF THEN statement placed before the FireDelegate call is required in this situation?

Thanks,

Jim

@lou.keller

0 ·

Cliff King Jim Montgomery commented · Jan 13, 2017 at 11:16 PM

Yes, you can add a 3rd column to the Data Collector to capture the zone label of the patient that triggers the FireDelegate call to the Data Collector. I believe there's an available option for recording patient attributes such as a label to a column, but if not, the expression is simply:

getlabelnum(patient,"a_My_Zone")

The column names of all columns of data defined for a Data Collector will show up as possible filtering options that you can reference when setting up custom sample sets in the user-defined graph associated with the Data Collector. I don't know how you've got your zones numbered, but let's assume you have zones numbered 1 through 5 in your model, and that you want to create a sample set to display the TAT for just zones 4 and 5. In addition, let's also assume you named your 3rd column "PatientZone" in the Data Collector. You would then create a custom sample set in the user-defined graph's settings window and add the following two filters:

PatientZone == 4
PatientZone == 5

or, because the two zones are consecutively numbered and no zones above 5 exist, you would be able to use a single filter such as this:

PatientZone >= 4

When defining custom sample sets with filters, it's important to understand that each filter defining a given sample set are AND'd rather than OR'd.

0 ·

question