Idea

Chris Smith suggested Apr 7, '17 Ben Wilson edited Aug 30, '17

Function to sample a value from a webpage

I think this is actually something that would be extremely useful for a lot of users, and also fairly easy to implement due to development in the software in the last few years. The idea would be something like webvalue(String - website url, String - HTML element id, Boolean - on Reset). The first parameter would be for the url of the site with the value. The second would be for the id of the element that contains the string. The third would be a value that determines if the value is obtained from the website on reset only and stored in a label to be used for future calls (if the function is called during the model run it would just reference the value in the label) or if a request is done each time it is called (this could really slow down a model, but if the value is changing on the web page frequently it may be worth it to the user). This would basically make it easier to maintain models whose values can be linked to web dashboards. It would also make it easier to create an emulation like simulation of a running production line.

Additionally, a built in browser with a sample tool that the user could use to auto-populate this function would be great...

A quick example of when this might be useful. A certain station on a production line might have a web dashboard that gives the weekly yield of the station. That value could be referenced from the webpage into the model, so that the model accurately reflects what is happening in the current week.

Ben Wilson commented Apr 7, '17 Ben Wilson edited Apr 7, '17

FlexSim Native (FlexScript and/or attached DLLs)

The functionality you're asking for is already available in the software, albeit from a lower level with more coding required than a single, simple function. However, the power is there, so if you wanted to wrap the existing functionality within a simple user command, you could do it today (minus the sampler idea). This could be a bridge between today, and some future date when FlexSim may include an easy web query command right out of the box.

For an overview of FlexSim's current web communication functionality, see these previous Q&As:

https://answers.flexsim.com/questions/29348/is-it-possible-to-use-the-server-sent-events-with.html

https://answers.flexsim.com/questions/21092/web-communication-within-flexsim.html

Ideally the site you're querying would have a nice API you could request against that would return just the data you're interested, preferably in some sort of well structured format like JSON, which you could then easily interpret within FlexSim.

Assuming the site you're querying lacks an easy API, you would then be in web scraping territory. The example model in one of my previous answers does just that using FlexScript's stringsearch command to drill down within the returned HTML to the data I'm interested in. Using FlexSim 2017's new dot-syntax, you'd have better, more power string commands (hello Regex :) with which to parse the HTML.

To really take this all the way, you'd want to find/build a CSS selector engine (probably like a C++ or other language program that you could compile to a DLL to attach to FlexSim). This would allow you to traverse a HTML DOM as you would within JavaScript. Or, better yet, do CSS-based DOM selection using FlexSim's built in WebKit web renderer, using jQuery. This method is outlined in another response to this idea.

Ben Wilson commented Apr 7, '17 Ben Wilson edited Aug 30, '17

FlexSim's Integrated Webkit

FlexSim includes Webkit, a fully featured web renderer. This gives you all the power of JavaScript and other web technologies, right from within FlexSim.

There are helper functions that allow a webpage running in FlexSim's Webkit to call a FlexScript function (fireFlexSimEvent()), and vice versa, a way for FlexScript to call JavaScript functions on the webpage (callwebscriptmethod()). In this manner you can bridge the two worlds.

So, using raw JavaScript, or JQuery or any other library of choice you could create a web application that gives you the power you want as far as querying external sites, parsing the data returned, etc. Then pass that data to your FlexSim model for whatever you need it for. This could be on demand (ie user must press a button, on the web page or within FlexSim proper, the query is made, and the data passed back to FlexSim), or tied to a trigger or other event.

I'll see about putting together an example and post it here.

· 2

Ben Wilson ♦♦ commented · Apr 07, 2017 at 10:41 PM

Attached is a model (usewebkittoscrapeformodelingdata.fsm) with a processor that gets a random process time from an external web site using WebKit. Here is how it works:

the only object with anything special is the Processor
it takes its process time from its procTime label
it has an OnEntry trigger where we get the next random process time
- the OnEntry trigger checks a global variable, webkitWindow, to see if it points to an open WebKit window. If not, it spawns the custom WebKit GUI I created using the GUI builder
- the OnEntry trigger, now that it has a good reference to the webkitWindow global variable, calls callwebscriptmethod() to call the JavaScript function returnWebData() that I coded into the webpage in the webkitWindow
- the OnEntry trigger then copies the value obtained from the web (that the JavaScript function returnWebData() writes to /Tools/webData) into the processor's procTime label

The custom GUI with a WebKit widget was built with the GUI builder, available from the Toolbox > Plus '+' button > Modeling Logic > Graphical User Interface. It is just a blank GUI with a single 'html' widget added.

In the html widget added to the GUI, you edit the html code. In mine, I included jQuery and coded a single function, returnWebData(). View the html node to view my HTML and JavaScript.

I also added an eventfunctions node with a webDataToFlexSim function in it. This FlexScript function is called from the JavaScript returnWebData() function via fireFlexsimEvent(), and is how the JavaScript passes the data from the remote web page back into the FlexSim model.

The principles displayed here could be simplified into easier user commands if desired, but this sample model shows all the underlying technology in action.

FYI, I also added an OnPreOpen to webkitDataGetter to close any other instances of my webkit view if any are open. Also a reset trigger on the processor to open up the webkit window, so that it is already spawned and ready for the first callwebscriptmethod() function call. If you try to call callwebscriptmethod() directly after createview(), the webkit window has not yet initialized, and so your call to callwebscriptmethod() dies silenty. Better to have the webkit view already open.

Respond back with any questions.

Update 2017-08-30:

-original sample model built with 17.0.0:

usewebkittoscrapeformodelingdata17-0-0.fsm

- newly created version of this sample model built with 16.0.0:

usewebkittoscrapeformodelingdata16-0-0.fsm

1 ·

usewebkittoscrapeformodelingdata16-0-0.fsm (19.1 KiB)

usewebkittoscrapeformodelingdata17-0-0.fsm (17.8 KiB)

2017-04-07-15-47-43-webkitdatagetter-gui-builder-t.png (27.3 KiB)

Chris Smith Ben Wilson ♦♦ commented · Apr 07, 2017 at 11:23 PM

Thanks Ben. This is awesome! I was trying to work through it and couldn't remember how to get a GUI to give me the information, but I knew that it was being done for portions of the software. This functionality helps us a lot with the maintainability of some of our models that are replicating production lines. Often the data often shifts between the time the model is made the time the data was gathered. Interfaces like this help to keep that data fresh. Thanks again.

0 ·

Michael Machado commented Apr 8, '17

Look that @Lucas Klein @Lucas Delago e @flavio.oliveira_de_brito

Idea

Function to sample a value from a webpage

3 Comments

Write a Comment

Things to know…

Your Opinion Counts

IDEA DETAILS

Related Ideas