Idea

Chris Smith suggested Apr 07 2017 at 4:19 PM Ben Wilson edited Aug 30 2017 at 2:21 PM

Function to sample a value from a webpage

I think this is actually something that would be extremely useful for a lot of users, and also fairly easy to implement due to development in the software in the last few years. The idea would be something like webvalue(String - website url, String - HTML element id, Boolean - on Reset). The first parameter would be for the url of the site with the value. The second would be for the id of the element that contains the string. The third would be a value that determines if the value is obtained from the website on reset only and stored in a label to be used for future calls (if the function is called during the model run it would just reference the value in the label) or if a request is done each time it is called (this could really slow down a model, but if the value is changing on the web page frequently it may be worth it to the user). This would basically make it easier to maintain models whose values can be linked to web dashboards. It would also make it easier to create an emulation like simulation of a running production line.

Additionally, a built in browser with a sample tool that the user could use to auto-populate this function would be great...

A quick example of when this might be useful. A certain station on a production line might have a web dashboard that gives the weekly yield of the station. That value could be referenced from the webpage into the model, so that the model accurately reflects what is happening in the current week.

3 Comments

Ben Wilson commented Apr 07 2017 at 5:10 PM Ben Wilson edited Apr 07 2017 at 10:45 PM

FlexSim Native (FlexScript and/or attached DLLs)

The functionality you're asking for is already available in the software, albeit from a lower level with more coding required than a single, simple function. However, the power is there, so if you wanted to wrap the existing functionality within a simple user command, you could do it today (minus the sampler idea). This could be a bridge between today, and some future date when FlexSim may include an easy web query command right out of the box.

For an overview of FlexSim's current web communication functionality, see these previous Q&As:

https://answers.flexsim.com/questions/29348/is-it-possible-to-use-the-server-sent-events-with.html

https://answers.flexsim.com/questions/21092/web-communication-within-flexsim.html

Ideally the site you're querying would have a nice API you could request against that would return just the data you're interested, preferably in some sort of well structured format like JSON, which you could then easily interpret within FlexSim.

Assuming the site you're querying lacks an easy API, you would then be in web scraping territory. The example model in one of my previous answers does just that using FlexScript's stringsearch command to drill down within the returned HTML to the data I'm interested in. Using FlexSim 2017's new dot-syntax, you'd have better, more power string commands (hello Regex :) with which to parse the HTML.

To really take this all the way, you'd want to find/build a CSS selector engine (probably like a C++ or other language program that you could compile to a DLL to attach to FlexSim). This would allow you to traverse a HTML DOM as you would within JavaScript. Or, better yet, do CSS-based DOM selection using FlexSim's built in WebKit web renderer, using jQuery. This method is outlined in another response to this idea.

Ben Wilson commented Apr 07 2017 at 5:15 PM Ben Wilson edited Aug 30 2017 at 2:21 PM

FlexSim's Integrated Webkit

FlexSim includes Webkit, a fully featured web renderer. This gives you all the power of JavaScript and other web technologies, right from within FlexSim.

There are helper functions that allow a webpage running in FlexSim's Webkit to call a FlexScript function (fireFlexSimEvent()), and vice versa, a way for FlexScript to call JavaScript functions on the webpage (callwebscriptmethod()). In this manner you can bridge the two worlds.

So, using raw JavaScript, or JQuery or any other library of choice you could create a web application that gives you the power you want as far as querying external sites, parsing the data returned, etc. Then pass that data to your FlexSim model for whatever you need it for. This could be on demand (ie user must press a button, on the web page or within FlexSim proper, the query is made, and the data passed back to FlexSim), or tied to a trigger or other event.

I'll see about putting together an example and post it here.

· 2

Ben Wilson ♦♦ commented · Apr 07 2017 at 10:41 PM

Attached is a model (usewebkittoscrapeformodelingdata.fsm) with a processor that gets a random process time from an external web site using WebKit. Here is how it works:

the only object with anything special is the Processor
it takes its process time from its procTime label
it has an OnEntry trigger where we get the next random process time
- the OnEntry trigger checks a global variable, webkitWindow, to see if it points to an open WebKit window. If not, it spawns the custom WebKit GUI I created using the GUI builder
- the OnEntry trigger, now that it has a good reference to the webkitWindow global variable, calls callwebscriptmethod() to call the JavaScript function returnWebData() that I coded into the webpage in the webkitWindow
- the OnEntry trigger then copies the value obtained from the web (that the JavaScript function returnWebData() writes to /Tools/webData) into the processor's procTime label

The custom GUI with a WebKit widget was built with the GUI builder, available from the Toolbox > Plus '+' button > Modeling Logic > Graphical User Interface. It is just a blank GUI with a single 'html' widget added.

In the html widget added to the GUI, you edit the html code. In mine, I included jQuery and coded a single function, returnWebData(). View the html node to view my HTML and JavaScript.

I also added an eventfunctions node with a webDataToFlexSim function in it. This FlexScript function is called from the JavaScript returnWebData() function via fireFlexsimEvent(), and is how the JavaScript passes the data from the remote web page back into the FlexSim model.

The principles displayed here could be simplified into easier user commands if desired, but this sample model shows all the underlying technology in action.

FYI, I also added an OnPreOpen to webkitDataGetter to close any other instances of my webkit view if any are open. Also a reset trigger on the processor to open up the webkit window, so that it is already spawned and ready for the first callwebscriptmethod() function call. If you try to call callwebscriptmethod() directly after createview(), the webkit window has not yet initialized, and so your call to callwebscriptmethod() dies silenty. Better to have the webkit view already open.

Respond back with any questions.

Update 2017-08-30:

-original sample model built with 17.0.0:

usewebkittoscrapeformodelingdata17-0-0.fsm

- newly created version of this sample model built with 16.0.0:

usewebkittoscrapeformodelingdata16-0-0.fsm

1 ·

usewebkittoscrapeformodelingdata16-0-0.fsm (19.1 KiB)

usewebkittoscrapeformodelingdata17-0-0.fsm (17.8 KiB)

2017-04-07-15-47-43-webkitdatagetter-gui-builder-t.png (27.3 KiB)

Chris Smith Ben Wilson ♦♦ commented · Apr 07 2017 at 11:23 PM

Thanks Ben. This is awesome! I was trying to work through it and couldn't remember how to get a GUI to give me the information, but I knew that it was being done for portions of the software. This functionality helps us a lot with the maintainability of some of our models that are replicating production lines. Often the data often shifts between the time the model is made the time the data was gathered. Interfaces like this help to keep that data fresh. Thanks again.

0 ·

Michael Machado commented Apr 08 2017 at 9:14 PM

Look that @Lucas Klein @Lucas Delago e @flavio.oliveira_de_brito

______

Cookie preferences

Your privacy is important to us and so is an optimal experience. To help us customize information and build applications, we collect data about your use of this site.

May we collect and use your data?

Learn more about the Third Party Services we use and our Privacy Statement.

Strictly necessary – required for our site to work and to provide services to you

These cookies allow us to record your preferences or login information, respond to your requests or fulfill items in your shopping cart.

YES

Improve your experience – allows us to show you what is relevant to you

These cookies enable us to provide enhanced functionality and personalization. They may be set by us or by third party providers whose services we use to deliver information and experiences tailored to you. If you do not allow these cookies, some or all of these services may not be available for you.

YES

Customize your advertising – permits us to offer targeted advertising to you

These cookies collect data about you based on your activities and interests in order to show you relevant ads and to track effectiveness. By collecting this data, the ads you see will be more tailored to your interests. If you do not allow these cookies, you will experience less targeted advertising.

YES

Are you sure you want a less customized experience?

We can access your data only if you select "yes" for the categories on the previous screen. This lets us tailor our marketing so that it's more relevant for you. You can change your settings at any time by visiting our privacy statement

Your experience. Your choice.

We care about your privacy. The data we collect helps us understand how you use our products, what information you might be interested in, and what we can improve to make your engagement with Autodesk more rewarding.

May we collect and use your data to tailor your experience?

Explore the benefits of a customized experience by managing your privacy settings for this site or visit our Privacy Statement to learn more about your options.

Idea

Function to sample a value from a webpage

3 Comments

Things to know…

Your Opinion Counts

IDEA DETAILS

Related Ideas

Cookie preferences

Strictly necessary – required for our site to work and to provide services to you

Improve your experience – allows us to show you what is relevant to you

Customize your advertising – permits us to offer targeted advertising to you

Strictly necessary – required for our site to work and to provide services to you

Improve your experience – allows us to show you what is relevant to you

Customize your advertising – permits us to offer targeted advertising to you

Are you sure you want a less customized experience?

Your experience. Your choice.

Idea

Function to sample a value from a webpage

3 Comments

Things to know…

Your Opinion Counts

IDEA DETAILS

Related Ideas

Cookie preferences

Strictly necessary – required for our site to work and to provide services to you

Improve your experience – allows us to show you what is relevant to you

Customize your advertising – permits us to offer targeted advertising to you

THIRD PARTY SERVICES

Strictly necessary – required for our site to work and to provide services to you

Improve your experience – allows us to show you what is relevant to you

Customize your advertising – permits us to offer targeted advertising to you

Are you sure you want a less customized experience?

Your experience. Your choice.