When Is A Webpage Considered To Be "loaded", In The Presence Of JS Etc

Information: I have no knowledge of javascript. none. I'm curious if there's any way to determine when a webpage is completely loaded? Let's say I have a crawler, that uses webkit

Solution 1:

It's in general impossible to say whether a page that contains asynchronous, script-driven content is truly done loading. Aside from the fundamental issue of the halting problem, it's possible for scripts or plugins to register for periodic timer events and continue modifying or adding to the page indefinitely.

The approach I've usually seen for determining when a page is done loading is when the entire DOM has been loaded, resources (images, stylesheets, scripts, etc.) referenced directly from that DOM have been loaded, and all script code has been read and executed through once. Text emitted via document.write() is treated for this purpose as if it was directly included in the source HTML. If you're using QtWebKit, I believe this is the behavior you will see if you connect to the signal QWebPage::loadFinished(bool). (You can get the contained QWebPage from a QWebFrameusing the accessor page().)

Deferred actions set up by the script code, whether by timers, events waiting for load of other resources to complete, or what have you, is not counted; media players and other plugins may complicate things further because each media type or even player may have a different standard of what constitutes "loaded".

A number of recent JavaScript libraries exploit this behavior to improve perceived page load times by loading an incomplete page containing just the first screen's worth of content plus some script, and not actually beginning to load images and content "below the fold" until after the first screenful or so is done loading and rendering. It's not very friendly to automated tools, crawlers or those who consider JavaScript a privilege to be earned by trusted sites, though.