For those that might be interested ...<br><br>Recently for work I told a customer that wanted headerless execution (server side) of Javascript that I "Couldn't do it".<br>Now over the last few years I've looked into the subject a few times, always being quite pessimistic about it.<br>
<br>Over the last year I've been doing iPhone development and using the WebKit API - Using the Cocoa api you can "headerlessly" execute a web page by just not drawing the display buffer to the screen.<br>Awhile after this customer asked for completely programmtic headerless rendering I had some free time so I looked back into the issue. Knowing that the WebKit api can be accessed in such a way for this to be "theoretically" possible I pushed on...<br>
<br>And that is when I came across Nokia's API - the Qt ('cute') API has WebKit support amongst many other things, and actually after working with the iPhone SDK I feel pretty at home using it.<br>It's an event driven api, written in C++ ...<br>
<br>After researching the Qt library a bit, I found that it had Python bindings (even better) and it's most recent version supported a very modern version of WebKit similar to what Safari uses.<br><br>I then started looking for a way of actually 'drawing' the webpage, I knew that if I could draw to a Frame Buffer I could probably programatically on a server save and render images of web pages.<br>
<br>After quite a lot of work I came across the work of several other individuals that had used the same process involving Qt - Although the code was a bit poorly implemented.<br><br>I took concepts from several of the resources I found and wrote a python application that uses Xvfb (the X Virtual Frame Buffer) to render a web page, on a server.<br>
All you need is Python, pyQt4, Xvfb, (a script called Xvfb-run that supposedly comes with Xvfb, but I had to install manually) and a Linux distro in a 2.6 kernel variant.<br>I have it working very well on a Fedora 9 distro with the apps listed above, if anyone is interested in my code example, or further instructions let me know and I'll throw it up on a website.<br>
<br>And all of this was sadly to crawl google results (google re-orders and dynamically controls results with Javascript client side, including business results) <br>If anyone wants the PHP functions that parse google (using DOM) let me know ...<br>
Along with this (sadly - I'm not proud of this) I wrote a Google Image results parser...<br><br>Customers are mad, because the code will obviously break when google makes any changes but it was quite the experiment in code :) ... and I got paid for it.<br>
<br>The image code was to (help) find logo's for companies / organizations.<br>Garmin's data provider (InfoUSA) tracks 13,000 franchises/organizations the system had about an 80% success rate on those organizations - check out an example here.<br>
<br>(type in a company name, like 'mcdonalds')<br><a href="http://apginc.net:8380/binja/test_image_query.php">http://apginc.net:8380/binja/test_image_query.php</a><br>