1. For a while now we've been keeping track of mobile browser performance using two relatively popular JavaScript heavy benchmarks that are a regular fixation in our smartphone reviews. If you've read any of those reviews, you should immediately be able to name them - SunSpider and RightWare's BrowserMark. Tracking JavaScript performance thus far has helped us codify and track SoC performance, but really understanding and quantifying overall browsing smoothness has remained a somewhat more daunting task. Moreover, JavaScript performance is just one facet of many that contribute to the user's perception of browsing speed.

Real web browsing performance is a unique combination of system performance, the Android browser itself, and what contributions or customizations (if any) the OEM has made in the shipping browser build. This is to say nothing of the numerous third party browsers out there. Qualcomm's Innovation Center recently made public a tool for gauging overall browser performance that it has used for a while to checkpoint and gauge performance in house and in collaboration with OEMs. We've used a subtest or two from it in a few reviews already, and it's named Vellamo.


Left to right: T-Mobile SGS2, Motorola Atrix, AT&T SGS2, HTC Sensation, LG Optimus 3D, Xiaomi Mi-One

First off, Qualcomm sent us five dual-core smartphones, almost all of which we've seen and tested before. Just to be sure, we verified that nothing has been tampered with and that all were running the latest shipping OEM ROM and started fresh with a complete reset. We also added another, the T-Mobile SGS2 bringing that total to six devices. 

Smartphones
  T-Mobile SGS2 Motorola Atrix AT&T SGS2 HTC Sensation LG Optimus 3D Xiaomi Mi-One
SoC 1.5 GHz APQ8060 1.0 GHz Tegra 2 1.2 GHz Exynos 4210 1.2 GHz MSM8260 1.0 GHz OMAP4430 1.5 GHz MSM8260
RAM 1 GB LPDDR2 1 GB LPDDR2 1 GB LPDDR2 768 MB LPDDR2 512 MB LPDDR2 1 GB LPDDR2
OS Android 2.3 Android 2.3 Android 2.3 Android 2.3 Android 2.2 Android 2.3

The selection we've made isn't meant to be totally comprehensive, but gives an idea for just how much performance varies from one OEM to another, largely as a function of changes made to the browser on their respective versions of Android. These devices are all dual core and what I'd personally consider of the latest generation (dual core, Android 2.3, e.t.c.). We're going to be updating our Smartphone Bench category soon (something I've been behind on, as of late) with Vellamo scores for devices as well. 

If you've been reading our reviews, basically every device except for the T-Mobile SGS2 (which we're going to review shortly) and the Xiaomi Mi-One should be relatively familiar. The Mi-One is from a Chinese OEM we haven't seen devices from before called Xiaomi, but is a surprisingly decent performer and has one of the most unique (and surprisngly speedy) skins I've ever seen on Android. 

Vellamo Test Breakdown

Vellamo's test are divided into a few categories, and start with Rendering tests which stress the performance of HTML5 <canvas> elements. Although Vellamo at present exists on Android as a standalone application, the tests themselves are entirely HTML and JavaScript and run inside Android WebView views which use the Android WebKit browser and whatever optimizations have been made by the OEM. Vellamo as an application simply exists around the tests to measure FPS, dropped frames, time, and report scores. 

First among these tests is See the Sun Canvas, which renders increasingly demanding stages of a 2D rotating butterfly in an HTML5 <canvas> element. The first stage starts out with cubes, then moves to circles, then circles of different color, and finally alpha-blended for the maximum load. 

Vellamo - See The Sun Canvas

The Xiaomi Mi-One is a new device for us, but delivers strikingly fast performance thanks to the work that Xiaomi has done to the stock Android browser. I found the SGS2's backing store pretty decent as well, but you can see how much difference there is in performance between the two despite packing the same 1.5 GHz dual core snapdragon (APQ8060 and MSM8260 differ solely in baseband - APQ lacks one).

The next test is Deep Sea Canvas which tests almost the same parameters as See the Sun, except tests are non-sequential here and are an attempt to fully load the platform.

Vellamo - Deep Sea Canvas

Interestingly enough, Exynos 4210 comes out on top of the pack. 

Next up is Aquarium canvas which tests the number of HTML5 canvas objects that can be rendered simultaneously. In addition, this test runs in a native Android Java implementation for comparison with the HTML5 <canvas> implementation in the browser. 

Vellamo - Aquarium Canvas

Vellamo calculates a score for each subtest alongside FPS and dropped frames, and we've chosen to report FPS wherever possible, but this test makes more sense being reported in base score. Interestingly enough the Sensation 4G edges out the faster T-Mobile SGS2.

Pixel Blender is next which draws multiple images over a static background and applies alpha blending. Vellamo calculates the graphics throughput of the browser in pixels per second to arrive at memory bandwidth for blitting and blending. 

Vellamo - Pixel Blender

Again there's a pretty shocking disparity between Samsung's browser and the Xiaomi, we're reporting the base score here as well.

Next up are the very familiar JavaScript VM tests which we've taken a look at many times before in numerous reviews. In Android's case, every browser is using the V8 JavaScript engine. First among these tests is SunSpider 0.9.1. We started using 0.9 and have moved to 0.9.1 for most things now, which still consists of a number of relatively short tests run 10 times. The final product is a completion time in milliseconds, and thusly lower is better. SunSpider is developed by the WebKit team of course.

Vellamo - SunSpider 0.9.1

SunSpider is to WebKit as the V8 benchmark suite is to, well, V8. It's another JavaScript benchmark which consists of a number of beefy tests (raytracing, regex, math, e.t.c.) and also exercises the VM's garbage collector. Vellamo runs v6 of the V8 benchmark which is latest.

Vellamo - V8 Benchmark (v6)

Next is SurfWax Binder, which measures the efficiency of JavaScript bindings code which links the VM to the rest of the browser's core and rendering core. It consists of a long series of nested function calls, which Vellamo measures in number of accesses per second. 

Vellamo - Surf Wax Binder

Where I feel Vellamo really shines however is this next suite, which we've shown scores from before, and this is the User Experience Test category. This is the area that until now has proven challenging to qualitate without using video or other subjective measures - things like scrolling performance and responsiveness. 

There are three tests here, starting with Ocean Flinger, which measures user experience when scrolling a page in a flinging manner. This is a test which consists of sample text, images, CSS, and JavaScript. ImageFlinger is a similar test which takes a dense grid of repeated JPEG images and scrolls it. Finally Text Flinger does the same thing, but with a repeated grid of densely packed text. We've reported Ocean Flinger in other tests before since it really is a combination of images and text (like we've already stated).

Vellamo - Ocean Flinger

Vellamo - Image Flinger

Vellamo - Text Flinger

I have to admit that Vellamo's numbers here closely mirror my own subjective scrolling performance perceptions when using the respective devices. The three scrolling tests represent a way to finally codify one of the more elusive (and yet easily perceptible) browser performance aspects. 

The final test is a networking loader suite which loads a page stored locally and times how long it takes in milliseconds. This is done in both hot and cold mode (with the caches cleared and not cleared) and is designed to test local performance. 

Vellamo - Networking Loader

Finally Vellamo also reports a total score which is a summation of all the base scores from each subtest. 

Vellamo Overall Score

The total score subjectively lines up with where I would mentally rank the devices - we've seen how competent Samsung's browser optimizations are (especially their browser's backing store), and how much sometimes HTC's Sense tends to weigh the platform down. Motorola's browser on the Atrix and LG's on the Optimus 3D take a more stock approach, and the Xiaomi surprises us with some serious optimizations from a company I doubt many in the US have heard of.

We're going to be adding (and simultaneously giving the same treatment) a few more benchmarks in preparation for a revamped testing suite for 2012 and an updated smartphone Bench as well. 

Comments Locked

82 Comments

View All Comments

  • admiralpumpkin - Wednesday, November 16, 2011 - link

    I'd like to see how the iP4S compares.

    Is that comparison possible?

    Yes, I know it's an Android app at the moment. How about an iOS app? Or a web app version?
  • tipoo - Wednesday, November 16, 2011 - link

    You can do the individual benchmarks like Browsermark and Sunspider
  • tipoo - Wednesday, November 16, 2011 - link

    My stock Nexus S got 609. Not too far behind the dual core Atrix, I guess Android's browser doesn't' take much advantage of more cores yet. I use Opera anyways, its GPU accelerated and you can really tell on image heavy pages (try zooming around engadget [desktop] while it loads on the stock browser and then Opera).
  • tipoo - Wednesday, November 16, 2011 - link

    *Opera mobile, not mini.
  • tiredad - Thursday, November 17, 2011 - link

    For info: my Nexus S with MIUI scored 879 which surprised the hell out of me.
  • tipoo - Saturday, January 14, 2012 - link

    Wow, 890 on cm7 with Trinity kernel...makes you wonder why the stock kernel can't be faster, mines not even overclocked.
  • The Saint - Wednesday, November 16, 2011 - link

    Wow, I'm shocked that both Anandtech and all of the commentators here don't seem to be familiar with MIUI.

    MIUI is the second most prominent Android ROM around, second only to Cyanogenmod. It's renowned for its good performance, and its UI which is heavily revamped to both (1) have a lot of improvements in ease of use and (2) unfortunately be a bit of a ripoff of iOS.

    Xiaomi is the company that's because MIUI, and they Mi-One is the first device they've launched that comes stock with MIUI. Furthermore, it has a bunch of features specifically applicable to enthusiasts that make it impossible to brick and easy to try different ROMs on.
  • Legendarydust - Wednesday, November 16, 2011 - link

    Tested my MyTouch 4g running CM 7.1 running stock 1ghz and overclocked 1.5ghz scores are below

    Stock 1.0ghz/Overclock 1.5ghz
    See the Sun Canvas: 6.98 / 9.26
    Deep Sea Canvas: 4.54 / 7.35
    Aquarium Canvas: 101.69 / 122.85
    Pixel Blender: 56.33 / 58.41
    Sun Spider: 3142 / 2163
    V8 Benchmark: 448 / 624
    Surf Wax Binder: 81.31 / 112.35
    Ocean Flinger: 32.23 / 41.32
    Image Flinger: 10.7 / 12.74
    Text Flinger: 31.9 / 40.27
    Networking Loader: 82.43 / 108.15
    Overall Score: 682 / 879

    Not bad overall for an Android device that is over a year old
  • fabarati - Wednesday, November 16, 2011 - link

    Just ran on my LG Optimus 2x (GR4/HP SR1), on Edge when testing.

    See the Sun Canvas: 16.46

    Deep Sea Canvas: 10.81

    Aquarium Canvas: 43.99

    Pixel Blender: 35.0

    Surf Wax Binder: 110.33

    Sun Spider: 2706

    V8 Benchmark: 296.0

    Ocean Flinger: 43.69

    Image Flinger: 13.59

    Text Flinger: 24.76

    Networking Loader: 97.82

    Overall Score: 731

    Link to a screenshot of the charts : http://i.imgur.com/IWVU0.jpg

    It seems that where my phone falls down is rendering. I'm guessing bad browser optimization?
  • fabarati - Wednesday, November 16, 2011 - link

    And at 1.2GHz the overall score is 926 points. Still with rendering issues,

    See the Sun Canvas: 13.38

    Deep Sea Canvas: 13.45

    Aquarium Canvas: 45.53

    Pixel Blender: 41.75

    Surf Wax Binder: 139.11

    Sun Spider: 1996

    V8 Benchmark: 718

    Ocean Flinger: 49.41

    Image Flinger: 14.29

    Text Flinger: 26.95

    Networking Loader: 155.54

    Overall Score: 926

    http://i.imgur.com/GFyad.jpg

Log in

Don't have an account? Sign up now