New Testing Methodology

Every twelve to eighteen months it makes sense to upgrade our test beds in order to best represent what is available on the market. How the upgrade occurs depends on what is being tested, and in the case of our APU reviews it is clear that due to the wide range of graphics options available, as well as at different price points, that we have to adjust our gaming testing.

For 2015 our CPU performance testing regime remains untouched aside from the late 2014 addition of Linux-Bench for a glimpse into Linux based performance. On the gaming side, our games have been updated to the following:

  • Alien Isolation (First Person Survival-Horror)
  • Total War: Attila (Strategy)
  • Grand Theft Auto V (Open World Sandbox)
  • GRID: Autosport (Driving)
  • Middle-Earth: Shadows of Mordor (Action-Adventure)

Because budgets for gaming graphics cards can vary, or users decide to keep the same card for several generations, we will be testing each of these titles in both low, medium and high end graphics setups. This means we can see where the bottlenecks are for CPU performance at each stage. We have also been able to source both AMD and NVIDIA cards for most of these areas, should one side of the equation scale more than the other.

The GPU sections are split into three based on where they fit in their independent stacks rather than for direct competition:

Low-end:
 - Integrated Graphics
 - ASUS R7 240 2GB DDR3 ($70)
 - Dual Graphics (where applicable)

Mid-range:
 - MSI GTX 770 Lightning 2GB ($245-$255 on eBay/Amazon, $330 new)
 - MSI R9 285 Gaming 2GB ($240)

High-end:
 - ASUS GTX 980 Strix 4GB ($560)
 - MSI R9 290X Gaming LE 4GB ($380)

On the low end, we have selected settings in order to make the current best integrated graphics solutions score between 45 and 60 frames per second. On the mid-range and high-end, we typically pull out 1080p maximum settings or almost-maximum.

The Shadows of Mordor (SoM) benchmark throws up a little interesting teaser as well due to the use of its Dynamic Super Resolution technique. This allows us to render at 3840x2160 (Ultra-HD, or ‘4K’) with our settings despite using a 1080p monitor. As a result, we also test SoM at 4K ultra with our mid-range and high-end graphics setups.

For the high-end setups, as we have managed to source 2 cards of each, means that where applicable we can test both SLI and Crossfire setups. We apply this to Shadows of Mordor at 4K as an extra data point.

For clarity, this means:

  Integrated
R7 240 2GB
Dual Graphics
GTX 770 2GB
R9 285 2GB
GTX 980 4GB
R9 290X 4GB
Alien Isolation 720p Ultra 1080p Ultra 1080p Ultra
Average Frame Rate Average Frame Rate Average Frame Rate
Total War: Attila 720p Performance 1080p Quality 1080p Quality
Average Frame Rate Average Frame Rate Average Frame Rate
Grand Theft Auto V 720p Low 1080p Very High 1080p Very High
Average Frame Rate
%FPS <60 FPS
Average Frame Rate
%FPS <60 FPS
Average Frame Rate
%FPS <60 FPS
GRID: Autosport 1080p Medium 1080p Ultra 1080p Ultra
Average Frame Rate
Minimum Frame Rate
Average Frame Rate
Minimum Frame Rate
Average Frame Rate
Minimum Frame Rate
Middle-Earth:
Shadows of Mordor
720p Low
 
1080p Ultra
4K Ultra
1080p Ultra
4K Ultra
4K SLI/CFX
Average Frame Rate
Minimum Frame Rate
Average Frame Rate
Minimum Frame Rate
Average Frame Rate
Minimum Frame Rate

For drivers, we locked down the 350.12 WHQL versions from NVIDIA soon after the launch of GTA V. Similarly, the 15.4 Beta drivers from AMD are also being used. These will remain consistent over the next 12-18 months until the next update.

All of our old (and new) benchmark data, both for CPU and graphics performance, can be found in our benchmark database, Bench.

We have a variety of benchmarks here, including legacy benchmarks such as CineBench 11.5 and TrueCrypt, which are not published in the main review. All CPUs/APUs that have been tested in our new 2015 style will be labeled in the dropdown menus by having its launch price listed, e.g. ’AMD A10-7850K (95W, $173)’. With any luck over the course of the next six months we will be adding new data and re-testing older processors for the database in order for our readers to compare old with new.

AMD A8-7650K Review AMD A8-7650K Test Setup, Overclocking
Comments Locked

177 Comments

View All Comments

  • jabber - Tuesday, May 12, 2015 - link

    Exactly.

    "Yayyyy I use 7Zip all day long! "

    Said no one...ever.

    I don't even know why people still compact files? Are they still using floppies? Man, poor devils.
  • Gigaplex - Tuesday, May 12, 2015 - link

    I've been getting BSODs lately due to a bad Windows Update. The Microsoftie asked me to upload a complete memory crash dump. There's no way I can upload a 16GB dump file in a reasonable timeframe on a ~800kbps upload connection, especially when my machine BSODs every 24 hours. Compression brought that down to a much more manageable 4GB.
  • galta - Tuesday, May 12, 2015 - link

    So it makes perfect sense for yoy to stay with AMD...
  • NeatOman - Wednesday, May 13, 2015 - link

    I use it everyday :( rocking a FX-8320@4.5Ghz for the last 3 years.. I picked it up for $180 with the CPU and!! Motherboard. I was about to pick up a 3770k too, saved about $200 but am about 15-20% down on performance. And if you're worried about electrical cost, you're walking over dollars to pick up pennies.

    I do it to send pictures of work I do, and a good SSD is key :)
  • UtilityMax - Tuesday, May 12, 2015 - link

    If you look at the WinRAR benchmark, then that result strongly suggests that WinRAR is multi-threaded. I mean, two core two thread Pentium is clearly slower than the two core but four thread Core i3, and quad-core i5 is clearly faster than Core i3, and Core i7 with its eight threads is clearly faster than Core i5. Hence galta's comment that AMD FX with 8 cores is probably even faster, but he says that this is not normal usage.
  • TheJian - Thursday, May 14, 2015 - link

    There is an actual checkbox in winrar for multithreading for ages now. ROFL. 95% of usenet uses winrar, as does most of the web. That doesn't mean I don't have 7zip installed, just saying it is only installed for the once in 6 months I find a file that uses it.

    You apparently didn't even read what he said. He clearly states he's using winrar and finds FX is much faster using 8 cores of FX in winrar. You're like, wrong on all fronts. He's using winrar (can't read?), he's using FX (why suggest it? Can't read?) AND there is a freaking check-box to turn on multi-threading in the app. Not sure whether you're shilling for AMD here or 7zip, but...jeez.
  • galta - Saturday, May 16, 2015 - link

    Last AMD CPU I had was the old and venerable 386DX@40Mhz. Where any of you alive back in the early 90s?
    Ever since I've been using Intel.
    Of course there were some brief moments during this time when AMD had the upper hand, but the last it happened was some 10 years ago when Athlom and its two cores were a revolution and smashed Pentium Ds. It's just that during that particular moment I wasn't looking for an upgrade so I've Intel ever since.
    Having said that, I have to add that I don't understand why we are spending so much time discussing compression of files.
    Of course that the more cores you have the better, and AMD happens to have the least expensive 8 core processor on the market, BUT most users spend something like 0.15% of their time compressing files, making this particular shinny performance irrelevant for most of us.
    Because most of other software does not scale so good in multithreading (and for games, it has nothing to do with DX12 as someone said elsewhere), we are most likely interested in performance per core, and Intel clearly has the lead here.
  • NeatOman - Wednesday, May 13, 2015 - link

    Truth is the average user won't be able to tell the difference on a system with a i3 running on a ssd and a A6-7400k on a ssd or even a A10-7850k which would be more direct competition to the i3. I build about 2-4 new Intel and AMD systems a month and the only time I myself notice is when I'm setting them up, after that they all feel relitivly close in speed due to the SSD which was the largest bottleneck to have been overcome in the last 10 years.

    So Intel might feel snappier but are still not much faster in day to day use of heavy browsing and media consumtion as long as you have enough ram and a decent SSD.
  • mapesdhs - Tuesday, May 12, 2015 - link

    Ian Cutress wrote:
    > Being a scaling benchmark, C-Ray prefers threads and seems more designed for Intel."

    It was never specifically designed for Intel. John told me it was, "...an extremely
    small program I did one day to figure out how would the simplest raytracer program
    look like in the least amount of code lines."

    The default simple scene doesn't make use of any main RAM at all (some systems
    could hold it entirely in L1 cache). The larger test is more useful, but it's still wise to
    bare in mind to what extent the test is applicable to general performance comparisons.
    John confirmed this, saying, "This thing only measures 'floating point CPU performance'
    and nothing more, and it's good that nothing else affects the results. A real rendering
    program/scene would be still CPU-limited meaning that by far the major part of the time
    spent would be CPU time in the fpu, but it would have more overhead for disk I/O, shader
    parsing, more strain for the memory bandwidth, and various other things. So it's a good
    approximation being a renderer itself, but it's definitely not representative."

    As a benchmark though, c-ray's scalability is incredibly useful, in theory only limited by
    the no. of lines in an image, so testing a system with dozens of CPUs is easy.

    Thanks for using the correct link btw! 8)

    Ian.

    PS. Ian, which c-ray test file/image are you using, and with what settings? ie. how many
    threads? Just wondered if it's one of the stated tests on my page, or one of those defined
    by Phoronix. The Phoronix page says they use 16 threads per core, 8x AA and 1600x1200
    output, but not which test file is used (scene or sphfract; probably the latter I expect, as
    'scene's incredibly simple).
  • Ian Cutress - Tuesday, May 12, 2015 - link

    It's the c-ray hard test on Linux-Bench, using

    cat sphfract | ./c-ray-mt -t $threads -s 3840x2160 -r 8 > foo.ppm

    I guess saying it preferred Intel is a little harsh. Many programs are just written the way people understand how to code, and it ends up sheer luck if they're better on one platform by default than the other, such as with 3DPM.

Log in

Don't have an account? Sign up now