Generating 3D Models in Mobile: Sony’s 3D Creator Made Me a Bobbleheadby Ian Cutress on September 2, 2017 9:15 AM EST
- Posted in
- Trade Shows
- Snapdragon 835
- IFA 2017
In a show like IFA, it’s easy to get wide-eyed about a flashy new feature that is being heavily promoted but might have limited use. Normally, something like Sony’s 3D Creator app would fall under this umbrella – a tool that can create a 3D wireframe model of someone’s head and shoulders and then implement a 4K texture over the top. What is making me write about it is some of the implementation.
Normally in a single photo, without subsequent depth map data, creating a 3D model is difficult. Also, depth data would only show points directly in front of the camera – it says nothing about what is around the corner, especially when it comes to generating a texture from the image data to fit the model. With multiple photos, by correlating points (and perhaps using internal x/y/z sensor data), distances can be measured for identical points and a full depth map can be done taking the color data from the pixels and understanding which pixel would be where in that depth map allows the wireframe model to be textured.
For anyone who follows our desktop CPU coverage, we’ve actually been running a benchmark that does this for the last few years. Our test suite runs Agisoft Photoscan, which takes a set of high-quality images (usually 50+ images) of people, of items, of buildings and of landscapes, and builds a 3D textured model to be used in displays, games, and anything that wants a 3D model. Normally this benchmark is computationally expensive: Agisoft splits the work into four segments:
- Point Cloud Generation
- Mesh Building
- Texture Building/Skinning
Each of these segments has dedicated algorithms and the goal here is to compute as fast as possible. Some of the algorithms are linear and rely heavily on single thread performance, whereas others, such as Mesh Building, are very parallel which Agisoft implements via OpenCL. This allows any OpenCL connected accelerator, such as a GPU, to be able to increase the performance of this test. For low core count CPUs this is usually the longest part of the full benchmark, however the higher core count parts move into other bottlenecks, such as memory or cache.
So for our Agisoft run in those benchmarks, we use a 50 image set of a building with 4K images. We get the algorithm to select 50000 points from each image, and use those for the mesh building. We typically run it in OpenCL off mode, as we are testing the CPU cores, although Ganesh has seen some minor speedup on this test with Intel’s dual-core U-series CPUs when enabling OpenCL. A high end but low power processor, such as the Core i5-7500T, takes nearly 1500 seconds, or 25 minutes to run our test. We also see speed up based on cache sizes and DRAM frequency/latency, but major parts of the app either rely on single thread performance exclusively or multithread performance exclusively.
Sony’s way of creating the 3D head model involves panning the camera from one ear to the other, and then moving the camera around the head to generate finer detail and texture information. It does this all in-situ, computing on the fly and showing the results in real time back on the screen as the scan is being done. The whole process takes a minute, which compared to the method outlined above, is super quick. Now of course, Sony’s implementation is limited to just heads, rather than something about buildings, and we were told by Sony that their models are limited to 50000 polygons. During the demonstration I was given, I could see the software generating points on the head and it was obvious the number of points was in the hundreds in total, rather than the thousands per static image, so there is a perceptible difference in quality. But the Sony modeling implementation still gives a good visual output.
The smartphones from Sony that support this feature are the XZ series, which have Snapdragon 835 SoCs inside. Qualcomm is notoriously secretive about what is under the hood on their mobile chips, although features like the Hexagon DSP contained within the chip are announced. Sony would not state how they are implementing their algorithms, if they were leveraging a compute API from the Adreno GPU, a graphics API, the Kryo CPUs, or something from the special DSPs housed on the chip. It also leads two different questions: do the algorithms work on other SoCs, or can other Snapdragon 835 smartphone vendors develop their own equivalent application?
Sony’s goal is to allow users to implement their new facial model in applications that support personal avatars, or exporting to 3D printing formats for real-world creation of a user’s head. My mind instantly pointed to who would use something like this on scale: console players, specifically on the Xbox and Nintendo devices, or for special games such as NBA2k17. Given Sony’s exists in the console space with their own Playstation 4, one might expect them not to play with competitors, although the smartphone department is a different business unit (and other Snapdragon 835 players do not have a potential conflict). I was told by the booth demonstrator that he doesn’t know of any collaboration, which is unfortunate as I’d suspect this as being a good opening for this tool.
I’m trying to probe for more information, from Sony on the algorithm or Qualcomm on the hardware, because how the algorithm is implemented on the hardware is something I find interesting given how we’ve tested desktop CPUs in the past. It also puts the challenge to other smartphone vendors that use Snapdragon 835 (or other SoCs) to see if this is a feature that they might want to implement, or if there are apps that will implement this feature regardless of hardware.
Post Your CommentPlease log in or sign up to comment.
View All Comments
Threska - Saturday, September 2, 2017 - linkSince one can't patent math this should be showing up in more places.
wiineeth - Saturday, September 2, 2017 - linkSony made a new depth sensor for smart phone (https://www.sony.net/SonyInfo/News/Press/201706/17... which they are going to implement in their next Flagship. I think once it gets depth sensor the quality of scans could be awesome.
Absolutely love sony and how they innovate and think differently.
Dr. Swag - Saturday, September 2, 2017 - linkThink different
JudgeDredd - Saturday, September 2, 2017 - linkSerious question: Why is "AnandTech" still using a years old version of Agisoft Photoscan (v1.0) instead of the latest version which is faster thanks to improved multi-threading, has more processing functions running through the OpenCL path (which obviously can run on CPUs & GPUs) and now also has a CUDA path for NVidia GPUs ? This renders this specific benchmark totally useless. Don't you use the latest HW drivers available at the time of review? Yeah? Then why don't you use the latest version of a software/benchmark/whatever when there has been several new versions released in the past years?
wpwoodjr - Saturday, September 2, 2017 - linkTo maintain consistency with past reviews
JudgeDredd - Saturday, September 2, 2017 - linkNah..after a certain amount of time it's totally ridiculous to use an obsolete version of a software. Everybody would be up in arms if they did the same with the games used in their benchmarking suite. Will anybody accept if they only ever used the launch version of a game even 3 years later when said game has been patched dozens of times since its release? Nope
Ian Cutress - Saturday, September 2, 2017 - linkI got a new version of Photoscan a couple of weeks ago, and I've been on the road since. I plan to run both old and new side by side for a while until we get a good amount of data. But to answer the first question, we initially (from 2014) used a special version made just for us that allowed us to automate the testing. Since then we dropped out of contact as there wasn't any real need to update the benchmark (as some benchmarks like CineBench R15 are super old when Cinema 4D R19 is out), but it has got on my radar again, and thankfully our contact at Agisoft was still working there. Part of my to-do list after IFA is to script that up: the mechanism is slightly different to before, so I'll have to optimize around some upcoming projects.
JudgeDredd - Saturday, September 2, 2017 - linkThanks a lot for the update Ian. Can't wait for you guys to run some benches on the latest version.
Lolimaster - Saturday, September 2, 2017 - linkTo make intel look good (the excuse being their benchmark compare tool) focusing mostly on single thread/turbo.
Ian Cutress - Saturday, September 2, 2017 - link'Because Ian is a shill' - this week I'm shilling for Intel, eh? The other week it was AMD. If you're going to call out that we have bad practices, perhaps try something original.
Especially when I can state there are over 80 different CPU tests in our database. More than half have some level of threading/multithreading. Agisoft specifically has different stages of thread/frequency/cache/interconnect bottlenecks, as there are four separate stages, each with their own requirements.