Intel Architecture Day 2021: Intel Unveils XeSS Image Upscaling Technology
by Ryan Smith on August 19, 2021 12:01 PM ESTAlongside a sneak peek at their forthcoming Xe-HPG architecture, the other big reveal today from Intel’s consumer graphics group comes from the software side of the business. Along with preparing Intel’s software stack for the 2022 launch of the first Arc products, the group has also been hard at work at their own take on modern, neural net-driven image upscaling techniques. The product of that research is Xe Super Sampling, or XeSS, which Intel is pitching as the best solution yet for high image quality and low processing cost image upscaling.
As briefly hinted at by Intel at the start of this week with the announcement of their Arc video card brand, the company has been developing their own take on image upscaling. As it turns out, they’re actually quite far along, so for today they’re not just announcing XeSS, but they are showing off footage of the technology as well. Even better, the initial version of the SDK will be shipping to game developers later this month.
XeSS (pronounced “ex-ee-ess-ess”) is, at a high level, a combination spatial and temporal AI image upscaling technique, which uses trained neural networks to integrate both image and motion data in order to produce a superior, higher resolution image. This is a field of research that has seen a great deal of research in the last half-decade, and was brought to the forefront of the consumer space a couple of years ago by NVIDIA with their DLSS technology. Intel’s XeSS technology, in turn, is designed to address similar use cases, and from a technical perspective ends up looking a lot like NVIDIA’s current DLSS 2.x technology.
As with NVIDIA and AMD, Intel is looking to have their cake and eat it too with respect to graphics rendering performance. 4K monitors are increasingly cheap and plentiful, but the kind of performance needed to natively render at 4K in modern AAA games is outside the reach of all but the most expensive discrete video cards. Ultimately looking to find ways to drive these 4K monitors with more modest video cards and without the traditional drop in image quality, this has led recent research into smart image upscaling techniques, and ultimately DLSS, FSR, and now XeSS.
In choosing their approach, Intel seems to have gone in a similar direction as NVIDIA’s second attempt at DLSS. Which is to say, they’re using a combination of spatial data (neighboring pixels) and temporal data (motion vectors from previous frames) to feed a (seemingly generic) neural network that has been pre-trained to upscale frames from video games. Like many other aspects of today’s GPU-related announcements, Intel isn’t going into too much detail here. So there are plenty of outstanding questions about how XeSS handles ghosting, aliasing, and other artifacts that can arise from these upscaling solutions. With that said, what Intel is promising isn’t something that’s out of their reach if they’ve really done their homework.
Meanwhile, given the use of a neural network to handle parts of the upscaling process, it should come as no surprise that XeSS is designed to leverage Intel’s new XMX matrix math units, which are making their debut in the Xe-HPG graphics architecture. As we saw in our sneak peek there, Intel is baking quite a bit of matrix math performance into their hardware, and the company is no doubt interested in putting it to good use. Neural network-based image upscaling techniques remain one of the best ways to use that hardware in a gaming context, as the workload maps well to these systolic arrays, and their high performance keeps the overall hit to frame rendering times small.
With that said, Intel has gone one step further and is also developing a version of XeSS that doesn’t require dedicated matrix math hardware. Owing to the fact that the installation base for their matrix hardware is starting from 0, that they’d like to be able to use XeSS on Xe-LP integrated graphics, and that they want do everything possible to encourage game developers to adopt their XeSS technology, the company is developing a version of XeSS that instead uses the 4-element vector dot product (DP4a) instruction. DP4a support is found in Xe-LP along with the past few generations of discrete GPUs, making its presence near-ubiquitous. And while DP4a still doesn’t offer the kind of performance that a dedicated systolic array does – or the same range of precisions, for that matter – it’s a faster way to do math that’s good enough for a somewhat slower (and likely somewhat duller) version of XeSS.
By offering a DP4a version of XeSS, game developers will be able to use XeSS on virtually all modern hardware, including competing hardware. In that respect Intel is taking a page from AMD’s playbook, targeting their own hardware while also letting customers of competitors benefit from this technology – even if by not quite as much. Ideally, that will be a powerful carrot to entice game developers to implement XeSS in addition to (or even in place of) other upscaling techniques. And while we won’t put the cart before the horse, should XeSS live up to all of Intel’s performance and image quality claims, then Intel would be in the unique position of being able to offer the best of both worlds: an upscaling technology with wide compatibility like AMD’s FSR and the image quality of NVIDIA’s DLSS.
As an added kicker, Intel is also planning on eventually open sourcing the XeSS SDK and tools. At this juncture there are no further details on their commitment – presumably, they want to finish and refine XeSS before releasing their tech to the world – but this would be a further feather in Intel’s cap if they can deliver on that promise as well.
In the meantime, game developers will be able to get their first look at the technology later this month, when Intel releases the initial, XMX-only version of the XeSS SDK. This will be followed by the DP4a version, which will be released later this year.
Finally, along with today’s technology disclosure Intel has also posted some videos of XeSS in action, using an early version of the technology baked into a custom Unreal Engine demo. The minute or so of footage shows several image quality comparisons between native 4K rendering and XeSS, which is upscaling from a native 1080p image.
As with all vendor demos, Intel’s should be taken with a suitable grain of salt. We don’t have any specific framerate data to go with, and Intel’s demo is fairly limited. In particular, I would have liked to see something with more object motion – which tends to be harder on these upscalers – but for now, it is what it is.
With all of that said, at first glance the image quality with XeSS is quite good. In some respects it’s almost suspiciously good; as Ian quickly picked up on, the clarity of the “ventilation” text in the above nearly rivals the native 4K renderer, making it massively clearer than the illegible mess on the original 1080p frame. This is solid evidence that as part of XeSS Intel is also doing something outside the scope of image upscaling to improve texture clarity, possibly by enforcing a negative LOD bias on the game engine.
In any case, like the rest of Intel’s forthcoming slate of GPU technologies, this won’t be the last we hear of XeSS. What Intel is demonstrating so far certainly looks promising, but it’s going to be their ability to deliver on those promises to both game developers and gamers that will matter in the end. And if Intel can indeed deliver, then they’re set to become a very welcome third player in the image upscaling technology race.
Performance Improvements For Intel’s Core Graphics Driver
Last but not least, while XeSS was the star of the show for Intel’s graphics software group, the company also delivered a brief update on the state of their core graphics driver that included a few interesting tidbits.
As a quick refresher, Intel these days is using a unified core graphics driver for their entire slate of modern GPUs. As a result, the work that has gone into the driver to prepare it for the launch of Xe-HPG can benefit existing Intel products (e.g. Xe-LP), and improvements made for current products get fed into the driver that will underpin future Xe-HPG products. While this is no different than how rival AMD operates, Intel’s expansion into discrete graphics has meant that the company has needed re-focus on the state of their graphics driver. What was good enough for an integrated product in terms of performance and features will not cut it in the discrete graphics space, where customers spending hundreds of dollars on a video card will have higher expectations on both fronts.
Of recent note, Intel has completed a significant overhaul of both its GPU memory manager and its shader compiler. The net impact of these changes includes improving game loading times by up to 25%, and improved the throughput of CPU-bound games by up to 18%. In the case of the former, by getting smarter about how and where they compile shaders – including eliminating redundant compilations and doing a better job at scheduling compiler threads. As well, Intel has also refactored parts of their memory management code to better optimize the VRAM utilization of their discrete graphics products. Intel of course just launched their first discrete product earlier this year with DG1, so this is a good example of the kind of additional optimization work facing Intel as they branch out into discrete graphics.
Finally, for features and functionality, the software group is also planning on releasing a suite of new driver features. Chief among these will be integrating all of their performance and overclocking controls directly into the company’s Graphics Command Center application. Intel will also be taking a page from NVIDIA and AMD’s current feature sets by adding new features for game streamers, including a fast stream capture path using Intel’s QuickSync encoder, automatic game highlights, and support for AI-assisted cameras. These features should be ready in time for the Intel Arc launch in Q1 of next year.
45 Comments
View All Comments
Oxford Guy - Tuesday, August 24, 2021 - link
‘Given that every game made in that period (anachronistic 'retro games' aside) intentionally took advantage of the intra-line, inter-line, and temporal blending effects of the CRTs they were viewed on, it is preferable to use one of the many CRT-emulation scalers available (not just 'scanline emulation, that's another anachronism that misses the point through misunderstanding’Yes. NES games, for instance, look vastly better for certain colors due to the influence of composite cabling — vs. the massively oversaturated colors outputted by most emulators. Browns are brown, not red. Etc.
The blurring makes things look more realistic, too.
Some software CRT emulators go too far with the blurring and some go too far with tube roundness distortion (considering the rather low distortion of a quality Trinitron and that even an ancient Zenith was available as a flat CRT). CRT quality varied and so did the calibration. I used to get guff for fixing oversaturated color from a man who was clearly partially colorblind. Neon TV colors aren’t just a symptom of the showroom.
The best CRT emulators should be adjustable.
Kurosaki - Sunday, August 22, 2021 - link
Dlss2.x and the like produces a blurry, choppy and artifacted image, we will never escape that. All for the purpose of getting to say "it runs in 4k" except it doesn't. It runs in a lower Res and is upscaled to a higher res, there's where we find the performance boost, a boost they could have managed to find via more RT-units for examlpe. AI upscaling is hogwash. But it seems to stick, just because you can claim it's an improving fairy dust mumbo jumbo thingy that makes the sausages not only prettier, but also run faster. AMAZING!mode_13h - Monday, August 23, 2021 - link
> All for the purpose of getting to say "it runs in 4k" except it doesn't.> It runs in a lower Res and is upscaled to a higher res
The way I look at it is like this: do you want 1440p on a 1440p display, or 1440p being nicely upscaled on 4k, so that you can gain the benefits of having a 4k monitor for things like web browsing and productivity tasks? And if you're going to buy a 4k monitor no matter what, do you want naive upscaling or something higher quality?
I recently upgraded from 1440p to 4k, myself, and I was surprised at the sheer amount of screen realestate I now have. Just for office and productivity purposes, it's like night-and-day. Since this was my work PC, I haven't run any games on it.
In the ideal world, everyone would just buy a couple RTX 3090 cards and could use native rendering. However, that's far from the reality. So, the question is to find the best compromise.
whatthe123 - Friday, August 20, 2021 - link
these techniques are already superior to TAA for smoothing out aliasing without destroying the entire image and hallucinate some sharpness. what exactly is the problem? there's nothing indicating that replacing the die space with shaders would give much of an uplift, especially with how memory limited modern GPUs are. AMD didn't ship RDNA2 with tensor, built it on a superior fab process, yet they perform similarly to nvidia's ampere that "wastes" space on tensor. So the chips are less versatile and not even appreciably faster, whats the point?Silver5urfer - Saturday, August 21, 2021 - link
The only advantage of DLSS is it's better than TAA disaster BS which is muddified garbage. I can be seen in RDR2 DLSS vs Native. Only MSAA is superior but sadly it taxes GPUs to a halt. Only powerful GPUs can handle that level of AA load. That said, yes it's unfortunate to see DLSS, FSR and this new XceSS bs take more space on the silicon.Intel is like doing marketing slides hyper mode just like CDPR. Esp 4K image quality upscale with AI lol, like what. Intel doesn't even do good GPU technology and it has been like that for a decade+ and magically they conjure Raja and his hype and create a GPU (talking about that Pointe Vecchio) and beat Nvidia top A100 and AMD's MI compute cards.
I'm not going to believe a single thing said by Intel in these past couple of days. Until the 3rd party reviews come. Also this BS GPU has to maintain proper FPS and stability on the past 2 decades of computer games AND Emulators. Intel doing that level of solid development ? Nope. Not a chance.
flyingpants265 - Wednesday, September 15, 2021 - link
The whole point is better performance per die space. If you had a 1440p screen and couldn't run the game, until now your only option was to run 1080p or at 85% resolution or something like that. Now you can run at "1440p" with supposedly twice the FPS, and still at better quality than your previous options.In this sense it actually does improve image quality, and seems to be a pro-midrange user feature. The visible image quality per clock cycle is increased.
mode_13h - Thursday, August 19, 2021 - link
> Ian quickly picked up on, the clarity of the “ventilation” text in the above nearly> rivals the native 4K renderer, making it massively clearer than the illegible mess
> on the original 1080p frame.
Yup. My eyes went right to that, as well.
> This is solid evidence that as part of XeSS Intel is also doing something outside
> the scope of image upscaling to improve texture clarity, possibly by enforcing a
> negative LOD bias on the game engine.
LOL wut? No. Pay close attention to the flow diagram in slide 92. See on the left side, where it says "Jitter"? That's the key. Super-resolution techniques *require* camera motion. By shifting the camera subtly, you can collect sub-pixel resolution, which is then usually projected into a high-resolution grid and interpolated. Their AI network can assist with the interpolation, to improve handling of tricky corner-cases.
Where this breaks down is if you have rapidly changing details that vary from one frame to the next, especially in ways that TAA-style motion vectors can't model. The frame-grab included is basically a best-case scenario, since it involves fixed geometry and presumably fixed lights.
SeannyB - Friday, August 20, 2021 - link
Maybe what was meant was a negative mipmap bias (which Nvidia calls "LOD bias" in their control panel IIRC).mode_13h - Saturday, August 21, 2021 - link
Okay, I take the point. Because, even supersampling a low-res texture is going to still result in a blur. So, you'd have to force the render path to use a higher-res version of the texture than it normally would, for that display resolution at that distance.Of course, we're presuming that it's even MIP-mapped. It's a little hard to tell, since they seem not to have used nearest-neighbor scaling of the 1080p version.
Zoolook - Monday, August 23, 2021 - link
Strange nobody mentions that the 1080p version looks like 640p, I don't think I've ever seen that low quality 1080p, the textures must be of extraordinary low res.