Add In Some HBM, Optane

The other angle to Sapphire Rapids is the versions with HBM on board. Intel announced this back in June, but there haven’t been many details. As part of Architecture Day, Intel stated that that HBM versions of Sapphire Rapids would be made public, and be made socket compatible with standard Sapphire Rapids. The first customer of the HBM versions of SPR is the Argonne National Lab, as part of its Aurora Exascale supercomputer.

This diagram it showcases four HBM connections, one to each compute tile. Looking at the package, however, I don’t think that there’s realistically enough space unless Intel has commissioned some novel HBM that is long and narrow as it is in the diagram.

Even though Intel said that the HBM variants would be in the same socket, even their own slide from Hot Chips says different.

Here the package size with HBM says 100x57mm, compared to the SPR which is 78x57mm. So unless Intel is planning a reduced version for the 78x57mm socket, it's going to be in a different socket.

It is important to note that HBM will act in a similar capacity to Optane – either as an HBM flat mode with DRAM that equates the two, or as an HBM caching mode that acts similar to an L4 cache before hitting main memory. Optane on top of this can also be in a flat mode, a caching mode, or as a separate storage volume.

HBM will add power consumption to the package, which means we’re unlikely to see the best CPU frequencies paired with HBM if it is up against the socket limit. Intel has not announced how many HBM stacks or what capacities will be used in SPR, however it has said that they will be underneath the heatspreader. If Intel are going for a non-standard HBM size, then it’s anyone’s guess what the capacity is. But we do know that it will be connected to the tiles via EMIB.

A side note on Optane DC Persistent Memory – Sapphire Rapids will support a new 300 series Optane design. We asked Intel if this was the 200-series but using a DDR5 controller, and were told that no, this is actually a new design. More details to follow.

 

UPI Links

Each Sapphire Rapids Processor will have up to four x24 UPI 2.0 links to connect to other processors in a multi-socket design. With SPR, Intel is aiming for up to eight socket platforms, and in order to increase bandwidth has upgraded from three links in ICL to four (CLX had 2x3, technically), and moved to a UPI 2.0 design. Intel would not expand more on what this means, however they will have a new eight-socket UPI topology.


Current Intel Hypercube

Current eight-socket designs use a twisted hypercube topology: two groups of four form a box, and one pair is connected to the same vertex on the other set of four, while the second pair is inverted. Make sense? No, not really. Essentially each CPU is directly connected to three others, and the other four are two hops away. With the new topology, each CPU gets a direct connection to another, which moves the design more towards a fully connected topology, however exactly which CPU that connection should go to, Intel hasn’t stated yet.

Security

Intel has stated that it will be announcing full Security updates for SPR at a later time, however features like MKTME and SGX are key priorities.

 

Conclusions

For me, the improved cores, upgraded PCIe/DDR, and the ‘appears as a monolith’ approach are the highlights to date. However, there are some very obvious questions still to be answered – core counts, power consumption, how lower core counts would work (even suggestions that the LCC version is actually monolithic), and what the HBM enabled versions will look like. The HBM versions, with the added EMIB, are going to cost a good amount, which isn’t great at a time when AMD’s pricing structure is very competitive.

It is expected that when Sapphire Rapids is released, AMD will still be in the market with Milan (or as some are postulating, 3D V-Cache versions of Milan, but nothing is confirmed) and it won’t be until the end of 2022 when AMD launches Zen 4. If Intel can execute and bring SPR into the market, it will have a small time advantage in which to woo potential customers. Ice Lake is being sold on its specific accelerator advantages, rather than raw core performance, and we will have to wait and see if Sapphire Rapids can bring more to the table.

Intel moving to a tile/chiplet strategy in the enterprise has been expected for a number of years – at least on this side of the fence, ever since AMD made it work and scale beyond standard silicon limits, regardless of whatever horse-based binding agent is used between the silicon, Intel would have to go down this route. It has been delayed, mostly due to manufacturing but also optimizing things like EMIB which also takes time. EMIB as a technology is really impressive, but the more chips and bridges you put together, even if you have a 99% success rate, that cuts into yield. But that's what Intel has been working on, and for the enterprise market, Sapphire Rapids is the first step.

The March of More Silicon: Connectivity Matters
Comments Locked

94 Comments

View All Comments

  • Noctrn - Tuesday, August 31, 2021 - link

    It seems 99% likely that we're looking at 8 cores per tile for a max of 32 cores in the package. Intel has so far proven incapable of making a single piece of functional silicon with more than 8 large cores on it using anything smaller than 14nm.
  • Kamen Rider Blade - Tuesday, August 31, 2021 - link

    All the rumors for Sapphire Rapids is pointing to 14 Cores per Tile in a 4x4 Grid.
    2x of those nodes are for Vertical/Horizontal Interconnect management.
  • repoman27 - Tuesday, August 31, 2021 - link

    Not exactly. 4x4 grid with 15 CPU core tiles and one IMC tile for the two DDR5 controllers. See my post further up for additional details and link to actual die shots.
  • thestryker - Tuesday, August 31, 2021 - link

    Except for the entire Ice Lake Xeon line which scales up to 40 cores...
  • dullard - Tuesday, August 31, 2021 - link

    Psst, don't go putting facts into discussions! It throws the rest of us off.
  • Kamen Rider Blade - Tuesday, August 31, 2021 - link

    *technically DDR5 puts two 32-bit channels on a single module, but as yet the industry doesn’t have a term to differentiate between a module with one 64-bit memory channel on it vs. a module with two 32-bit memory channels on it. The word ‘channel’ has often been interchangeable with ‘memory slot’ to date, but this will have to change.

    What about calling them 2x 32-bit Sub-Channels?
  • TeXWiller - Tuesday, August 31, 2021 - link

    JEDEC calls them independent channels, that is 2 channels per module. Hence, an eight DIMM server board has a 16 channel memory system. I don't know why these terms are in flux everywhere.
  • Ian Cutress - Wednesday, September 1, 2021 - link

    Because people have been interchanging module and channel for years, and one module = one channel. The fact that DDR5 moves down to 32-bit channels from 64-bit channels means I'm going to be sprinkling the word controller around two be absolutely specific.
  • kpb321 - Tuesday, August 31, 2021 - link

    The lower core count versions will certainly be interesting.
    If the comments are correct and it is 14 cores per tile then you'd have 56 cores max. Certainly you could see them doing 52 and 48 core versions from die harvesting with 1 or 2 cores disabled per tile but the further below that you go the less it makes sense. On the other hand looking at the high level chip diagrams you pretty much have IO going around the entire outside of the cluster of tiles. I'm not sure how much smaller you can make the tiles and still have enough room for all the IO. What's the min core count going to look like? Are there going to be a 16 or 20 core version? Are they still going to use tiles for those or design a different monolithic die?
  • Spunjji - Friday, September 3, 2021 - link

    It would be logical to infer that they're going to need at least one more monolithic design, to allow for designs with fewer tiles with the same number of memory channels.

    Unless they just leave the lower-core designs with less memory bandwidth, which would be a product segmentation strategy of sorts, I guess?

Log in

Don't have an account? Sign up now