Like many system architects around the world, we had high hopes for 3D XPoint, a variant of phase-change memory (PCM) that was released to much fanfare in July 2015 after years of joint development by Intel and Micron. This represents a new type of memory in the system hierarchy that sits between main memory and Tier 1 storage, and promises to enable larger, cheaper main memory and faster Tier 1 memory.
There are large gaps between system memory hierarchies, especially between Tier 1 storage and the main memory in the CPU, so this can seriously affect overall system performance and system cost. But perhaps what 3D XPoint has to overcome is the gap between the labs and fabs Intel and Micron have jointly invested in in Leahy, Utah, and the world’s data centers.
That gap has proven difficult to bridge, which is one reason Micron announced the closure of its 3D XPoint business and announced the sale of the Lehi fab. We think Intel will buy the wafer from Micron now that Intel is the only company selling 3D XPoint and Micron owned the Lehi fab a few years ago after the 2nd Gen PCM memory variant was perfected factory. But maybe not, because Micron didn’t say.
And, with Intel’s upcoming “Ice Lake” Xeon SP processors with eight memory channels, instead of the six memory channels on the Xeon processors that have been in use for many years, Intel already has some memory capacity and memory bandwidth The new layout – which will help drive the adoption of Optane DIMMs on the last two generations of Xeon SP servers – will be reduced. The timing may not be a coincidence. But Micron’s sudden and complete response was a little surprising.
In a conference call with Wall Street, Micron President and CEO Sanjay Mehrotra said that Micron will stop 3D XPoint development immediately and will work on leveraging the Compute Express Link (CXL) protocol to link the CPU to the physical external (but logical connection) storage and accelerator devices.
“We believe this transition will better serve our customers’ needs and, importantly, it will improve returns to our shareholders,” explained Mehrotra. “Given the expected impact of CXL and our emerging memory products on future data centers, Our assessment of the 3D XPoint market opportunity informed our decision.”
It’s not a state secret that 3D XPoint didn’t take off as expected, and we’ve said many times that Intel’s Optane came to market much later than expected, slower than expected, with less capacity and more expensive than expected. This is not a good thing. In other words, Optane SSDs are much faster than flash SSDs, and Optane PMEM sticks are persistent, whereas DDR DIMMs require power to remember things.
One of the limiting factors for Micron, which owns its Lehi fab, which makes 3D XPoint memory, is that underutilization of the fab is affecting its non-GAAP profit, which is $400 million a year. To give you some perspective, Micron’s fiscal 2020, which ended in September of that year, had sales of $21.44 billion, operating income of $1.16 billion, and net income of $988 million. As a result, this shortage of 3D XPoint sales is a major hit to Micron’s mid- and bottom-line profits.
But we think the situation is more complicated. Intel had hoped to use 3D XPoint with its “Skylake” processors and “Purley” server platforms, we told everyone shortly after their May 2015 release. We didn’t know exactly what Intel was planning at the time, but the documents we got did say that the Purley system has 4 times the capacity of DRAM memory at a lower cost and persistent data is 500 times faster on the device than DRAM. times. The era of NAND flash. That’s the “Apache Pass” Optane DIMMs, which we call the Optane 100 series, and have been delayed until April 2019 for “Cascade Lake” Xeon SPs.
It’s been nearly four years since 3D XPoint first came out and shipped in the all-important DDR DIMM form, and even then, only Intel servers, Cascade Lake, and then the follow-up “Cooper Lake” four- and eight-way could use them.
System architects and their CIO clients (or, as the case may be, the boss) are wary of any single-source technology that cannot be deployed across a wide variety of CPUs and motherboards. Therefore, we believe the lack of enthusiasm for Optane DIMMs is Intel’s own fault. Intel needs to ease the release of Cascade Lake and Cooper Lake processors, which will reduce memory capacity and bandwidth compared to AMD Epyc 7002, IBM Power9, Ampere Altra and Altra Max, and Marvell ThunderX2 alternatives in the mainstream market. Lake’s Xeon SP is years late due to delays in Intel’s 10nm chip manufacturing process. (So that’s another unintended effect of 10nm latency; a cascading failure, if you will.)
Imagine if Intel (and Micron) took the same approach to 3D XPoint memory in 2015 as Intel did to the CXL protocol in 2019. Well everyone with competing protocols – IBM with CAPI and OpenCAPI, Xilinx with AMD and Arm with CCIX, Nvidia with NVLink, and Hewlett Packard Enterprise and Dell with Gen-Z – are behind CXL because Intel got them involved Will become a market for volume markets and provide some input.
3D XPoint, a variant of PCM memory that scales main memory well with little performance impact, should not be prevalent throughout the server industry, increasing sales and lowering prices so much that Intel and Micron Technology will no longer compete with it. In the last step, only Intel is the only one that is good at it, and it could have teamed up with Micron (or even Samsung) to make it for the world.
That’s theory. In practice, it’s hard to add another layer to the memory hierarchy, and Intel hasn’t done a great job of making 3D XPoint transparent to applications and the OS – nowhere near as good as MCDRAM and DDR4 in “Knights Landing” Do more near the main memory block.
We just have to look at MemVerge, which we covered when it was released in April 2019. When Cascade Lake servers started shipping, we saw a “memory manager” created through MemVerge that really had the potential to make a mix of DRAM and Optane memory easy to use. Optane SSDs have never been used much, but maybe it would be more interesting for system designers if the capacity was increased by a factor of 10 or 100.
As if that wasn’t enough, Micron, after talking to users and system builders, realized that readers would fully understand that, in many cases, memory bandwidth limits applications more than memory capacity. Samsung HBM won the gaming and computing card business with Nvidia and AMD when Intel and Micron jointly developed 3D stacked memory via through-silicon vias (TSV) and used it in Knights Landing CPUs. Now, HBM2 and HBM2E and the planned HBM3 stack memory.
Instead of trying to sell 3D XPoint in a DIMM, Micron will try to figure out how it can use CXL to link external memory and flash to servers on the PCI-Express bus, and sell devices that take advantage of that capability. In November 2020, Micron said it was still on the way) and was trying to decouple from CXL in a 3D XPoint SSD form factor, which was sold as an X100 drive.
This memory bandwidth issue is real, and we’re not sure how memory devices are used on the CXL bus (whether or not they use main memory semantics, or are bit-addressable rather than byte-addressable). The PCI-Express bus for most systems also took a hit, and we’ll say again, IBM’s idea of putting all I/O on a chip (whether it’s memory access, NUMA interconnect, or peripheral interconnect) uses the same high-speed signals SerDes.
The world that both Micron and IBM see is getting more interesting.
IBM’s CPU designers can do this and are implementing it in the Power10 using DDR5 buffer memory, however we don’t see future processors taking this approach.
Sumit Sadana, executive vice president and chief commercial officer of Micron’s four divisions, summed up the situation: “The value proposition of 3D XPoint is to operate as persistent memory at a lower cost than DRAM, or as significant storage space, Faster than NAND. Data center workloads and customer demands have continued to evolve over the years since 3D XPoint was first released. With the proliferation of data-intensive workloads and the increase in AI in data-centric applications, CPU- DRAM bandwidth has become a limiting factor for overall system performance. Also, with the development of CPU architecture, the number of CPU cores has increased significantly, for which we need more DRAM to ensure sufficient memory bandwidth per CPU core. This These trends are driving the continued growth of server DRAM content.”
To be fair, the way Intel and MemVerge use Optane is to increase memory capacity, not bandwidth, since memory controllers and memory slots are constant across the system. If you need more memory bandwidth, you’ll have to use another type of memory, such as NEC’s Aurora vector accelerator or Fujitsu’s A64FX sensible vector HPC-style CPU. In these cases, they use the same HBM2 memory as high-end graphics cards and GPU compute accelerators. If it wants to increase memory bandwidth in the CPU complex, CPU makers have to do it, and Micron has to do everything unless it wants to get rid of MCDRAM or become a player in HBM2 memory. Aside from lowering watts and raising memory clocks, Micron has very little control over memory bandwidth. And CXL won’t change that, although it will create a second class of slower memory that resides outside the CPU’s complex fabric on the PCI Express bus rather than the memory bus.
Another problem with 3D XPoint is that NAND flash is getting cheaper and the rate of change in Optane SSD costs has not kept pace. Persistent storage, as far as Sadana is concerned, “is always a strategic long-term market opportunity for 3D XPoint,” and “over time, 3D XPoint-based SSD products are expected to cease to be a niche market.”
Micron fully acknowledges the difficulty of changing the programming model in the system to add new memory layers to the hierarchy, and says it will run on memory products that use CXL with less barriers to adoption. While 3D XPoint may be the right answer in the long run, the use of these CXL memory devices, and what we believe are their limitations and advantages, will hinder the adoption of 3D XPoint DIMMs, which Micron is not even doing. So far, maybe we haven’t been able to make a server or two.
Micron said it will stop 3D XPoint development immediately and will stop making 3D XPoint chips after contract commitments are fulfilled in the coming quarters, adding that it is in talks to sell the Lehi fab, which the company hopes to do by the end of 2021. .
For Intel, it’s time to start making choices.
The Links: G104X1-L03 6MBP150RTC060
