TechEmpower Benchmarks and the Microsoft ASP.NET Core 1.0 Performance Story

I’ve had lots of conversations with fellow CTOs about the TechEmpower Web Framework Benchmarks.  Some really appreciate the value that they bring to help them understand performance characteristics of different frameworks.  Depending on the Technical Performance Requirements for your system, this could be really valuable information that is part of your framework selection process.  However, I’ve also had fellow CTOs tell me that they don’t find the test credible or that they don’t understand how their favorite framework doesn’t perform better.   Frankly, those two statements are often correlated. But when Microsoft is talking about “huddling around the benchmark” and “only making a pull request when it’s an order of magnitude of Node.js” – I would say that the benchmarks are providing real value to the development community.

Let me step back and tell a bit more of the story here.

You may or may not be aware that Microsoft just announced the release of ASP.NET Core 1.0: 

Today we are excited to announce the release of ASP.NET Core 1.0! This new release is one of the most significant architectural updates we’ve done to ASP.NET.  As part of this release we are making ASP.NET leaner, more modular, cross-platform, and cloud optimized.  ASP.NET Core is now available, and you can start using it today by downloading it here.

There’s a lot to like about ASP.NET Core 1.0.  It is a viable contender for all sorts of development efforts.  One of the things that makes us even more excited is that Microsoft has focused on is Performance as a core attribute:

With a significant rewrite of the web framework, we addressed some performance issues and have set aggressive goals for the future.  We’re introducing the new Kestrel web server that runs within your IIS host or behind another host process.  Kestrel has been designed from the start to be the fastest .NET server available, and our engineers have recorded some benchmarks to prove it.  With the backdrop of the standard TechEmpower Benchmarks, the team used these same tests to validate the speed of Kestrel and have some impressive numbers to report.

Another announcement also touts the benchmarks:

We used industry benchmarks for web platforms on Linux as part of the release, including the TechEmpower Benchmarks. We’ve been sharing our findings as demonstrated in our own labs, starting several months ago.

How did Microsoft get to this kind of performance.  Scott Hunter, Director of Program Management on the App Plat team at Microsoft, tells a bit of the story on the DotNet Rocks Podcast (starting around 19:00). 

I got a rash of customers who said to me, “Hey, we went to this TechEmpower Benchmark site.  And we saw where .Net was and where other technologies are, why should we be using your stack?”
…
Damian said – “I’m going to build perf lab and take a look at this thing.”
…
In the team room, the team would be huddled around the benchmark saying, “We got another 10,000 or 50,000 or 70,000.”
…
would only make a pull request when it was an order of magnitude of Node.js.  If I can get 2 Nodes, then I’ll do a PR. 
…
It became this thing in the team room where people kept piling in, and it became important.  Then as we started putting the numbers out there, the response was crazy.  The pinnacle of the responses was … Satya [Microsoft’s CEO] got an e-mail from somebody in the Valley, which we ended up seeing at some point. The person was basically saying, “Hey, I just want to let you know that non-Microsoft and non-DotNet people down here are actually looking at the numbers that one of your teams is doing and we find them super-exciting. He said there’s chatter on Slack channels and stuff from people who not be even thinking or talking about us.”

The Damian mentions is Damian Edwards.  You can see a talk he does on Vimeo also telling a bit of this story.


I have to mention that early in the video Damian asks:

Who’s heard of TechEmpower – okay most people. 

Wow, really - who's that audience?

Damian takes us through how they looked at the Benchmarks and what led them to achieving some remarkable results posted on their intro page:

This is exactly the kind of thing that we were hoping at TechEmpower when we came up with the benchmarks.  The fact that Microsoft made it a focus and applied resource to produce such exceptional performance is commendable, and the result is a solution that is provides tremendous value to ourselves and the developer community more broadly. 

Great job Microsoft! 

At TechEmpower, we are very happy to have been part of your journey.