Performance Tuning and ASP.NET Whidbey

We are now in the final mini-milestone of Whidbey feature development -- which is the time when we start to really push hard on performance, even if it means cutting features to hit the necessary performance goals (which is one of the hardest things to go through when shipping a product -- and thankfully one that we haven't had to-do yet with Whidbey).

We've had a dedicated a performance developer and tester working on perf throughout the Whidbey product cycle, which has helped tremendously in getting to where we currently are at.  Dedicated performance resources is a best practice I've always been a big proponent of, and the only sure fire way I've found to ensure that performance always stays top-of-mind (otherwise new feature work always seems to creep up -- with performance too often getting pushed down the priority stack).  When we formed the ASP.NET team from scratch back in 1997/1998 we actually dedicated the second developer we hired on the team towards working full-time on performance (no feature work whatsoever) -- and it payed huge dividends in terms of the final product we shipped.  Hopefully we'll see a repeat dividend pay off for Whidbey.

We have about 100 micro benchmark scenarios that we run on a weekly basis for ASP.NET, each on 1P, 2P and 4P boxes.  This gives us a good ability to track in real-time where the product currently stands, as well as to better pin-point checkins that cause regressions (we can basically isolate the checkin to a week period and investigate further based on that window).  To better prevent perf regressions we also have a few simple performance checkin suites that all developers need to run and pass before any checkin is allowed (we do a clever trick where we time and run-through some scenarios in both classic ASP and ASP.NET, and fail the test if the multiplier difference between the two falls below a certain level -- which gives us a good way to measure regardless of memory/processor speeds on the system).

Of the 100 or so benchmark suites that we measure, there are 4 that I really care about right now.  Basically they are:

1) Hello World Page.  This is a simple page that litterally runs a tiny amount of code that just spits out “Hello World” and has no server controls on it.  Not terribly real-world -- but it does do a good job measuring our core code-path execution length and timings.  It also is very sensitive to our http header and request/response buffer management (code that executes on every request regardless of scenarios).  If it sucks perf-wise, nothing on top will look good.

2) MSN Home Page.  This isn't technically the MSN Home Page today - but rather one from about 6 years ago (when we first started getting serious about performance).  It does a pretty complex rendering of a page that has about 45k of HTML, and executes a lot of rendering code logic.  It doesn't hit a database (instead all the values are read from disk once and then cached in-memory), which gives us a good benchmark that enables us to evaluate just core ASP.NET and CLR execution performance.  It also doesn't use any ASP.NET Server Controls (it is a straight port of an old .asp version of the site -- and so uses <%= %>rendering style logic).  But it does give us a good test that enables us to evaluate how ASP.NET handles rendering “real-world“ size pages (as opposed to small byte hello world apps), and specifically measure how our async i/o system performs under high throughput and output loads (our buffering code here has been carefully tuned with literally thousands of hours of hard work).  It also does a good job evaluating how the CLR is doing in terms of both JIT code performance, and specifically with GC (garbage collection) memory allocation and collection.  If it regresses, no real-world app on the server will do well (regardless of whether there are server controls or not).

3) WebForm Common Page.  This is a page with about 150 server controls declared on it, and which measures both rendering and post-back performance costs.  It doesn't do any user logic (no database access, page_load, or event handler code), which enables it to serve as a great benchmark to meaure and highlight the core ASP.NET Control framework performance costs.  We measure both throughput execution on the page (requests/sec) as well as ViewState and overall Rendering sizes in terms of bytes (example: we watch to make sure we don't blow out Viewstate serialization size that would increases overall response payload size, and increase browser latency).  We have individual performance tests that measure each individual ASP.NET control (to measure and watch for regressions on each control), but this core WebForm Common Page test does a good job making sure we keep an eye on fixed page framework costs.  If it regresses, every page with a server control regresses.

4) Data Page Scenario with a DataGrid and ADO.NET SQLReader Connection.  This measure a simple database query (grabbing 100 or so rows from a database using ADO.NET) and then binding it to a grid to display the values.  Again not a super-complex page -- but one that does a good job measuring our data controls framework, as well as the performance of ADO.NET in server scenarios.  It is heavily dependent on the control framework in terms of performance (if test #3 sucks above -- this test will be bad too since it ends up generating hundreds of controls at runtime), as well as raw ADO.NET (which the ADO.NET team perf tests separately). 

Where we are at with Whidbey

Right now we are in decent shape with the above tests.  Adam Smith did an absolutely awesome job tuning our core execution request code earlier this milestone, which has enabled us to see improvements in ASP.NET Whidbey performance of approximately 20-25% over ASP.NET Everett for test scenarios #1 and #2 above (the Hello World and MSN application scenarios). 

Scenario #3 above still needs more work, although we've made a lot of progress since the Alpha (where it was way, way, way off from our previous Everett release).  The challange is that we have lots of new features and code in Whidbey with the page framework, which necessitates multiple tuning passes in order to bring performance at par or better levels.  Our general model is to spend a few days profiling the heck of out of a scenario, open 20-25 performance bugs to fix each of the identified issues, assign them out, and then drive folks to check-in fixes.  We then repeat the profiling process again, find and fix the next set of optimizations, and then repeat again.  Over time the “performance onion“ slowly gets peeled back more and more, and additional optimization opportunities reveal themselves.

Right now we are still about 15% off from where we want to be when we ship on test #3, but have identified some significant optimizations that we think will get us close.  The biggest optimization remaining will be to take a hard look at our memory allocation within the page framework.  Our new Whidbey features have added several extra fields to the core System.Web.UI.Control base class, which currently end up generating an additional 36 bytes per control per page request.  This doesn't sound like much at first glance, but adds up quickly when you realize that every control in the system has this extra 36 byte memory allocation added onto it (regardless of whether they need it or not -- for example: the literal control for static content), and that you can have hundreds of controls instantiated on each page.  When doing a few hundred requests a second, you can end up allocating several megabytes of additional memory per second on the system.  This ends up increasing GC pressure on the box, which can significantly impact server throughput.

The fix we are working on now will be to defer memory allocation as much as possible (only doing it when necessary), and to be more creative in how we store fields (example: don't put them on the core control class -- instead moving them one level deeper within internal classes that the control class then generates on an as-needed basis).  We think that once these changes are made, test #3 will improve dramatically -- and hopefully get us either close or above the bar we've set to ship.

Scenario #4 is gated right now to some extent on the core control framework test covered by scenario #3.  My hope is that once we get #3 to hit our goal, we'll see #4 come fairly close to hitting its goal too (assuming ADO.NET hits its performance goals).  This should set us up well for the next round of optimizations, as we broaden our focus to look at more tests -- and start benchmarking existing real customer applications on V1.1 (Everett) against V2.0 (Whidbey). 

Assuming we get the above 4 tests in good shape for the beta (either at goal or within ~5% of them), I'll feel fairly comfortable about where we ultimately end-up on performance with Whidbey.  There will still be a lot of work and investigations left to-do, but we'll have our core performance work foundations in place, and will be able to finish tuning the product on a steady glide-path to final RTM.