Frequency and predictability of builds for Windows Insiders
March 9, 2015Gabe Aul said:.....
Why it is a challenge to just tell a date for the next build
It is counter-intuitive, but doing this actually makes builds get published more slowly and have less fresh content than by leaving the date open ended. Why?
- If we announce a date, we’ll want to have a very high confidence of hitting it. It’s frustrating for you to hear a date and be let down if we miss it, and it’s frustrating and distracting for us too. Not only that, but it slows down our engineering since many of the same people who are scrambling after a missed date would otherwise have been making more forward progress on the product.
- Because we’d want that very high confidence we’d pick a date that was further out than if we were living on the edge. We’d give ourselves some time to deal with bugs and re-spin builds if we needed to.
- If we have a great build in hand, as often happens, leading up to the date we would hold on that build rather than ship it. We call this putting the build in ‘escrow’. Why not just ship it early? Well, some people get upset about the surprise, but also it sets expectations that sometimes we really mean a date and sometimes we don’t. We want people to know that when we say a date they can count on that date.
- In the worst case, if we’re chasing down a tough bug and run out of time, we may miss the date. This is of course way worse than being early. We’d have let down people who were counting on us to deliver on the date we said we would.
Let’s play that out hypothetically for the next build coming out. Today is 3/9 and we want to ensure we get a build out in March. If we communicated a target date, to be sure we could meet our commitment we’d likely pick a date like 3/26. It gives us time to stabilize and it’s on a Thursday (we usually like to avoid Mondays and Fridays.) Between now and then we’d still be getting new feature payloads, but we’d fork to a stabilization branch somewhere around 3/17 or so and only take selective changes. It’s easier to stabilize without a lot of additional new code, so we’d cherry pick key fixes. On 3/23 we’d have a candidate build, and we’d flight that out broadly within MS to make sure we could find any gotchas and meet our date with confidence. Hey, that doesn’t sound too bad does it? Except in the ‘worst case scenario’ where we miss the date and people are let down, it means a predictable date about once per month with kind of up to date code.
But now let’s talk about how we’re really trying to approach it. Today is 3/9 and we’ve not set a date for the next build. I have a build in hand that we produced on Friday. It was validated by our test automation, and will go out through our internal rings and get installed and used by thousands of people at Microsoft. It is the freshest code with all newest features and fixes. If it passes all of our evaluation criteria it could be in your hands late this week or early next week. That means that we could feasibly get multiple builds out in March rather than just one, and they’d have more up to date code than if we did it the other way. Yes, I know, that is pretty big talk considering it has been more than 40 days since our last build; and here I am talking about multiple builds per month. I’m sharing our aspirations and what we’re building towards, and we want to be working in that new way vs. the way we used to do it. Not having the constraint of a fixed public date for each build helps us get there faster. Here’s a real example though: We had a debate internally about whether we should announce a date for 9926 – the build we shipped the day after our 1/21 Windows 10 event. The choices we had:
- Set 1/23 as the announced date, knowing it would have been a build produced weeks before without many of the features we demoed.
- Set 2/15 as the announced date, giving ourselves the flexibility to have a good build in escrow.
- Don’t announce a date, use the ring promotion process, and go when ready.
We went with option 3 and it paid off. We got a much fresher build out, with more features and fixes, and we were able to ship on 1/23 as we’d aspired.