User Experience Designers...Listen up!

A couple days ago I posted a question here (and on some other discussion lists) regarding how usability can fit into the Agile process. I revised it slightly and added some clarification:

What are the best methods you’ve experienced with regard to incorporating quality design & usability practices into Agile software development?

There is some debate in the software industry regarding the scope of UI design work to be completed before Agile software development can begin. What are your thoughts? When is a detailed prototype necessary? When is it not necessary?

I promised to post my thoughts so here they are.

The method that I am currently using involves some upfront design time which I consider the architecture phase. During this phase is when you gather and process all the data about various types of end users and their goals. Throughout the architecture phase, we work on low-fi wireframes and/or mockups that address high level global interaction patterns such as type of navigation, inline editing vs. popups vs. separate edit screens, etc. I liken this to the engineers' decisions to use a particular technology, library or persistence strategy. Some of this stuff needs to be thought out early since changes later are expensive and destabilizing. Because, end user satisfaction is a top priority for our customers, I propose doing usability testing during this phase (and all through development). 

While wire-frames and paper prototypes can certainly be tested, we have found that nothing works as well as a clickable prototype. Someone suggested above that just build the real thing and test that. We have repeatedly seen that a very detailed prototype can be built in 4 to 6 weeks where the actual product can take 6 -12 months. Of course with agile development you have something every 2 - 3 weeks but it does take a while to get to something that can be a true test of the user experience. The level of detail in the prototype and wireframes/mockups is really determined by the complexity of the interaction. If one pattern can be reused, there is no reason to prototype every use case of it.

This process can work very well but requires some commitment of time before development begins. This doesn't have to be a year long effort but depending on the type of application somewhere between weeks and months seems appropriate. Regardless of the amount of time required, this is often labeled "Anti-Agile or Waterfall". 

What lead me to asking my original question was my desire to know how others are doing it from a tactical perspective. Can the process I describe above scale? Is it truly anti-Agile? If it is, does that really matter if the result meets user's needs better? 

If the goal is to deliver really good software, not just really good code, I am ok with being slightly less Agile.