Where does QA fit in Agile?
Want to make an Agilist squirm? Ask them where your QA process fits into this Agile/Scrum miracle they are implementing. “Well you’ll unit test and you’ll continuously integrate and quality will improve!”, they’ll say. “That’s great, but what about my QA team? Where do they work?”
One of the dirty secrets of Agile is that is a very development-focused software methodology. That’s not to say it shouldn’t be, I’m just pointing out that most people evangelizing the benefits of Agile practices typically comes from a development background. Their knowledge and experience focus primarily on what does and does not work for developers. So when we talk about the roles and responsibilities of quality assurance personnel, suddenly the answers aren’t quite as easy to come by. That makes sense, but real teams have real QA needs and they need to figure out where in the world all of this work actually occurs.
When I think of QA, I think of three major activities:
- New feature testing
- Feature integration testing
- Regression testing
I think you can even further simplify things by saying:
- Testing new features and how they integrate
- Testing everything else that we shouldn’t have broken but might have
“But what about validating bugs?” You can just classify those as new features and it’s all good.
So using a Scrum-like schedule with a requirements backlog and fixed-length iterations (I’m a big, big fan of both), you end up with something like this:
Scrum and scrum-like methodologies, as they should, rigorously control what and when new work enters into the development team’s world. It’s probably the single-most beneficial attribute of the Scrum process. It’s certainly my favorite. But where do testers fit in? Here’s the “by the book” answer:
While there’s an incredible amount of rigor over what enters the development phase, there isn’t so much rigor around what goes to test. Every team I’ve ever spoken to that says they test and develop in the same iteration has a magic point in the iteration where the handoff occurs. Before that point, the focus is on new development. After that point, the focus is on testing. For example, if the iteration is a week long, the plan is for development to develop new stuff Monday through Wednesday and deliver that to testers on Thursday. Then testers are supposed to test like mad until the iteration is over. This is probably a more accurate picture of what actually occurs in this scenario:
Testers are on easy street until they get a dump of stuff from the development team and then it’s testing on the pain train.
Why are developers afforded control over when they start new work but testers have to be at the mercy of development’s schedule? I don’t think it should be that way. The testing effort should be planned, estimated, and controlled exactly like the development effort. So I propose you fully stagger your iterations. Not a partial stagger, but have your testers be a full iteration behind. Testers only test what development declares “done”, just like developers only develop the requirements that are complete. It looks something like this:
Typically this proposal generates three major critiques from the keen Agilist:
- But that looks like waterfall!
Truthfully, there’s no escaping waterfall. You can’t be “done” developing something that isn’t already “done” being analyzed. You can’t be “done” testing something that isn’t already “done” in development. The magic of iterative development, which is a centerpiece of Agile methodology, is that it breaks big problems down into smaller, more manageable chunks. Things still flow downhill like they’ve always done, and always will. - But that means, if your sprints/iterations are a month long, it could literally be two months before you test something that development coded!
If that’s too long, then your iterations are too long. There are other consequences to shorter iterations, but ultimately the way you shorten the time from inception to delivery is by carving up your project into smaller and smaller pieces. - Every new bug is new work that you haven’t planned for! Where do you carve out time in an iteration to do bug fixes?
You should always subtract time from your iteration capacity to do bug fixes. In fact, you’re probably doing this already if you use iterations. You simply track the time you spent last iteration fixing bugs and if that’s not enough, take more capacity for bug fixes. If it’s too much, reduce the capacity required for bug fixes.
There are a number of other scenarios that should be accounted for, but there’s not enough bandwidth on the web to deal with all of them. That said, there’s on scenario that every team eventually faces: how to deal with the cumulative nature of the testing effort. As you do more and more development, new complexities emerge because the system you’re developing has a growing number of features, capabilities, and interdependencies. You have more and more stuff to test even though the pace of development hasn’t increased. The more stuff you have to test, the more bugs you have to fix. Over time, the testing effort and bug fixing effort will outgrow the time you have in an iteration. How do you deal with it?
It’s kryptonite to many Agilists, but my recommendation is that you plan to do “stabilization Sprints”. These are iterations completely focused on “catching up” with the testing work and bugs that have accrued over time, but have continuously been prioritized lower than new feature work. If you don’t think I have a clue what I’m talking about, know that the Visual Studio team uses stabilization Sprints/iterations. At the ALM Summit, I specifically asked Stephanie Saad how they deal with bug and testing accrual, and she told me point blank that they stabilize from time to time. It may not be how things “should” be done, but I firmly believe it is the best way to handle the software development realities confronting teams, even ones as smart and sophisticated as the teams that make Visual Studio.
I know I’m just scratching the surface here, but that’s enough for now. I’d love to hear your opinions, compliments, and scathing criticism of my take on the topic.
Update: I’ve published a follow-up to this post that you can read here. And don’t forget to follow me on Twitter for updates and other info related to ALM and TFS.
79 Comments
Post a comment
Trackbacks & Pingbacks
- December 30, 2010 – TFS Links and Quick Hits | Learn TFS
- QuickLinks for December | (Agile) Testing
- Quality assurance e metodi agili | Sviluppo Agile
- Where does QA fit in Agile? – Part 2 | Learn TFS
- camestres.com » Links for 5/1/2011 - paolo manca's blog
- May 5, 2011 – Visual Studio and TFS Daily | Learn TFS
- July 7, 2011 – Learn TFS Daily | Learn TFS
- TFS Process Templates and Top Level Structure « Technology Services
Sorry to say but this is common suboptimal “implementation of scrum”, some refer it “scrumfall” and some think that it is not scrum at all. You simply get feedback “an iteration too late” and your software is never done when “exiting the development phase”. Smell of this implementation is usually definition of done saying “ready to be tested”.
Anyway, here is how to make it better:
* Form a cross-funtional team with developers AND testers in it – their definition of done is “releasable product/project increment” that is tested
* There might not be sense to have analysis as “iteration”, analysis is continuous flow
How does the tester then participate in the development sprint to get the items you highlighted done:
* New feature testing
- In sprint planning the testers will help the team to figure out the acceptance criteria for the incoming feature. They might even express that as executable acceptance test (see .. or they might do that within the sprint.
- During the sprint testers are doing exploratory testing to find the corner cases of the feature and “how it fits to the whole”.
- In the demonstration testers help the product owner to see that the acceptance criterias pass and they are there also to make sure that if something is not done then it won’t be demoed
* Feature integration testing
- Integration in the large: testers may work on emulating the external systems or testing these cases (by automating the regression suite)
* Regression testing
- This is the “product” that testers will do step by step by automating the acceptance tests
It might sound that testers need to be developers. The testing is collaborative activity taken by the “team”. If testers need something to be developed then the developers will develop that. Pairing helps a lot in here. Sometimes testers even help the developers to do unit testing (tester has different point of view which is usually very beneficial).
Many teams (including ours) have successfully managed to build security and performance testing suite that can be executed part of the continuous integration/deployment and have also continuous monitoring in place (to improve these suites). Still there is testing (usability comes into my mind) that might be very hard to fit into sprints .. even by taking 1 small piece in the time.
Thanks for your well thought out comments.
A couple of points:
- I’m familar with Scrum-fall and think its a perfectly acceptable way to develop software. One size does not fit all when it comes to development practices.
- As a consultant, I don’t have any power to control the teams I inherit. My job is to get the most out of the teams I have, not to hire the teams I would like. On the tester side, this is especially true, as most organizations simply are not willing to devote considerable resources to their QA effort. It’s a “nice to have” in their eyes, and I’m just trying to get them able to make the most out of the investment they HAVE made.
- Some organizations cannot get by with “testers need to be developers”. Many test teams have to be experts first, and technologists second. For example, I worked with one team entirely made up of PhDs in geophysics and geosciences. The applications they developed required a very specific specialty in order to properly assess “correctness”.
If your teams has successfully integrated development and test into the same iteration, you certainly should not stop doing it. Do what works for you and your team – I just see a lot of teams that are adopting Agile really struggle with the lack of rigor around their testing effort.
I am not suggesting “one size to fit all” (actually I thought your post was telling that one cannot do testing within sprint).
) Just to demonstrate this we have dropped scrum: http://huitale.blogspot.com/2009/10/huitale-way-is-it-scrum-or-is-it-kanban.html
I hear your pain and I am currently helping an organization who is in similar situation (this is how they’ve implemented it). Somehow I feel that you are paying attention for short term gains by not solving the root causes (or choosing your battles very carefully..).
About PhDs and correctness. These PhDs should be excellent for getting input to an ATDD effort.
You may be right on the short term gains. I’d say that’s definitely my primary bias – I need to enact improvements and I need to do so quickly.
Thanks for the link, I’ll check it out.
While your sentiments are in the right place I think you’ve missed some key concepts in the analysis if scrum practices. I reckon you might do yourself a favour with a face to face session with a scrub expert (rather than with self taught practitioners).
Fail. I think about tweeting this as an development process anti pattern.
That’s certainly your prerogative. Thanks for reading and hope that eventually I get to something you find value in.
I thought I should post a comment just to say how completely and utterly you have misunderstood Agile and suggesting that you read and ponder over the Agile Manifesto and its 12 Principles.
And then I thought I just wouldn’t bother.
I’d categorize myself as an Agilist and someone who recognizes all 12 principles, but ultimately those principles and the manifesto are nothing but platitudes. The fact is, teams in the 60s, 70s, and 80s who were using classic waterfall (and even teams that are in the midst of utter chaos) believed they were embracing all of the philosophies that are claimed to characterize Agile.
The value of Agile is in the application of its practices.
Sorry that it all seems to have rubbed you the wrong way – I’m just trying to find solutions for the teams I work with.
In short: what is described here as “scrum” or “agile” sw development I do not recognize as such. What I see here is one of the 9/10 cases where people think they are doing / have seen done something agile but in reality have not.
Testing it AT LEAST as important part of the team’s work as the “design” activities. It should be done in parallel and be incorporated in the Definition of Done, thus forcing the testing to be done within the sprint. Unfortunately because of dependencies between corporate level sw development teams there is often no other way than to test stuff in the next sprint. I have seen this happen in 9 out of 10 “agile” projects and in each case it has been an excuse for not to make a real change in the way software is being made.
One successful agile setup: have cross functional, cross discipline teams. PO writes the requirements only from customer point of view: teams worry about the implementation. During sprint all the teams share the same codebase, integrate many times a day per developer to main which a real continuous integration system constantly builds and radiates from. Testing activities happen inside the sprint parallel to other development activities. At the end of the sprint all DONE work is presented from a development kind environment to stakeholders. Have a retrospective, keep what worked, change what didn’t. Rinse and Repeat.
I have seen this being done (yes, these are the 10% cases) and it works. Yes, there are still lots of problems to deal with but thanks to the CI and the DoD demo you know where you really are and you know what the effect of a change you take really is.
Oh and: “Testers only test what development declares “done”, just like developers only develop the requirements that are complete”
In agile software development requirements are never considered to be nor expected to “complete”, quite the opposite. You should instead expect that the requirement WILL change and structure your way of working (even coding style) as such
Good thoughts. I agree that what I described some may not recognize as such. I probably should have clarified that what I’m typically engaging are teams that argue they use Scrum and this is what they came up with.
Ultimately, I see the biggest problem with incorporating testing “throughout” is that it assumes infinite capacity. Test is truly never done, and teams that pepper testing throughout typically struggle to plan, track, and estimate their testing efforts effectively. I think that’s my biggest obstacle to an “Agile” testing approach.
I also don’t see many “cross functional, cross discipline teams”, and I think many teams struggle because they try to apply Agile to the teams they have, not the teams Agile says they should have. So they try to put the square Agile peg in their organization, which sometimes is a round hole, and they end up throwing out the baby with the bathwater. Agile, including iterations, continuous integration, and a product backlog are rejected even though those practices could have provided immense value.
So my approach is to try to take the practices that apply widely (like those I just mentioend) and compromise in areas where practices aren’t quite as essential.
I’d take your ideas more seriously if you’d tell us who you are, what your experiences are, and why we should consider what you have to say about Agile. I can’t find an author’s name or bio anywhere on this site.
FWIW, you’re off base on your assertions about QA and Agile. In Agile done well, QA folks/testers are an integral part of the development team, working alongside developer/builders and customers/product managers all throughout the iteration. Not as a separate group receiving handoffs from dev. They are part of the whole team and participate in product demos and retrospectives to continuously improve the product and the process.
Have you read any of Brian Marick, Lisa Crispin, Janet Gregory, Elisabeth Hendrickson, James Shore or Gojko Adzic’s work? They offer good advice about how to better integrate your QA/test people into the team and work toward a whole team approach.
In the meantime, who are you?
My experiences are working with teams for half a decade trying to introduce “Agile” into their existing world. I take a more pragmatic approach, as “Agile” typically requires a cultural shift that organizations simply cannot adopt, so my approaches tend to meet them in the middle where they can take advantage of the rigor that Agile introduces without falling prey to the consequences of simply using “Agile” or Scrum as an excuse to continue in the chaos. I find this especially true when dealing with QA – as a consultant my job is to bring solutions to the teams that organizations employ, not to bring solutions to the teams I wish they employed.
I don’t think who I am really matters – that tends to be an excuse to wield a appeal to authority fallacy. The arguments stand or they don’t, and certainly, I know full well there are legitimate criticisms to my approach and philosophy.
Good question: many teams do encounter the issues you raise. The answers, not quite so good. See the new article on my web site, “Back to waterfall isn’t QUITE the right answer”, at http://xprogramming.com/?p=2053
Thanks!
Thanks for the link and your response.
I’m all in favor of pretty much all the points you mentioned, I just don’t see them applying well to the test teams I actually encounter. To be frank, your response assumes “strong testing skills” that I don’t believe are as common as we’d both like.
And while my post definitely contains a bit of rhetoric, I’m not anti-Agile. I’m actually very a big fan of agile practices, but definitely a bit frustrated with the platitudinal evangelism that I see many Agilists pushing. I’ve talked to literally hundreds of real teams that struggle to apply what they perceive to be empty statements (e.g. the entire Agile manifesto, as it reads) to the real roles, responsibilities, and behaviors of their teams.
I’m now subscribed to your blog. Good stuff.
Let me backtrack over that. (1) You’re trying to get your testers to work with the developers on the same sprint (2) It’s not working as well for you as you’d hope, and the reason is a lack of strong testing skills, (3) your reaction is to change the process, increasing the length of the feedback loop back to devs.
Isn’t there an alternative to (3) ? I’m suggesting that the short feedback loop is the better thing, and the real problem to address is the skill levels of your testers and/or the way testers and devs interact ?
Good thoughts.
I’m actually coming at it from the vantagepoint of the hired gun who has been commissioned to take an EXISTING team and get better results out of it. So yes, the root cause of my approach is definitely what you point out: the real problem is the skill levels of the testers and the way testers and devs interact.
I’ve worked with several dozen test teams, and I can’t say I’ve seen a single test team that can confidently work alongside development and provide the quality that is required. It’s either the result of a lack of testing skills and experience or a lack of bandwidth. Virtually never is a QA team’s capacity accepted as the limiting determiner of velocity. Everything, planning-wise, centers on around how much dev can get done. If we need more capacity, we add more to dev first.
I’m of the opinion that most test teams, and the quality and capacity of those teams, truly reflects the investment an organization is willing to make in their testing effort. Lack of control and rigor of the testing effort exacerbates the quality and capacity problems, so the test team has to get control over their responsibilities, and my recommendation is the result.
Hope that puts my thoughts into context. I don’t think they are off course given the context, and I reject Mr. Jeffries connotation that my post is an attempt to “debunk” Agile. I’m a very big proponent of iterative development with backlogs, unit testing, static testing, and the like. But I’m also of the opinion that we have to give teams methodologies that scale to the teams they have, not the teams they’d like to have.
Sorry for being long winded and hope that helps my case more than it hurts it.
One other point (and not trying to be mean or snide), I completely reject this from your article:
“The best Agile teams can produce software that is essentially defect-free in a single iteration.”
I just don’t think that reflects the reality of modern software development in any way, shape, or form. The only teams not creating software with defects are teams not doing anything.
“I just don’t think that reflects the reality of modern software development in any way, shape, or form.”
No, it just doesn’t reflect your reality. It does reflect mine when working the way Ron says.
That may be, but I can say, with extreme confidence, that you both are in the vast, vast minority. Consequently, it means that how you develop software doesn’t have as much applicability to providing assistance for the real teams many are faced with improving.
Hi, Ron,
I read your response. Good rebuttal.
But it seems to me that regarding unexpected volume of defects to be nothing more than rework (and possibly an indictment of forecasting accuracy), and then opposing tactical response to the defects, amounts to conflating strategy with logistics.
Forecasting is important, you’re right. It enables agility in response. But prohibiting a tactical response because it indicates forecasting failure undermines the very value of forecasting — which you said is of paramount importance.
What use is forecasting if you are unable to respond to unexpected issues? Sounds kinda waterfall, dunnit?
Stabilization does not have to be shooting the canary in the coal mine. You can still reassess forecasting, requirements, criteria, etc., if there’s evidence of a problem.
But also bear in mind, high volume of defects may be indicative of other things. From the story to the UAT, maybe everyone says “done” but they didn’t test it, and the tester finds some showstoppers. Maybe it’s architectural and it constitutes a separate sprint just to make it right.
I always thought the most important benefit and fundamental principle of Agile was agility.
Nice straw man argument. Good luck with that. Small surprise that it’s posted on a site that pushes tfs.
I’m not sure you kept your moral authority by arguing against what you perceive to be a straw man by responding with an ad hominem.
Seriously though, do what works for you and your teams. Unfortunately, it doesn’t necessarily scale to everyone.
I don’t think you can rationally have a conversation online or off about the problems of agile. There is too much being made from the people associated with them from consulting, classes, and books.
See, making agile perfect means the only people not getting it is the team. How do they get it then? They hire the manifesto team.
Sad but very true.
It can be frustrating, but I’m committed to making sure MY side is as rational as I can be. So that means we’re halfway there.
Really? Have u heard about ATDD and/or BDD? In both practices the testers (not QA by the way) are ahead of development. Dev/Testers/Business get together before development to establish acceptance criteria and/or tests.
Mind u, ATDD & BDD are far from being as widely used as TDD. And TDD itself is far from being mainstream or properly applied. So really, I am not even sure of what point you are trying to make. If you ask me the magic question I’ll tell you QA is built into the process, not anybody’s job description or separate step.
The person in your QA team would be part of a team with devs and testers. Devs work very closely with testers during the whole process from the moment the story is pulled in until is done done. There are several books explaining this and user groups with this info.
Cheers
Thanks for your comment.
I understand your points but, as I tried (and maybe failed) to point out in the post, when the testers “work very closely” with developers their effort is very inconsistent and their responsibilities (and expected deliverables) are very poorly understood. I think too many pushing Agile do software testing, the specific activities of a classic quality control personnel, a disservice. All of the activities that go into improving quality – static testing, unit testing, etc. – are not the bulk of a software testing effort. The bulk of the effort, for the vast, vast majority of organizations, occurs after features are declared “done” by a development team.
I’m not completely averse to testers being involved throughout the process, it’s just that few understand what that means and those that do are even poorer in their execution.
As I’ve tried to articulate in some of the comments, my opinion is primarily based on the activities of mainstream test teams – the real teams in the real shops I work with. Most of those teams struggle at many of the basics when it comes to test discipline and process: test management, test metrics, etc. It’s a rejection of “qa is build into the process” because that is ultimately a platitude.
“The bulk of the effort, for the vast, vast majority of organizations, occurs after features are declared “done” by a development team.”
The concept of “Done” in Agile isn’t owned by the developers alone. It should be a combination of the developers, QA and with Product Management having the final say on whether a feature or backlog item is actually completed from a business perspective.
What I see from your diagrams is a problem I see repeatedly… teams trying to do too much, or as a corollary, they have an overly optimistic view of their actual capacity.
Try working on a single work item such that it is done according the criteria I mentioned above. Then move on to the next item. The number of items you can do in parallel will be roughly equivalent to the number of QA people you have. The idea is that each item will be production-ready when it is declared done, not just ready for the developers to throw over the wall to QA.
In the end, that process will be much more indicative of what your team can actually deliver over a given timeframe, without the “crunch period” and compression of the testing stage that I commonly see.
I think you nailed it: “teams trying to do too much” and teams that “have an overly optimistic view of their actual capacity”.
Further, most teams are built such that their development capacity far outpaces their test and analysis capacity, which is actually another problem that my approach has the potential to address. But that’s for another time.
“Further, most teams are built such that their development capacity far outpaces their test and analysis capacity, which is actually another problem that my approach has the potential to address.”
I absolutely agree. I routinely see teams with development to QA ratios from 4-1 to 8-1. On an encouraging note, one group with whom I’m currently working has a ratio of about 2-1, and they’re trying to improve that.
Interestingly, an FPGA hardware group I recently coached had a 1-1 ratio. Their philosophy is that anything more than 0 defects is unacceptable. These people, of course, came from a hardware world where a single defect could cost 10′s of millions of dollars, and they have brought that attitude to what’s effectively software development on the FPGA. I believe what the software world needs is a whole lot more of a hardware-like attitude.
Now as for the applicability of this to a discussion about QA in Agile, it does have an effect on the team’s ability to complete work. When I tell a team not to code new stuff after QA gets something to test, I get the “that’s wasting time!” complaint. The goal, though, isn’t to be “100% utilized”, but rather to maximize the throughput of completed work. If developers are waiting for QA, there are plenty of things they can do: increase microtest coverage, help automate regression tests, help automate the build & deploy process, etc. While this may appear to make the team as a whole (which includes QA) slow down, they actually move faster over time owing to minimized defects and therefore minimized rework.
This isn’t conjecture or theory. I’ve been doing this under the XP or Agile name for 10 years. I’ve done it with both greenfield and legacy systems. I’ve done it as a temporary coach, and as a player/coach for over multiple years with a team.
>>I’d categorize myself as an Agilist and someone who recognizes all >>12 principles, but ultimately those principles and the manifesto are >>nothing but platitudes.
I’m confused. You state in the same sentence that you recognize the principle and that they are platitudes.
For me that are two contradicting statements.
Yes agile is not easy to implement. Yet the majority of the testing community is in favor of agile because it makes it possible to do proactive testing instead of re-active testing.
It looks like you have been trying to combine agile and testing and had a bad experience. Sorry to hear. Yes I had some of these myself. And still I know that it is the best possible combination there is. The hard part about agile is that it does not hide problems.
If your organisation can’t handle problems, you are correct. Only that is not about agile, that is your organisation having trouble with transparency.
I didn’t articulate that well. I recognize (and have implemented) Agile methodologies, but I believe the principles of the manifesto are, ultimately, meaningless. No team is equipped with a solution upon reading, learning, and memorizing the manifesto. I believe that teams that developed software with a strict waterfall long before Agile was anything but a way to describe athletic prowess thought they adhered to each and every one of the principles. The Agile manifesto is not descriptive, and that’s what teams desperately want: a clear example illustrating how work is done in their organization.
I might go so far as to agree that combining agile and testing is the best possible combination, but I think my objection is that such a combination assumes team resources and capabilities that I am just simply not encountering. So my position is a reaction to that.
- even delivery of stories (or units-of-work) throughout the iteration: A common symptom is fag-end delivery of all stories. This as you said, puts undue pressure on the testers to push everything over the line. I’d agree that this happens.. but it isn’t an agile problem.. it’s an execution problem.
Options: Pull work (try kanban), minimize work in progress, use velocity to select fewer stories, if that doesn’t work break stories down further such that you get testable increments every 2-4 days
- QA needs to adapt to this new model.
It certainly isn’t easy but you need to streamline your execution till you get there. The “chaos” at the end isn’t a given. However to avoid this chaos, the team needs to step up its game. The shift is harder for the QA team to master. QA needs to be productive in generating new tests – a common smell I have seen is leaving them alone or expecting them to morph into a developer. Devs need to pitch in to develop a DSL or whatever is needed for them to focus on what they are good at – testing.
* New feature – Post the iteration planning, QA sets off to automate the agreed upon acceptance tests for the new stories and checks them in. Dev teams work in parallel to get all acceptance tests to pass. So there is no reason for testers to wait for “a handoff” twiddling thumbs or planning. When the code passes all the acceptance tests, QA can then perform ad-hoc testing & log bugs if any. So you may need a day or two after this for the devs to fix them.
* Feature integration tests – i’d try brainstorming to get as many of them known upfront rather than wait post-integration.
* Regression – should be set of all existing tests and run at the push of a button.
Finally where I agree with you is that this is hard to emulate and sustain.. takes attention, skill and discipline. Failure to achieve this is just a sign that you need to improve… rather than a reflection on the method.
Good points.
To your last statement, it’s also a reflection of the teams I’m engaged with. My approach is a very specific reaction to the realities of how teams are trying (and failing) to implement Agile today. Ultimately, they have poor practices and discipline in QA and their approach is making it more difficult to improve upon the core competencies.
Perhaps a mature QA team can justifiably handle development and testing in parallel, but I’ve yet to see a truly mature QA team. I’m trying to find solutions for the teams I see.
This problem is quite old and discussed in various articles/posts/etc already. Proposed solution is a clear anti-pattern that should never be followed. Better solutions are:
1. Develop and test by feature (by user story). Testing should start immediately when user story is implemented.
2. Develop by feature and get rid of iterations. It means a transition to Kanban, which solves testing problem naturally.
3. Have as good as possible automated regression testing suite.
In our team we have this problem as well. Kanban transition was a real pain-killer.
Your proposal is good stuff, but be careful when you say something like “should never be followed”. Teams made software successfully long before the word “agile” ever had meaning within a software context, and teams are making software successfully despite following every “anti-pattern” in the book.
You CAN make software successfully and use approaches other than Agile.
All of that said, thanks for reading. I’ll definitely be articulating a follow-up due to the huge response that I’d love to get your thoughts on.
>> Teams made software successfully
Are you sure? I don’t think we are making software successfully even now with all modern tools and techniques. Most projects still fail.
Also I didn’t mention word “agile” anywhere. I just provided some techniques that fight “QA phase” problem.
While failure is certainly widespread, people have made software successfully using what are now considered archaic techniques and “anti-patterns”.
The skeptic in me sometimes wonders if all of our new approaches to building software have ACTUALLY improved the success rate. Sometimes I think we’ve just redefined success and, therefore, our metrics magically improve.
You’re right – you didn’t mention “agile” anywhere. Sorry about that.
> Testing should start immediately when user story is implemented.
Of course, ‘Testers’ can get involved before that. Anyone with a ‘can I break it ?’ mindest can be working on the scenarios (for acceptance tests) with ‘BAs’ before coding begins. Quotes indicate that I’m talking about roles here, not job titles.
I followed, and when I say “test” I’m speaking in the same way. For example, even with my “Scrum-fall” approach, I’m not opposed to testers getting involved form the outset, but that effort has to be measured and prioritized alongsize all of their other responsibilities.
Trent – when you say “I’m not opposed to testers getting involved form the outset, but that effort has to be measured and prioritized alongsize all of their other responsibilities”, I’m curious about what you mean.
Why are you trying to measure test effort? Prioritisation I can understand, but what’s the measurement giving you?
What I’m trying to do is make sure that the limited time and capacity dedicated to testing is used as efficiently as possible. While the various forms of testing at various parts of the lifecycle have benefits, one cannot believe that they are all of equal benefit. Test work must be prioritized and worked in a similar manner to other work on the team.
We should also understand the cost of each of the activities testers are involved with, so we can use the cost-benefit of those activities to prioritize them.
It’s my belief that the vast, vast majority of test teams are understaffed and simply demanding more resources is not a solution. So I’m trying to figure out the best way to get the most out of the resources that have been granted. Certainly that means accepting software of lesser quality than we’d like, but that’s a decision for the business to make given an adequate understanding of what it means (which is also an output the test team should provide).
Alarm bells are now ringing. Managing costs ? Targetting efficiency ? Can I suggests some reading on system thinking. John Seddon (many videos of his on the net) is one person who clearly presents the message “Try to manage costs, and costs go up”. There’s also a similarly themed video from Dan North promoting effectiveness over efficiency.
I can only speak from my own experience (small team) but I have to say that everything you reveal about the way your testers are ring-fenced from your developers is surprising. If you put them all in one big room and removed the barriers would they organise themselves differently ? With their eyes opened to the possibilities (one example: automated acceptance tests done before or in parralel with development) I think they might.
If Agile is incompatible with tracking (requisite to managing) cost and efficiency, then Agile is incompatible with doing business. Period. I just don’t think that’s the case.
I also find recommendations that we need to “remove barriers” and “put them all in one big room” assume, incorrectly, that all teams are self-motivated and sufficiently mature in the core competencies of their roles and responsibilities. Teams don’t struggle from being compartmentalized – I think that’s a straw man Agile proponents attack. On the contrary, teams communicate too much in too many incongruent ways.
They communicate via email, phone, meetings, water cooler discussions, instant messenger, dashboards, work tracking tools – and on and on and on. They have to sort through repetition and contradiction to understand the business problem, and the message becomes lost in the noise. Instead, these teams need better, more controlled communication, not fewer impediments to communication. This is accomplished through work tracking, daily standups, and a clear expectation that status and effort are communicated effectively.
As I’ve said elsewhere, I believe teams capable of actualizing and benefitting from the approach that seems to be encouraged by many commenters are exceedingly rare, making the advice of diminished value. If your team is making it work, by all means continue doing it. Perhaps it’s a superior solution, all other things being equal. But all other things are not equal, and my recommendation is a reaction to that.
I don’t think anyone thinks that tracking is incompatible with agile. However you seem to be saying that the necessary activity of tacking tester time and cost is one of the things that prevent tighter working between testers and and devs and all the benefits that brings (shorter feed back loops, less bugs in existence at any one time because they’re caught sooner etc). Distilling that further, it sounds like tracking tester effort is more important than delivering quality software.
Is that really what you’re saying ? I might be guilty of twisting your words a little here, but can you see what I’m driving at ?
Here’s Dan North on “Efficiency” — http://is.gd/iL2FO — well worth 50 mins of anyone’s time.
Trent,
> As I’ve said elsewhere, I believe teams capable of actualizing and benefitting from the approach that seems to be encouraged by many commenters are exceedingly rare,
Rare, maybe, hard-to-achieve, yes I’ll agree. Which is why I tacked on this comment after the “all in one room” thing
> With their eyes opened to the possibilities (one example: automated acceptance tests done before or in parallel with development) I think they might
Andy
Interesting discussion. You’ve obviously gotten the attention of the Agile community!
After reading all the comments, I don’t have much to add except to make a distinction:
- The principles of the Agile Manifesto are not platitudes just because the companies and teams you work with don’t have the courage to apply them.
This post seems to be a description of major impediments to being Agile and how to live with these impediments instead of removing them. For some companies, that may be the best they can do. That does not mean something better is impossible or utopian. It means some, maybe even most, have chosen not to improve in that way.
I seems you are doing the best you can in organizational structures that don’t want to change. Keep going!
Thanks for the comments, and I think you’ve nailed my impediments. From some of the other criticisms, it seems many think that changing culture or changing staff is simply a matter of snapping one’s fingers, but my goal is to get the most out of the resources (and quality of those resources) that I’m granted.
And I agree that something better is certainly possible, and that *may* very well be test and dev occurring purely in parallel. I just don’t think that scenario scales well to the vast majority of the teams out there, and as I attempted to point out in the original post, the teams that I’ve encountered that claim to be doing such things weren’t having a great deal of success, and some of the reasons why they weren’t having much success were beyond their ability to affect. In some cases, it was truly an aptitude issue, but those types of issues are beyond my control.
Platitude – Wikipedia, the free encyclopedia
A platitude is a trite, meaningless, biased, or prosaic statement, often presented as if it were significant and original.
I think without working processes, and the people behind them. The Agile Manifesto indeed falls into the above definition of “platitude”.
I have worked in waterfall (it was terrible at it’s worst), Agilefall (worse than waterfall by FAR, IMO) and now working in what I believe to be a genuinely Agile/scrum environment (awesome so far)
I think that to dismiss a blog post this thoughtful as ignorance or to attack it with self-righteous dogma is the antithesis of the first and last edicts of the manifesto.
“Individuals and interactions over processes and tools”
Agile/Iterative process is a “process” by attacking this blogger with dogmatic statements about how ignorant he is regarding the “right” way is to develop software is a way of putting a process ahead of a person.
“Responding to change over following a plan”
If you see his suggestions and instantly decry him by saying “All iterations must be in sync! Staggered sprints are WATERFALL!” then in my opinion, you are saying. “In agile we must all follow the same plan, regardless of our circumstances. Change is only desirable if we’re going from something else (and therefore stupid) to proper “Agile” process!”
Dogma doesn’t help anybody. I have discussed staggered sprints with my CTO and it’s something we’ve seriously considered. In the end we’ve decided to stay with the broadly accepted model all of the critics of this post seem to espouse. However, I believe my CTO is genuinely embracing the manifesto because he’s willing to consider adapting our process to fit our circumstances. At one point I think he even said “I’m trying to adhere to non-staggered sprints because it’s the widely accepted process but if it doesn’t work for us we’ll try staggering and see how that goes.”
So while I am still unsure whether or not this post is “the right way” or “a right way” I continue to encourage this discussion. I also want to applaud Mr. Nix for staying so level and professional in dealing with his most vocally dogmatic critics.
I don’t care what process you’re using, as long as you put people first and what you’re doing works for your circumstances.
Great post, thanks for keeping us all on our toes.
There is just so much wrong here it’s hard to know where to begin.
QA is not the same as Test no matter what software development process we’re talking about. But this post seems to equate QA and QC.
The “by the book” diagram showing the testers spending the first half of the iteration in “planning” is incorrect. The “by the book” answer from authors who know what they’re talking about (e.g. Lisa & Janet’s Agile Testing book) is that the testers integrate their activities with the team all the way through the iteration/sprint.
The illustration with the big red CHAOS on the interactions between testers and developers is a classic example of what Jerry Weinberg would say is not a crisis but the end of an illusion. When testing doesn’t start until halfway through an iteration, and the result is chaos, it means that in the absence of concrete, empirical test results, the team has been operating under the illusion that they were making progress. In reality, they were producing a pile of steaming code. The answer (as Ron said) is to start testing earlier, not later. It’s counter intuitive, since our instincts say that if something hurts we should do what we can to delay the pain. But delaying feedback just creates worse results even if it seems to feel better.
Trent’s subsequent comments about what he sees passing for Agile “in the real world” and his defense of that position saying that others who have experienced the more disciplined form of Agile are in the “vast, vast minority” just reinforce the fact that he hasn’t seen a truly Agile team. This post isn’t about where QA fits in Agile. It’s about how compensating strategies for dealing with frAgile contexts.
In short, I’m really sorry that this post is getting so much attention because it misrepresents QA, testing, and Agile so very very badly.
I think it’s completely fair to say that I haven’t seen a “truly Agile team”, but will say I’ve seen a large number of teams that I’m confident are representative of the mainstream in software development. As I’ve commented elsewhere, my position is a reaction to the capabilities and competencies of the teams I’m engaged with, and I firmly believe my approach is far more palatable for software teams than test/dev within the same iteration.
Every criticism seems to presume a “truly Agile team”, and I’m of the mind that such things are rarer than hen’s teeth. So I have to find solutions that fit, not solutions that I’d like.
And while I’m disappointed you don’t feel my thoughts are deserving of attention, it’s worth pointing out that the attention seems to primarily come from those who are attacking my position, not those suggesting my position is how things should be done.
Hey, Trent, I have to side with your critics, on this one, but I do want to point out a couple well-stated nuggets that I really liked.
“Why are developers afforded control over when they start new work but testers have to be at the mercy of development’s schedule?”
I feel the same way. Quality takes time. Sometimes we try to find the right process or methodology or execution to implement quality and sometimes the answer is simply provide adequate time. QA managers have a responsibility to win time for their team.
Stabilization sprints: yes. If they’re warranted they’re warranted. No further justification needed.
(I was annoyed at first at the anonymous blog, but it took me 3 seconds to scroll down to the comments to figure out who it was.)
But more annoying was that instead of listening to Trent’s concerns, some Agilistas were so mad that they saw straight through his reason and fired back with platitudes.
Ron Jeffries writes in his blog:
“It requires great skill to determine as many checks for the software before the iteration starts, and to automate enough of these to let the developers run them as they develop, not just at the very end.”
And…
“If the software is then passed to testers, those testers can run the automated checks quickly if they care to, but are able to spend their valuable time looking for something interesting, rather than manually checking basic results.”
The Agilista’s main vehicle for testing always seems to be “checking”, not investigation. It’s weird that they are quick to say there are no silver bullets, but imply that checking-via-automation is the only way to kill werewolves.
Yes, checking IS a useful approach, but what about sapience? What about investigation? exploration? discovery of emerging behavior and acting on its implications?
To me, the Agilista solution reads like this: “Testing is easy. Just have a really smart developer pair with a really smart tester to know (code) all of the requirements and then build tests that automatically catch (and fix, and retest) all of the bugs before the next sprint.”
Does that sound realistic and achievable to a team that is learning how to adapt to the promise of Agile Development?
To me, a stabilization iteration might make sense. It could mean that testing revealed problems the programmers and testers didn’t think about during Sprint Planning. Isn’t that a good thing?
It could also mean a test revealed something on a platform *other* than what they thought they’d need to support. The programmer might not have been aware of risky variables or platform dependencies elsewhere. Isn’t that a good thing to devote a sprint?
The need for “stabilization” could mean any *number* of things, not just that the staff isn’t “doing it right”.
Nix’s post shows that he’s like a lot of people when it comes to Agile — he’s trying to understand it enough to feel confident that it can works in his context. Whether the staggered sprint works or not, what’s wrong with trying it? If he truly is misguided, he’ll feel the pain of his idea and adapt to something that provides value. How very Agile of him.
First let me say that I’m not sure I’ve been on more than one supposedly agile project. However, I’ve been involved in plenty of iterative processes as both a developer, and now as a tester and I can tell you one thing I’ve learned. The longer you wait to bring the testers into the loop, the longer it takes them to understand what they are going to be testing, what needs testing, what risks there are, and that means time wasted. I have been on projects where Testers were brought in and then not kept in the loop as the project developed, so they were developing ‘scripts’ in a vaccuum basically. When it came time to test, how many scripts ended up being wrong/broken, and in error, due to changed assumptions, or needs that were not communicated. I’ve seen Testers have to push back on that enough to know that you need Testers involved in a Whole Process duration of the product.
We as Testers need in on the ground floor. We can test before there is any code. We can look at logical ideas, and begin asking questions that will help to make the code better up front before it is ready to run on a test instance. In the process, we learn about the product, where it seems to be headed and are in a better position to begin ET. Yes you could take time to try and write scripted tests in an agile world, but I think, most would agree that simply creating a document when that area of code may change, before testing, may be more of a waste than its worth. Why not determine a charter and use Exploratory Testing to seek it out.
That’s my feeling on the matter. I understand the very general feel you’ve put into this post, and in some ways I understand the sentiment, But I feel that having testing throughout the process is the answer, whether its agile, or not.
I agree with a number of your thoughts, and probably didn’t do a very good job in my post of appreciating the value testers can (and do, even on the teams I’ve worked with) provide throughout the entire process. One concern that I have, is that the approach seems difficult to manage and difficult to derive a clear understanding of where the testers have provided value. For example, in waterfall days our solution to missed and incomplete requirements was simply “another brain in the requirements meeting”. That has been correctly rejected by the majority of modern software development methodology, but in some ways saying “testers should be involved at the beginning” seems like a return to that thought process.
I think the key, for me, is figuring out where testers provide the most value, what test activities provide the most value, and how can we take the limited test resources we have and apply them to maximum benefit.
I think my rebuttal to your (valid) point of the longer you wait to bring testers into the picture, the more risk being introduced that can affect quality, is to say that you can mitigate those risks by short iterations (1-2 weeks) and by rationed participation during early stage efforts. I will say that I think the problem you mentioned where it will take testers longer to “understand what they are going to be testing, what needs testing, what risks there are” means time-wasted is incorrect. That time is still spent, it’s just spread throughout the project making the apparent cost seem smaller. So in the end, its not really more expensive, it’s just paid all at once.
My rebuttals aside, great points and good input.
I don’t know what other firms have or haven’t done. I just have my own personal experience, and in the one project most clear in my mind, when Testers were finding development being late, and having to change test plans at the last minute due to communication issues, the solution was the testers were involved. Now that doesn’t mean they have to be in every meeting, but they certainly need to know when changes are being made. That’s all I am saying.
Here’s my point. What is the point of Testing? Is it to just verify that the Happy Path works? If that’s all then, maybe you don’t need up front involvement, However, if you, like me find that requirements change, especially as projects grow, then you’ll realize that often times it is only the testers that realize the kinds of regression issues that will happen as a result to change in the code at X or Y. Many times developers only see their own niche or module, and they get focus blindness because of it. Testers can help make them aware of that issue, and the sooner they know a change is coming, the sooner a potential change to schedule to add more time to test those ‘unexpected things that a developer hasn’t considered in their own clickish meetings’ can be resolved.
Now, I admit, I don’t have a lot of experience in the Agile Realm, but I have seen iterative development in action. I’ve also seen fully designed, tested, and released project features done in a matter of weeks, so I know it can be done. The Key IMO is to keep the Tester’s in the loop.
I totally agree with many of the commenters above. However, as well as you, some of them are also missing some points. And I actually think that is because not so many of you have the roots in the testing or QA. (whatever you call it, the tasks and responsibilities of supporting developers to deliver good software)
Just as different development communities, there are different testing communities. They differ in many ways, and some are easier to adapt to Agile than others. The thing is, that you dont talk very much about what is included in the QA/testing tasks, more than mention
“Most of those teams struggle at many of the basics when it comes to test discipline and process: test management, test metrics, etc.”
And this statement just tells me how much you only have had to deal with traditional testers, that not yet have realized that their world of metrics and documents does not add to the product value for the customer, unless required (thereby paid-for).
I am a tester myself, having worked in an agile team of 9 developers for 2 years. The setting we are working in does not allow the perfect scrum/agile to be carried out, but as long as the team values the agile practices, and I as a tester carry out the tasks and test coaching needed to help developers create value, we are doing just fine, delivering and testing within the same iteration.
So please do some research on agile testing before trying to solve problems that others already have solved successfully. There is no silver bullet for it, but if all you have is one tester in a big team, make sure he is good and gets all the support and help he needs for him to be able to help as much as possible with what he is good at, testing!
Btw,
Q: What does the tester do when developers are coding?
A: Sitting in the developers lap if needed, with the deeper knowledge of PO/customer needs and requests
Here are some of the things that I’d like to suggest reframing.
Testers are NOT the QA team. Quality assurance is done by people who charter and perform the development work. The work of testers is testing and quality assistance; that is, assisting the people who are in a position to assure quality.
I think it’s a swell idea to think critically early in the sprint about the wonderful and the harebrained ideas that we might have early in the sprint, and use that critical thinking to shake out problems early. Testers might be specialists in that role, but there’s nothing to suggest that the task is exclusive to them (just as there’s nothing to suggest that programmers will be the only people to write code).
I think it’s a swell idea to identify, early in an iteration, things that we want to check. And if we want to automate those checks, then it’s a swell idea for anyone on the team who is capable of automating them to do that, be it a programmer, a tester, or a documentation person, for that matter. And then I think it’s a really good idea not to presume that we’re done thinking about what we’re going to test.
I think it’s weird to believe that everything that we might want to know about the program can be expressed in the form of checks, whether those checks are performed by humans or performed by machines. And I think it’s fanstastical to presume that code for which the writing is completed on the last Thursday of the iteration can be “completely” tested on Friday (it can’t be completely tested ever, of course, but what of that?). Checked, maybe. Tested? No. Testing is a potentially infinite process that someone decides to stop. That decision happens when they decide that a product has been sufficiently explored and investigated to inform a positive answer to the question, “Is this whole product done?” I see nothing in the Agile Manifesto (but plenty in Agile dogma) that suggests that, for a week or two, the principal focus might be on investigative work rather than code-writing work. It seems to me that if questions are piling up faster than code, it might be a good time to slow down and pay off some credulity debt.
I think it’s weird that those who tout the self-organizing team are so eager to deny the possibility of some team to organize themselves as they see fit.
—Michael B.
As a TFS Technical Specialist and Scrum Coach I’m proud to say I won’t be recommending your services. Thanks for making that easy.
Well here’s to hoping I never need your recommendation. Thanks for making that clear.
Some key words and phrases that suggest you’ve not worked on a project which has utilized the Scrum framework in practice and therefore makes me question the applicability of your leasons:
* developer focused software methodology
* differentiating developers from qa
* rigorously control what and when work enters development team’s world
* handoff
* dump of stuff from development
Hi Trent,
I have to admit, when I first read your article my head was full of all the “wrongness” that it was full of. Also, when I first read your article it hadn’t had any comments but I’d already seen it catching some attention on Twitter. Since then, I’ve read through most of the comments and had to reread the blog just to remember everything
My initial reaction was because I thought you were coming from a practitioners standpoint of “this is how I’ve optimally created agile teams”, and I think I understand now that this is not the case, but more like “I’m doing the best with what I’ve been given”…I can appreciate this situation.
I’ve never really thought of myself as an “agilista” as some put it, but I like to do what makes sense and I think the values and principles in the manifesto do make sense. Though, I’ve never really thought of agile as “right” and waterfall as “wrong” but more that, through evolved understanding and experience, what we now call “agile” is more efficient/effective and waterfall is less efficient/effective…it’s a /better/ place to be, and I personally strive for better ignoring best.
This seems to be the same attitude you take with your teams. Get them to better. It may not be “truly agile” (whatever that may mean, though Elisabeth’s acid test is a good start!). My concern is that this seems to be an end-state. You seem to say you’ve been given sub-standard material to work with and so this is the best you can do. Honestly, if the teams I coach were so bad-off that the best “better” I could get them to was the state you propose above, I would certainly make sure they (the client, and certainly the team) knew that they could be doing /way/ better. I would actually question accepting such an engagement, but knowing me I’d like the challenge
You’ve said this seems to be the case with most (all?) the teams you’ve worked with, and you feel they represent a good sample of “normal”. I work with mostly with non-tech orgs that have an internal swdev function and this has not been my experience. Perhaps one issue is that the orgs you are working for do see testing as a nice-to-have and you are being hired in to only help a separate QA team/dept? I would be curious to know what responses you get when you ask to integrate coders and testers? Would any of your clients even be willing to try it on a pilot project, or temporarily for a few iterations?
It is difficult sometimes to paint the bigger picture or help the client understand the increases in value they could experience if they elevated their teams/situation from your model (of course your model might be better than what they had before). Perhaps sometime could be spent working with the client in other areas while you are onsite? Or maybe suggesting some pair-coaching? At least let them know this isn’t and end-state.
It’d be great to chat with you more on this entire subject sometime…feel free to d msg me on twitter (@78mgb) if you’d be interested in figuring something out.
Oh…while I’m here
I don’t agree that the manifesto is just platitudes; meaning (I think) “full of nothing useful”. You mention they don’t give people any clear direction or a “thing to do” and I disagree. While the manifesto isn’t a methodology or set of practices it /is/ a set of values and principles. To me values are things you hold yourself to and continuously reflect upon. Principles are patterns to be applied to situations. While this does require more thought and perhaps a touch of introspection and is in general “fuzzier” than “do this specific thing”, I’ve found it to be very useful and I would say that I “do” a lot with it
Hi Trent,
Please consider rewriting your first and second paragraphs, for which “flamebait” is the kindest description that comes to mind. That would greatly improve the post and the response you get. In particular words like “squirm”, “miracle”, and “dirty secret” are guaranteed to provoke anger and misunderstanding.
Suggested rewrite: “Many self-identified Agilists lack an adequate understanding of QA activities, and Agile discourse in general is overly focused on the developer’s world”.
If you say it that way then, right or wrong, it’s at least clear that it’s your opinion, that you’re prepared to own it and take responsibility for it.
Your criticism is noted, and valid. That said, rewriting is dirty pool, so it has to stand as-is. I’ve tried to be as clear as possible in my comments that I do own and take responsibility for it, even as it is written.
I do think the rhetoric I used does capture many increasingly mainstream beliefs about Agile proponents, and unfortunately, the response to my article didn’t do much to dispel that notion. It was biting, but it does articulate my belief that Agile is generally oversold as a solution to software development ills.
So now that I’ve banished my pissy-pants attitude, here are my, hopefully, constructive thoughts.
Differentiating the QA group from those development team, even in thought or words, is disruptive of collaborative efforts. I can’t take credit for this, but I find it is more helpful to describe the development team as those that develop the product. That can include coders (often a better replacement word for devs), testers, tech writers, copy writers, magicians, etc. Isolating any one of these roles from the team in the context of a sprint that demands their attention is the antithesis of a crossfunctional team.
Inability to accomodate activities of any one role in delivering working software during a sprint suggest’s either a) the sprint is too short and/or b) the team is accepting too much into the sprint.
2 week sprints are a common smell I see. It’s REALLY hard for many teams and products to deliver “Done” software in two weeks which also delivers value. That’s not to say many teams don’t do it, but for many products it is hard.
There is nothing wrong with a team selecting only one Story for a sprint if that is what their capacity allows. 1 done story that can be shipped is better than 10 unfinished stories that cannot.
I do offer my apologise for the arrogant comments and thank Adam for calling me on that. Nothing productive can come from such things.
Just a little correction of your use of the term QA in connection with testers. That is not really correct. Testers test and do testing and verification work. That has little to do with QA. It is a part thereof though. QA is a management practice and task that has it’s focus on methodology, process and/or some form of audit.
In a Scrum/Agile context QA would be next to everything the ScrumMaster does for example. Things like adhering to process, implementing process improvements, guiding dev practices,….
Although I do admit the range of tasks expected from a tester is different from company to company. This is especially true for agile but even then I’d challange anyone calling a tester/testing “QA”.
Very, very interesting post and discussion! It happens all too seldom that somebody shares thoughts and conclusions from real experience, and from so much of it. I’m a coder – and I can only dream about working with dozens or even hundreds of teams. Thanks for sharing Trent!
My all time favorite book on software development is “Trenches” by Henrik Kniberg, probably because it is based on real experience. Let me quote from Chapter 14, How we do testing:
“The acceptance test phase hurts. It feels distinctly un-agile. Although we can’t get rid of it, we can (and do) try to minimize it”.
Kniberg also suggests that you might try to “Define some sprints as “test sprints” where the whole team works as an acceptance test team”.
It seems to me he basically agrees with Trent.
Thanks for the heads up on that book. I’ve got the PDF downloaded and will try to go through it over the next couple of weeks.
As I’ve reflected on this topic, I’m finding that my critics make assumptions about teams, their capacity, and their capabilities that I generally don’t accept. If I’m correct, then their advice doesn’t have all that value in the wider world of software development, because teams are generally tasked with getting the most out of what they have.
“test and develop in the same iteration has a magic point in the iteration where the handoff occurs” that’s true only up to a point – which would be the test execution phase. And your diagrams reflect that perfectly. However for my SCRUM team the Quality Assurance starts with Backlog Grooming (formulating how to test) and ends at the Sprint Review (ensuring all DoD items are complete before a demo to PO). The way we achieve this is NOT being slammed at the end of Week 2 but instead delivering smaller stories first. Your test execution is then bite-sized instead of manic at the end. And tester-Dev collaboration is high so no surprises and quick execution phases. There is usually time for all to ensure Quality is delivered – both Devs and Testers.
I should point out we have failed about 50% of our Sprints – but have done it gracefully. Meaning only a couple of things got missed off. Not whole Backlog Items. Usually due to external issues (infrastructure issues, change of direction) In this company failing does not mean sharp intakes of breath and poor-you looks. It’s all about transparency.
Maybe, I am a bit late to stumble upon this thread and comment… I happen to manage a maintenance project and had the same issue of managing tester’s work. Earlier, we”ve committed to the customer that we will deliver atleast one enhancement every week. The first iteration was as usual, disastrous. We committed on 4 enhancements and didn’t deliver even one. We discussed this in the retrospective and decided to apply timeboxing to development itself. ie, The first enhancement will start on Monday and no matter what, an alpha version is deployed for testing on Wednesday. The testing would be done for a day or two followed by bugfixes and finally tested on Friday and released on Friday. In the meantime, the developer would work on the next enhancement and will deploy it for the final testing on Friday. Though we are able to test 2 rounds on the first enhancement we ensured that the 2nd enhancement would also be tested atleast once if not twice. We were able to stick to this team rule almost till the end. This was my first agile implementation and we didn’t use TDD or Automated testing.