1) Maintaining test environments with unmonitored and, as a result, unknown and unrepeatable data, under the assumption that it is the functionality of the application that is important and the data it is fed is immaterial.
2) Testing end user sessions on a different platform, under the assumption that there will be very little difference in functionality (”Firefox on Ubuntu is the same as on Windows”)
3) Pointing new-starters to a wiki-page, under the assumption that anything else they need to know shall be relayed by ‘asking around’
4) Arranging development teams such that each person only has deep understanding of one part of the system, under the assumption that all issues that arise will be easily solved by experts collaborating at short notice.
5) Loading test environments with various non-production-like configurations, under the assumption that removing them and accessing the system from an external network will be exactly the same as interaction from an end user
6) Writing test suites and user acceptance scenarios without involving the business, under the assumption that the dev and test teams know how the end user will interact with the application just as much as the business
7) Remaining with a project strategy that clearly isn’t working, under the assumption that because everyone knows how it works it cannot be improved
8 ) Believing that exploratory testing involves dividing up areas of the system for QAs to play with for a few days, under the assumption that it shall be as effective as the clearly defined, structured practice outlined by Cem Kaner & co.
9) Writing automation test suites with no clear structure or framework, under the assumption that this produces a full regression suite and the product is thus well tested.

All the above examples are illustrations of assumptions that have been made on the SDLC. Each one lowers the quality bar and introduces unknown levels of doubt to the project model.

Whilst it is perfectly understandable and normal that cost, time and resource restrictions always play a part in the decision-making for and during a project (e.g. building a smaller test environment than the production equivalent due to financial constraints, removing functionality from the backlog in order to release earlier), these are value-based decisions with a known consequence.

There is a clear dividing line between these project decisions, based on factual origins and known consequences, and those based on unfounded assumptions producing unknown results. The latter is where quality is lost in projects.

Regardless of whether your organisation has dedicated QA personnel or not, there comes a time when the people who have built a feature would like ’someone else’ to have a look at it prior to a release. Someone who will provide an objective, prejudice-free assessment of the work. This honest feedback is invaluable just before ‘flicking the switch’ and can save a lot of time and money, and prevent embarrassment.

The QA/test team may have worked wonders throughout the sprint and prevented numerous issues from remaining in the final product but, along with the developers, they will have been 100% focused on the feature and are thus at a slight disadvantage when it comes to user-focused testing. They know it too well and can be prone to overlook the obvious.

Although the product owner is representing the customer/end user, they also have been involved full-time on the functionality during the sprint and have their ideas on what will work and what won’t. Through in-depth discussions their opinion may have become distorted on what works and what the customer won’t like.

What is needed are fresh pairs of eyes provided with a mission to ‘play with the feature’ and see what they think. An end user, someone who has been handed an app for the first time and asked “What do you think?”. Considering first impressions last, and with the myriad of options available in the software space creating a very competitive environment, something that either isn’t intuitive, doesn’t work as expected or just looks crap is going to die a death in production.

Now, employing a team of usability experts, crowd-sourcing or a team of offshore trained UAT people will certainly get the job done, but it will also certainly cost What I suggest is to use the people in the company and call a bug bash. Teams are given 30 minutes to play with another team’s new feature and provide feedback within that timeframe. The related product owner then reviews the comments and acts on them as they see fit. Here are the core rules :-

1) The exercise lasts 30 mins, not a minute longer.
2) A team shall be informed directly before the bug bash of what they’re going to look at (make it at most a 3-word title, to provide focus but also to allow people the freedom to do as they wish within that guidance).
3) Everyone in the company can join in (it’s actually encouraged).
4) All testing is performed on the same test environment.
5) No discussion is allowed between team members.
6) Feedback can consist of bugs found, questions, comments, general feedback, whatever.
7) The remit is to “play with the feature and see what you find/think”.

After the 30 minutes, everyone returns to their normal work.

The method of submitting comments can be via an existing tool in the company. I’ve done this with Jira, but it runs the risk of people responding to issues and then a chain of communication starting up, thus dragging the bash into the rest of the day. What is wanted is feedback for the product owner to do with as they see best - even having people use post-its and hand them in after the 30 minutes would work.

The benefits of this exercise? It’s a short, timeboxed activity where technically-minded ‘guinea-pigs’ provide focused information on a new feature and its integration with all others in the system. It also acts as an unofficial demo for each team and as a learning exercise for all.

The downsides? To be honest I can’t think of an argument against a company forming as a team to check what they’re about to provide to a customer.

One pointer might be the timing of the event during the sprint - to perform it so near to a release may prevent any meaningful changes being made. My outlook on this is that by the end of the sprint there should be no catastrophic news to broadcast so the bash should mostly uncover small glitches in functionality, recommends for future work and overall opinions.

And if something really bad is found then it can be scheduled for resolution in the next sprint and, most importantly, it has been prevented from going live.

It just makes great QA sense. So do it

My view? It depends on the definition

When it comes to asking whether a tester who is involved in a fully functioning scrum team, who is working daily with developers to define acceptance criteria (yes, I believe everyone should have a say), collaborating on test environment construction, investigating broken builds and similar complex tasks, are they 100% independent and flying the flag for the QA dept? I’d have to say no.
What they’re doing, and I believe it’s what they should be doing, is working with the team to build the best possible feature in that sprint.

If that’s the case, when is a tester working as an independent consultant for the QA cause then? Well, confusingly enough, at the same time. Whilst they’re knee deep in code and environments and repository failures, the tester needs to still keep their QA hat on and think of the bigger picture. There has to be someone in the team tasked with ensure quality assurance is being thought of and has a voice throughout the sprint.
This might take the form of reminding everyone to document their unit tests, double-check with the product owner on a specific piece of functionality and broadcast the findings, point out that code coverage has fallen for testing (be that for any type), whatever. They must remain focused on ensuring the best quality code is produced and is exactly what the product owner wants. And this requires an independent mindset to achieve it.

So the life of an agile tester is a complicated one. They have to be independent whilst not, involved up their neck in code but also standing back and observing the team’s productivity, and ensuring that there is adherence to the quality processes in place. A tough call indeed, and one that I think should merit huge salaries to compensate

During the majority of the project the team I was in performed agile testing - along with providing instant feedback to the developers on evolving functionality, we built an automated regression test suite for UAT-level coverage. At the same time though, due to lack of appreciation of the problem in hand and not looking to the future, the project management team neglected to work out what would happen when the two paths would ´collide´ during the final SIT/UAT testing phase. This lack of foresight meant that when the teams using the traditional cycle wished to start the six week-long testing of their product with ours in month 5, they required a stable and fully developed application on which to do so. The problem was that at that time our team was still producing new features which, until completed, would not allow this form of structured testing to be executed. No thought had been given to the need for all integration aspects to be there - I had raised it as an issue early on but was still unaware of the size of catastrophy winding it´s way to our door.

What ensued was a very problematic and chaotic spell of SIT where daily deployments were attempted and large numbers of blocking defects were uncovered. This introduced huge overheads such as continual environment updates, constant questioning on exactly what functionality was there and available to test and the basic fact of frequently changing code providing real instability and thus multiple test reruns. It also meant that the testers had to get involved in manual testing phases of integration between the modules whilst trying to keep up with new feature development.

The clear points it showed me were

1) If a project team permits the use of agile methodologies for development of one module and not for others (for whatever reason - in this case due to multiple companies being involved), there is a much higher risk of not completing a thorough testing effort and much more administration/tracking overheads.

2) If agile methodologies are to be used, EVERY task must be put into the product backlog. Attempting to manage the project with more than one base planning method will (I think) come unstuck every time (please correct me if I´m wrong!).

Rather than plan the delivery of functionality for every sprint in the project on day 1 and then try to link it all to the waterfall plan of other modules over a 6 month (or so) period, compile the normal product backlog and add the integration tasks for the other modules to it. Then manage the project on a sprint-by sprint basis, monitoring expected delivery dates from the traditional cycle(s) and adding them to sprints a week or so in advance. Then there will be much more accuracy in estimates, expectations and MUCH less guess work.

Or just do it all using agile, it´ll be much easier

At present I´m working on a project where I´ve had to split the test team into two due to several companies working to build a complex product together. One team is performing agile QA in a sprint team, the other is working ´the old way´.

It’s a sweeping generalisation because there are a lot of quality assurance tasks that an agile tester does before they touch an automation tool - they have to work with the team to define the acceptance criteria of the user stories, define and document the test environment to be used, perform exploratory testing on whatever code is available, review developed functionality etc.

However once they’re ’settled into’ the sprint there are developers performing TDD and producing code in a Continuously Integrated Environment and testers producing test code to ensure business-level descriptions/requirements are met.

Of course there are all the defects to be tracked and retested, and associated documentation to be produced but I’ve found that when I provide this eight-word description to people new to agile/agile QA/basically non-traditional ideologies in general, a glimmer of comprehension and (almost :-)) acceptance appears.

So, has anyone else found a quick way to describe to someone what an agile tester does? It may sound like a trivial thing to discuss but having been first QA person into several companies now, it’s very good to have ‘buzz-phrases’ to hand that provide quick descriptions to people who still have the mental block.

Taking time out from the day to day work problems and spending it instead on listening to experts in the field, meeting them and having the opportunity to pick their brains face to face, this can refocus you and prevent getting bogged down with particular problems.

So, I’d recommend as many people in the Spain area to go to this :-

www.qatest.org

The Spanish software test community needs to work together to raise the profile of testing and agile QA in general.

At the moment numerous teams (containing the usual mix of product owner, scrum master, developers and user experience personnel) are working on their product. There is a QA team who’s task is to provide resources to all - and yes, the usual story persists in that there aren’t enough of us to go round!

So this raises the question, how to provide the best possible QA services when you can’t work on a project 100% of your time?

I know, one of the key rules of agile is being broken here i.e. the project team can not be interrupted during sprints. In an ideal world we would simply add sufficient QA people to be able to maintain the continuous creation and maintenance of automation test suites and provide QA input on all the user stories. The problem is that the ideal world doesn’t exist (or not yet anyway - I still haven’t retired as a millionaire :-)).

A way to tackle this problem is to change the priority of tasks that QA people will do. Instead of focusing on the automation tests, use what precious time is available to ensure the other required work with the developers is as thoroughly completed as possible.

This means performing the usual informal chats to learn about the product, provide verbal input on scenarios you’d like to see unit tested and put into the CI builds, be super-thorough on explicit completion criteria for the User Stories … basically attempt to influence the mindset of the team to focus even more on QA. Yes, this is already the role of an agile QA person, but spending time frantically writing the correct amount of automation tests (and ultimately failing!) is not what I see as making the best use of the resource.

Depending on the time available after performing the QA tasks, automation of UAT will assist the team greatly. Focusing on the completion criteria or other scenarios stipulated by the product owner, these scripts will ultimately provide the high level regression suite and give the rest of the team confidence and visibility that the product is indeed developing sprint after sprint.

This QA task prioritisation is a good way to ensure the load of work doesn’t bring the person to breaking point and working nights and weekends - rushing the verification of code and practices always results in poor quality.

So, anyone agree/disagree with this? It is a bit ‘controversial’ to not attempt to complete full automisation of testing for each sprint but until such times as testers have 3 pairs of hands or cloning machines, limits to what can be achieved need to be set.

Without the discipline of staff receiving new process ideas and actually acting on them, it’s a lost cause from day 1. Many very intelligent people have voiced some fantastic ways to use agile in certain circumstances, when to perform xyz, how to react to abc and when not do anything, but if the people involved do not ’stick to the programme’, failure beckons.

With this in mind, how can a team be ‘energised’ into willingly accepting change and, probably most importantly, continuing to work with this change as the weeks pass by? From my perspective, it firstly requires people who are open to adopting new practices, who are willing to give something a try and who are enthusiastic enough about their individual tasks and about the team. It’s a team game after all and if people are only focused on their work and how good it looks to others, they’re not going to have much time/interest in discussing the latest tweak in the process.

A highly productive agile team requires people who are capable of social interaction and who grasp that the team’s performance is severely hindered if even one member doesn’t pull their weight. Everyone needs to be ‘up for it’, asking each other about their tasks, showing interest in what they’re doing and how they are going about it. And this leads to the QA link.

An agile team which contains a QA person needs to have everyone genuinely interested in what that person is there for. “Why do we have a QA person when we’re doing TDD?”, “What are they doing to improve the way my code works?”, “Why should I listen to their recommendations when I think my unit tests are the bomb?”. If all team members keep in constant contact with the QA person (and the QA person knows what they’re doing :-)), the difference in productivity is very noticeable.

This is where the discipline comes in - people need to leave old habits behind. No resistance to reading QA test plans, no resistance to discussing what seem like bizarre use cases and certainly no resistance to actually doing some manual testing if required. And if the QA person can show that the functionality isn’t as requested, it’s back to the drawing board.

From these points it shows that a QA person has a lot of responsibility in an agile team, so it takes someone with the technical knowledge, drive and (I’ll use it again) discipline to bring out the best in an agile QA function.

So how best to perform QA in say a two week iteration? One way is as follows :-

* The user stories are chosen and the team members select which tasks to perform from this.
* Developers are given the morning to sit and work out which unit tests and low-level functional tests they will write. The QA person then meets and discusses this with the developer, finding out (at the high level) what the expected functionality is.
* The QA person has the opportunity to request that the developer adds more tests to the suite, change existing ones, etc - the idea is that a QA input has been provided before the developer writes a line of code.
* Agreement is made and the developer starts to write the unit tests (with Javadoc/XYZdoc to create documentation) and then the code.
* At the same time the QA person works their way round the rest of the team to a) learn the functionality to be implemented and b) to provide QA input.
* The QA person takes the code from the previous iteration (which is working and usable) and writes high level tests using a QA automation tool. These tests are demonstrated at the end of the present iteration. Their structure is already known as they were performed manually in the previous iteration.
* The QA person performs exploratory testing on the newly developed functionality and compiles a manual test suite to verify this.
* The Developers complete their unit tests, javadoc and code, all is verified to build and the tests pass.
* The QA person’s scripts from the previous iteration are continually run against the present build throughout the sprint to ensure that no regression issues creep in. This, along with the manual tests passing, ensures a stable build to release to production.

Co-ordination, communication and co-operation are the keys (as in every project). Without this the rest of the team quickly loses touch on whether the QA people are manually testing the present release, automating the tests from the previous one, finding bugs in either or reviewing the unit tests. The fact that there is an ‘overlap’ of activities for the QA person between iterations means that all automated tests must be written, related bugs created and resolved, and all verifications passing for the end of a sprint. If anything drags into the next release, the structure quickly breaks down.

So, that’s one process that works, does anyone else have one? Differences from the one described?

——————-

Addendum: 15th August 2009

After reading the above article again recently, a glaring omission came to the fore. In the structure I described, I mentioned that code from the previous iteration would have automation tests written for it. Now this is bad. Well, it could be a lot better at least.

No doubt I wrote this idea from past experiences (as I’ve never been given ‘formal Agile QA training’ - which raises a point, is there such a thing?) and it highlights a big problem in Agile QA. If there’s compromise to be made in adoption of agile processes, the traditional ‘last piece of the puzzle’ usually carries the can. And that means code not being ready for demo or test until the final day of the sprint.

A highly functioning agile team will take the user stories and develop them in such a way that every 2/3 days there is something testable. In the classic case of the login user story, within a couple of days there’ll be an interface (which will no doubt change during the sprint) with entry fields and some form of simple backend to receive values.  As the sprint progresses the rest of the functionality will be added (e.g. parameter checks, link to a member page, error handling, db structure changes …).

The key point is that there is something of substance that can be checked through exploratory testing within the first couple of days and there will be more work that can be checked throughout the rest of the sprint. This allows automation scripts to be written within the same sprint such that, when demo time arrives the QA person has their corresponding automation suite ready - and demos this as well!

Think of it, if a product owner gets to see a working demo of new functionality, and then gets to see all of the testing scenarios checked for it. Their level of confidence in the stability of the product is sure to rise, along with all members of the team.

So, if your agile team can function in such a way that the necessary automation scripts are written and working for the present sprint, you are doing very well. Showing these scripts is firstly a way of advertising what QA is doing for the project and the benefits it provides, and also really helps to build the respectability of the QA effort company-wide.