QA Test challenges & possible solutions

QA/Test problems

The real challenges that Q.A faces while testing Mobile Applications are as below: • Time pressure • Inadequate processes • Lack of physical devices • Lack of test plans, Validation issues • Diversity in Platforms and different OS currently present in the market such as Android, I-phone etc.

The top 10 people challenges have been identified:

Training in Testing Good Relationship with Developers. Good Relationship with Customers. Managers need to understand about testing. [Think – Do your Company has Culture of Testing?] Using of Tools. Time on Testing Fighting a lose-lose situation Having to say No Testing “Over the wall” Software Having to say “no”

iOS Unit Testing Challenges:

Toooo many steps to get started No green bar/red bar Can’t run individual tests Unit test template code contains to much Unit tests can not be debugged out of the box Little documentation Multiple targets Only includes very basic testing support No mock framework included

Other issues:

lack of automation few devices lack of training/education. lots of tools, too few process. improve communication/collaboration. better use of continuous integration/testing. error prone million lines of code, zero test no best practice no code review lack of devices. No cross platform test solution. *drawback worth mentioning around using the iOS simulator is that it can only be run under OS X, and it doesn’t seem possible to run multiple instances of the simulator at once. This means that parallelizing your tests will involve running multiple OS X host machines, either physical boxes (we have used mac minis successfully) or virtual machines.

1. Most of the build are failed and Most are unstable, In such cases most of the defects get hidden and we are not able to find them.

2. This functionality is developed before I joined this project.

3. The patches were applied by development team at the end of release date, And due to these patches most of the closed defects were re-opened and this defect is one of them.

4. Some developers forgot to Check-in code while new build was in progress and hence this defect came about.

5. The most Common issue is that When I was testing this functionality it was working fine, I think the system was disturbed and thats how this defect came about.

6. Some one changed the default configuration. Bad Signs…

   Do you have any long-running branches that are not closed? At integration time, do you run for the hills? Integration is a painful and long event? You defer integration?You have no confidence in the quality of your (branched) code - before it is integrated into the mainline? You have no way of consistent reproducing dev, stage, UAT environments? You all work in silos: developers, QA and Operations?

. QA/Test goals - possible solutions

Ground rule is to start testing as soon as the developing starts to ensure sufficient quality and avoid overrun schedules due to bugs being found late in the sprint or iteration. Testing is a part of any functionality or non-functional requirement. So the features or tasks are completed when the implementation is done and the unit tests are passed.

Getting programmers to agree to using static analysis on projects, running them against an existing codebase. Preventing defects that impact application security, reliability, and performance Complying with internal or regulatory quality initiatives Ensuring consistency across large and distributed teams Increasing productivity by automating tedious yet critical defect-prevention practices Successfully implementing popular development methods like TDD, Agile, and XP

Static analysis — static code analysis, data flow static analysis, and metrics analysis Peer code review process automation—preparation, notification, and tracking Unit testing — JUnit and Cactus test creation, execution, optimization, and maintenance Runtime error detection — race conditions, exceptions, resource & memory leaks, security attack vulnerabilities, and more

Metrics gained by the quality assurance are used to steer the project and to analyse the project status and quality. The following metrics are gathered and used: • Number of defects and their majority in the defect tracking system • Code review session logs • User feedback form usability tests • Test log from usability tests • Test cases and test logs • Code coverage • Exploratory testing logs and charts • Acceptance test logs

Why (automated) testing? Find a problem early and you ●Can fix it quickly ●Save money on QA testing ●Do not get negative customer feedback ●Deal with feature requests instead of bugs ●Avoid production problems ●Can refactor (and change) without breaking old stuff

Save time, effort and tedium of repetitive testing with each iteration. Faster feedback for developers. Higher-quality before app goes to QA. Prevent regression. Reduce need for manual testing. Provide higher test coverage. Extra form of documentation.

what lies ahead... Automated UI tests using screenshot diffs ●Install and upgrade tests ●Performance tests ●Stability tests (low memory, memory usage...) ●Deployment automation to many markets ●Continuous deployment

. Automated unit and integration testing: Run all and individual tests easily. IDE integration Debug mode, reasonable log output. Asynchronous tests with timeouts. Expressive assertions, Mocking. Continuous integration.

Why Unit Test? Improves design Facility change and refactoring Simplifies integration Provides executable documentation

Automated acceptance testing Small distance between user stories and actual test code (DSLs?). Expressive and efficient to write. Extensible High-level (robustness against “minor” UI changes). Test in realistic environment with real data on multiple real devices, on multiple iOS versions. (but one device at a time, and only iOS>=4.0). Continuous integration.

Build, static analysis (clang), + run unit tests with code coverage.

Run build and tests for each check in Setup for various development branches Run release build as one click action ●Create website with project details as well as analysis of build history ●Provide user interface for non development user e.g. to publish release

Avoid “works on my machine” problems - Reproducibility ●Free up developer machine/time – I don't have time to run all tests before each commit ●No IDE dependency (less setup problems) ●Rapid feedback in team ●Improved communication Capture logcat from emulator/device

Beyond testing Static analysis Test code coverage Trending Look at Sonar Automated merging of feature branches Site build for documentation and more Automated UI tests using screenshot diffs Install and upgrade tests Performance tests Stability tests (low memory, memory usage...) ... towards Deployment automation to many markets Continuous deployment Find problems before your customers find it Implement new features confidently without breaking existing functionality Use continuous integration to improve communication in your team and software quality

Few Useful Points on Role of Testers in the team:

• Testing usually starts once the development is completed.
• Testing needs to be incorporated into every stage of the software development Life Cycle.
• Testers need to validate whether the requirements specification documents are in line with the needs of the project/organisation.
• Testers need to ensure that the design documents are in line with the requirements specifications.
• Testers need to ensure that the test cases and test plan are created based on the requirements specifications.
• Testers need to ensure that the defects are detected before the test cycle starts so that the cost spent will gradually decrease.
• Detecting defects at the later stage of the product will increase the cost spent to rectify the defects substantially.

Agile methods are a family of development processes, not a single approach to software development. Agile follows ‘adaptive’ planning rather than ‘predictive’ planning in conventional Waterfall Process.

Customer satisfaction by rapid, continuous delivery of useful software. Working software is delivered frequently (weeks rather than months) Working software is the principal measure of progress. Even late changes in requirements are welcomed. Close, daily, cooperation between business people and developers Face-to-face conversation is the best form of communication. Projects are built around motivated individuals, who should be trusted Continuous attention to technical excellence and good design. Simplicity Self-organizing teams Regular adaptation to changing circumstances