Beruflich Dokumente
Kultur Dokumente
0 er-t5 Ov o ho w
Adam Laiacano
Data Scientist and Engineer at Tumblr
Improve, Dont Abuse: Use Big Data for Your Customers, not Against Them
A BZ Media Event
Contents
Feature
Software Quality: More Than Testing ........................4
By Lisa Morgan
Test/QA Automation
Increasing Testing Productivity with IDTs Automated Test and Re-Test (ATRT) ......................15
Innovative Defense Technologies Gold Sponsor
Test Management
Real-Time Test Management from Zephyr Drives Instant Visibility .....................17
Zephyr Gold Sponsor
Mobile Test/QA
Master Software Quality with Orasi ......................19
Orasi Silver Sponsor
Editorial Director Alan Zeichick Senior Art Director Mara Leonardi Copy Editor Adam LoBelia
Lead Writer Lisa L. Morgan Customer Service/ SD Times Subscriptions +1-847-763-9692 sdtimes@halldata.com
BZ Media
President Ted Bahr Executive Vice President Alan Zeichick
BZ Media LLC 7 High Street, Suite 407 Huntington, NY 11743 +1-631-421-4158 FAX +1-631-421-4130 www.bzmedia.com info@bzmedia.com
A Supplement to SD Times March 2013 2013 BZ Media LLC, All Rights Reserved
MARCH
fast as possible without understanding the business implications of doing so, said Parasofts Ariola.
Speed Kills
Many organizations have embraced agile practices with the goal of releasing smaller pieces of software faster. At the same time, the nature of software and the environments in which it runs have become more complex. Balancing the needs for both speed and quality can be challenging as software ratings and reviews indicate. The world is operating from a quality perspective the same way its always operated: It essentially views tests as something that happens at the end of a cycle, and so errors are discovered really late, said Ariola. While testing earlier in the software development life cycle isnt a new concept, the notion of what should be tested earlier is changing. Parasoft and TechExcel are good examples. Parasoft has a virtual test environment developers can use to more effectively prevent errors from appearing in actual test environments. It allows developers to do functional testing and performance testing in addition to the usual unit testing and static analysis. Parasoft also has a development-testing platform that enforces quality processes, including static analysis, unit testing, coverage analysis, code reviews and runtime testing, so software teams can ensure that quality-assurance standards are being met consistently. TechExcels DevTest test-management and quality-assurance solution also enforces quality standards. It integrates with defect-tracking tools and allows test cases to be tied to requirements, so quality can be built into products earlier in the software life cycle. You need to quantify quality in a standard way that can be applied to both agile and non-agile teams, said TechExcels Zhou. Quality control should begin as soon as the first requirement is committed. Once you know what features youre going to build, you should be thinking about how they should be tested so they can deliver a high-quality experience to the end users.
MARCH
be executed faster than would be possible to do manually, dit. When the failures occur, users blame the brand, not the although testing more doesnt always mean testing smarter. platform. Clouds are also lowering software-switching costs. Developers get all this time to write the code, then they Many software publishers have replaced or complemented want to shove it into a pipe where automation happens and their on-premise solutions with SaaS-based alternatives. then clean code comes out, said Parasofts Ariola. AutomaMonthly SaaS subscriptions are currently popular, although tion is exposing just how brittle the testing infrastructure is. some users are clamoring for pay-per-use and shorter-term Parasoft and TechExcel are addressing the higher-level probmodels. The trend is extending to software testing, which is lem of having enforceable quality standards in place that apply why Orasi offers a cloud-based version of HP LoadRunner that to both automated and manual testing. Innovative Defense is available in 24-hour increments. Technologies provides consulting services that help organizaThe lower out-of-pocket costs and easy onboarding of SaaS tions apply automated testing practices in solutions and mobile apps versus traditionways that align with predefined outcomes. al software translate to lower costs of IDT also offers military-grade test-manageWhile developers wont be switching, underscoring the need for comment and analysis solutions that are used by prehensive software testing. Parasofts Arioreplacing QA engineers the U.S. Department of Defense, DoD conla replaced an SaaS-based marketinganytime soon, there is tractors and enterprises that are doing lifeautomation solution because it failed to cycle software testing in highly complex load critical data in a timely fashion. The merit to testing more environments. final straw was an executive presentation aspects of quality earlier. While automated testing can accomplish in which the graphics failed to load initially, more in less time, fast release cycles can followed by graphics that displayed stale nevertheless be negatively impacted by time-intensive status data. In three days, he migrated all his data assets to a new meetings and traditional reporting practices. Zephyr is platform and integrated the new solution with Salesforce.com. addressing this issue with real-time test-management soluWhen mobile app users get frustrated, they simply delete the tions that provide 24x7 visibility into the status of tests via app. In the case of on-premise software, customers may simply dashboards. decide against upgrading if the switching costs are too high. Customers lose confidence when your software quality is Complex Environments Complicate Testing poor, said TechExcels Zhou. You can try to make up for it The growing number of platforms, protocols, systems, with a fancy marketing campaign, but if youve lost your cusdevices, networks, apps, software components and software tomers trust, theyll stick with the older versions. This cost of versions continues to complicate software testing. TechExmaintaining old versions is a huge expense for ISVs, and it can cels Zhou said that to ensure quality, test cases have to be also hurt your future growth. designed for each requirement, and that the permutations Bottom Line for all the possible runtime variables need to be specified for Quality cant wait for late-stage testing if software teams want each test case. to ensure it. While developers wont be replacing QA engineers Parasoft is addressing environmental complexity with simanytime soon, there is merit to testing more aspects of quality ulated test-environment and cloud/SOA/API testing solutions. earlier in the software life cycle in a manner that doesnt derail Were seeing major disruptions in the SDLC today because software development. While agile development teams often clouds are challenging the concepts of where software lives include developers, testing/QA, and maybe end users, testing and how it should behave. We also have mobile devices that are by developers is not a substitute for QA, and QA is not a substiforcing us to rethink the user experience and whats importute for designing quality in to begin with. tant, said Ariola. Then we have the concept of APIs, which are Zhou suggested assigning QA floaters to development driving composite applications, so testing is a more complex teams so that more comprehensive tests can be created while endeavor. software is being developed. The alliance gives developers betClouds represent three problems. First, they are forcing ter insight into quality, and QA teams get better insight into software teams to expand testing outside their own walls. Secwhat needs to be tested and why. ond, when services and software run in the cloud, they are subWe have to remove the constraints associated with waiting ject to cloud disruptions. One high-profile example was the until a project is complete in order to test, said Parasofts AriNetflix outage that occurred on Christmas Eve 2012, which, as ola. If you really want to reduce risks, you have to [test earlier reported by TechCrunch, was caused by errors caused by Amain the cycle], compress your defect-discovery curve, and prozon Web Services Elastic Load Balancing API calls. Amazon mote methods that prevent errors from getting into the softWeb Services disruptions have also negatively affected such ware development life cycle in the first place. z high-traffic sites as Foursquare, Instagram, Pinterest and RedTHE 2013 SD TIMES SOFTWARE QUALITY SOURCEBOOK MARCH
platform, software teams can implement and enforce processes that reduce the risk of software failures. The platform ensures more comprehensive static analysis, unit testing, coverage analysis, code reviews and runtime testing, as well as more accurate traceability. When you shift quality left, developers become inundated with fixing defects at the same time that they are responsible for building new features. Because a developer only has so much time, you must promote methods that prevent errors from getting into the software development life cycle in the first place, said Ariola. Service Virtualization and Development Testing are now converging in a way that not only helps organizations start testing earlier, but also reduces the amount of defects that are injected into the software. This means that you can not only test earlier, you can also actually test less and go to market significantly fasterwith more innovative, better-quality deliverables. Notably, Parasofts Development Testing platform enables are the days of end users organizations to implement being QA testers. Software and enforce a defect-prevenquality must be achieved tion strategy that ensures best with a repeatable process. practices. Developers can build Wayne Ariola security into their applications, prevent costly defects, and get the traceability and auditability they need to comply with industry standards. When software teams embrace policy-driven development, they can assure the reliability and performance of their applications. We introduced the software policy concept eight years ago in the industrys first development testing platform. Developers always say that theyre going to do static analysis or unit testing, but there was no way of measuring the effectiveness of it or determining if it was being completed, said Ariola. Using our Development Testing platform, you can declare how the code should be delivered to satisfy business expectations. Then you can automate adherence to the policy, aligning development with business goals.
Reduce Risks
Delivering more software features in less time means trade-offs between coding and testing. With Parasofts Development Testing
THE 2013 SD TIMES SOFTWARE QUALITY SOURCEBOOK
ity is not always contemplated at the development-task level. Using DevTest, software teams can account for all test phases using detailed histories of test cases, data and test results. The centralized test library also improves test standardization, reuse and revision control. DevTest allows you to comprehensively represent all aspects of quality assurance by utilizing test cases that are predefined in a template, said Zhou. The template defines the functional testing procedures and the expected result. DevTest allows easy scheduling of automated and manual tests, and also provides real-time test-result reporting, tracking
THE 2013 SD TIMES SOFTWARE QUALITY SOURCEBOOK
11
TEST/QA AUTOMATION
Gold Sponsor
Increasing Testing Productivity with IDTs Automated Test and Re-Test (ATRT)
that characterizes the systems behavior and evaluates the test est organizations are increasingly challenged with needing to results against requirements and performance specifications. test larger, more complex software baselines in less time, while Large complex systems can generate gigabytes or terabytes of also being able to ensure a high-quality software product is data that are not possible for people to evaluate. They have the released. Innovative Defense Technologies (IDT) has developed an information needed, but as their systems grow, they discover they automated testing suite of tools called Automated Test and Re-Test cant evaluate everything because its too time-consuming and (ATRT) to address these testing challenges. With the use of IDTs labor intensive, said Gauf. Because ATRT AM is able to analyze ATRT suite of tools, enterprises with mission-critical systems, such against requirements and performance specifications in an autoas the U.S. Department of Defense (DoD), can increase the volume, mated fashion, you can increase test coverage and reduce the time test coverage, and complexity of integration and acceptit takes by an order of magniance tests while dramatically reducing test cycle times. tude compared to manual If you want to release software In todays environment, software teams want to get methods. faster, well focus on those to market fast at the lowest cost without comproTesting and QA teams often tests that are consistently in mising software quality, said Bernie Gauf, presionly have time to evaluate a dent and CEO of IDT. ATRT is an innovative techthe critical path. small subset of the data colnology that we have consistently seen improve Bernie Gauf lected, and as a result have the testing productivity by more than 75%. potential to miss errors. ATRT AM is capable of analyzing all the test Since 2006, IDT has been pioneering new data, pinpointing errors, and explaining how the test results differ ways of designing, developing and implementing from the way the system is expected to perform. best-in-class automated software testing solutions for the DoDs complex systems. The compaImprove Test Outcomes ny also offers automated test-strategy and design services that Automation speeds up testing, but by itself may not help organizaimprove the effectiveness of testing and QA programs. tions achieve critical-path goals. According to Gauf, technical teams often start automating processes without considering what Automate Tests with Confidence theyre automating and why theyre automating them. Because IDT offers two robust DoD-approved ATRT solutions that support theyre approaching automated testing incorrectly (e.g., not being the entire testing life cycle: ATRT Test Manager (TM) and ATRT selective about the tests theyre automating, etc.), they mistakenly Analysis Manager (AM). conclude that automation is not making the type of impact the ATRT TM is designed to support the testing of large, complex, organization expected. heterogeneous multicomputer systems. ATRT TM is non-intrusive IDT helps customers define an Automated Testing Strategy to the system under test, and provides a model-based approach to thats tailored to their unique requirements and test programs. GUI automation, which does not require the development of any With an Automated Testing Strategy in place, its possible to identest scripts. tify the critical-path tests that will yield the highest ROI based on Another thing that differentiates ATRT TM is its ability to evaluobjectives. ate all messages at the message-traffic level. Its message-based You can automate anything, but you should be automating interface testing capability spans multiple protocols, including what makes sense, said Gauf. If you want to release software SOAP over HTTP , TCP/IP , UDP and Multicast. faster, well focus on those tests that are consistently in the critical As software products get more complex and depend on interacpath for each software release, and require the most time and peotion with other software products, the interfaces between compople. That way, you can dramatically reduce your testing cycle, and nents become critical, said Gauf. You have to be able to validate you have the ability to repeat the tests in future releases. the performance of software products at the interface-specification IDT designs and builds automated tests in partnership with cuslevel to have high confidence that once systems are deployed, they tomers internal teams. IDT also builds custom turnkey automatwill work as expected. ed-testing solutions that customers can simply execute. The soluATRT TM also allows you to combine GUI-based automation tions are built using ATRT technology, so they are capable of with message-based automation in one test. In addition, ATRT TM supporting the entire testing life cycle of a system, including phasprovides a central repository for testing artifacts from test-case es such as component testing, system integration, system testing development and documentation, to requirements traceability and and test result analysis, among others. test-case results. In addition to its products and services, IDT offers on-site and remote training for customers who want to learn how to implement Accurately Analyze Massive Data Volumes ATRT effectively. The training programs range from introductory to Complex, high-reliability systems often generate more test data expert levels. than can be analyzed using manual approaches. IDTs ATRT AM For more information about IDTs ATRT products and services, automatically analyzes all test data using a model-based approach visit www.idtus.com. z
MARCH
15
TEST MANAGEMENT
Gold Sponsor
working in a globally distributed fashion. E-commerce platform provider GSI Commerce leveraged both JIRA and Zephyr Enterprise Edition to manage testing across 30 projects, 90+ releases, and 12,000 test cases spread across a team of 280 people in three global locations. The company can now easily access and reuse testing assets globally; create custom reports; and customize the workflow in one system without losing the customization in the other. Zephyr Enterprise Edition is available both as a SaaS solution and as an on-premise solution.
Zephyr for JIRA is a native add-on built exclusively for the JIRA 5 platform. It completes end-to-end project management in JIRA by adding testing to the overall planning, development, bug-tracking and reporting processes. Zephyr for JIRA is best suited to project Zephyr Enterprise Edition Improves Complex, teams that want to seamlessly If you do not have a single integrate testing into their Global Test Management workflows. Zephyr Enterprise Edition manages all repository that everyone is Zephyr for JIRA gives you aspects of software quality, and is suited for accessing, you lose a lot of sophisticated testing capabilidepartments and enterprises that run multiefficiency. ties inside JIRA, said Shah. ple projects in parallel, have structured testing Samir Shah Project teams that have stancycles, and do automated testing or a combidardized on JIRA want to incorporate testing within the exact same nation of automated and manual testing. It interface. We released it a year ago and, since the day it was offers full two-way integration to JIRA. launched, it continues to remain as either the No. 1 or No. 2 Top Project teams still behave as if they work Grossing add-on each week on the Atlassian Marketplace. in the same location, said Shah. They have Transportation and language service company ProCare used long conference calls that substitute for meetings in which they ask Zephyr for JIRA to add more structure to its testing process and for status reports; we automate these types of manual processes. to create a repeatable process. Before using Zephyr for JIRA, ProUsing Zephyr Enterprise Edition, data-driven decisions can be Care had been using JIRA, Confluence and spreadsheets. Because made while the information is still relevant. And, when anyone a lot of the work was manual, bugs would surface after releases. anywhere makes a change to a requirement, test case, assignment, Now, instead of guessing when test cycles might be completed, defect or document, the data is automatically pushed out to dashProCare can instantly view the top defects affecting tests, and use boards so stakeholders can view the current status. burndown charts to predict when test cycles will be completed Updated information is automatically displayed on your dashright within the JIRA interface. The company is benefitting from board, so you dont have to press the refresh button or ask someone more detailed analyses of bugs, easy navigation between test casfor it; the data finds you, said Shah. Traditionally, testers have had es, and the traceability that comes from linking test cases to to send their daily reports to managers who package and send the issues. reports to project managers. PMs then extract pieces of data and send it up to management. The whole process is fraught with latenGet Zephyr Community Edition Free cy and inefficiency. Instead of foregoing test-management tools because the out-ofWith Zephyr Enterprise Edition, there are no reports to run or pocket costs are too high, thousands of users have turned to Zephyr metrics to calculate. It all happens automatically. Whats more, the Community Edition, which is free for project teams of up to 10 software includes a knowledgebase that stores apps, release verusers who want the benefits of real-time updates and the ability to sions, features, functions and tests so test assets can be reused. It build a knowledgebase. The community edition can be easily conalso offers built-in APIs for tight integrations to the users preferred verted to Zephyr Enterprise Edition with a new license key. automation frameworks, providing manual and automated test We decided to offer a free version for small teams that needed the metrics in a single view. functionality, flexibility and maintenance of an enterprise-class soluPeople dont think about the knowledge that walks out the tion but were only working with an open-source budget, said Shah. door, said Shah. If you do not have a single repository that everyLearn more at www.getzephyr.com. z one is accessing, you lose a lot of efficiency, especially if youre
THE 2013 SD TIMES SOFTWARE QUALITY SOURCEBOOK MARCH
17
The Best
SharePoint Training
in the World!
Choose from more than 85 classes and tutorials that will let you put together your own custom SharePoint training experience.
Whether you need help migrating to SharePoint 2013 or are still making the most out of SharePoint 2010, you will find the SharePoint training you need at SPTechCon. Many of our speakers are Microsoft MVPs, and all are SharePoint experts. A rigorous review of the classes after each conference allows us to weed out the weaker classes and speakers. Classes at SPTechCon average almost 9 on a 10-point scale, and classes scoring below an 8 are not invited back. No wonder SPTechCon is the largest independent SharePoint training conference in the world.
If your goal is to learn something from a trusted leader in the industry and get an opportunity to network, you must go to SPTechCon.
Brian Perryman, SharePoint Developer, Health South Corporation
A BZ Media Event
@SPTechCon
www.sptechcon.com
MOBILE TEST/QA
Silver Sponsor
KEYNOTE!