Experimentation In Software Engineering

Like other sciences and engineering disciplines, software engineering requires a cycle of model building, experimentation, and learning. Experiments are valuable tools for all software engineers who are involved in evaluating and choosing between different methods, techniques, languages and tools. The purpose of Experimentation in Software Engineering is to introduce students, teachers, researchers, and practitioners to empirical studies in software engineering, using controlled experiments. The introduction to experimentation is provided through a process perspective, and the focus is on the steps that we have to go through to perform an experiment. The book is divided into three parts. The first part provides a background of theories and methods used in experimentation. Part II then devotes one chapter to each of the five experiment steps: scoping, planning, execution, analysis, and result presentation. Part III completes the presentation with two examples. Assignments and statistical material are provided in appendixes. Overall the book provides indispensable information regarding empirical studies in particular for experiments, but also for case studies, systematic literature reviews, and surveys. It is a revision of the authors’ book, which was published in 2000. In addition, substantial new material, e.g. concerning systematic literature reviews and case study research, is introduced. The book is self-contained and it is suitable as a course book in undergraduate or graduate studies where the need for empirical studies in software engineering is stressed. Exercises and assignments are included to combine the more theoretical material with practical aspects. Researchers will also benefit from the book, learning more about how to conduct empirical studies, and likewise practitioners may use it as a “cookbook” when evaluating new methods or techniques before implementing them in their organization.

Basics of Software Engineering Experimentation is a practical guide to experimentation in a field which has long been underpinned by suppositions, assumptions, speculations and beliefs. It demonstrates to software engineers how Experimental Design and Analysis can be used to validate their beliefs and ideas. The book does not assume its readers have an in-depth knowledge of mathematics, specifying the conceptual essence of the techniques to use in the design and analysis of experiments and keeping the mathematical calculations clear and simple. Basics of Software Engineering Experimentation is practically oriented and is specially written for software engineers, all the examples being based on real and fictitious software engineering experiments.

Optimize the performance of your systems with practical experiments used by engineers in the world’s most competitive industries. In Experimentation for Engineers: From A/B testing to Bayesian optimization you will learn how to: Design, run, and analyze an A/B test Break the "feedback loops" caused by periodic retraining of ML models Increase experimentation rate with multi-armed bandits Tune multiple parameters experimentally with Bayesian optimization Clearly define business metrics used for decision-making Identify and avoid the common pitfalls of experimentation Experimentation for Engineers: From A/B testing to Bayesian optimization is a toolbox of techniques for evaluating new features and fine-tuning parameters. You’ll start with a deep dive into methods like A/B testing, and then graduate to advanced techniques used to measure performance in industries such as finance and social media. Learn how to evaluate the changes you make to your system and ensure that your testing doesn’t undermine revenue or other business metrics. By the time you’re done, you’ll be able to seamlessly deploy experiments in production while avoiding common pitfalls. About the technology Does my software really work? Did my changes make things better or worse? Should I trade features for performance? Experimentation is the only way to answer questions like these. This unique book reveals sophisticated experimentation practices developed and proven in the world’s most competitive industries that will help you enhance machine learning systems, software applications, and quantitative trading solutions. About the book Experimentation for Engineers: From A/B testing to Bayesian optimization delivers a toolbox of processes for optimizing software systems. You’ll start by learning the limits of A/B testing, and then graduate to advanced experimentation strategies that take advantage of machine learning and probabilistic methods. The skills you’ll master in this practical guide will help you minimize the costs of experimentation and quickly reveal which approaches and features deliver the best business results. What's inside Design, run, and analyze an A/B test Break the “feedback loops” caused by periodic retraining of ML models Increase experimentation rate with multi-armed bandits Tune multiple parameters experimentally with Bayesian optimization About the reader For ML and software engineers looking to extract the most value from their systems. Examples in Python and NumPy. About the author David Sweet has worked as a quantitative trader at GETCO and a machine learning engineer at Instagram. He teaches in the AI and Data Science master's programs at Yeshiva University. Table of Contents 1 Optimizing systems by experiment 2 A/B testing: Evaluating a modification to your system 3 Multi-armed bandits: Maximizing business metrics while experimenting 4 Response surface methodology: Optimizing continuous parameters 5 Contextual bandits: Making targeted decisions 6 Bayesian optimization: Automating experimental optimization 7 Managing business metrics 8 Practical considerations

This third edition of Design of Experiments for Engineers and Scientists adds to the tried and trusted tools that were successful in so many engineering organizations with new coverage of design of experiments (DoE) in the service sector. Case studies are updated throughout, and new ones are added on dentistry, higher education, and utilities. Although many books have been written on DoE for statisticians, this book overcomes the challenges a wider audience faces in using statistics by using easy-to-read graphical tools. Readers will find the concepts in this book both familiar and easy to understand, and users will soon be able to apply them in their work or research. This classic book is essential reading for engineers and scientists from all disciplines tackling all kinds of product and process quality problems and will be an ideal resource for students of this topic. Written in nonstatistical language, the book is an essential and accessible text for scientists and engineers who want to learn how to use DoE Explains why teaching DoE techniques in the improvement phase of Six Sigma is an important part of problem-solving methodology New edition includes two new chapters on DoE for services as well as case studies illustrating its wider application in the service industry

This book gathers chapters from some of the top international empirical software engineering researchers focusing on the practical knowledge necessary for conducting, reporting and using empirical methods in software engineering. Topics and features include guidance on how to design, conduct and report empirical studies. The volume also provides information across a range of techniques, methods and qualitative and quantitative issues to help build a toolkit applicable to the diverse software development contexts

This book identifies challenges and opportunities in the development and implementation of software that contain significant statistical content. While emphasizing the relevance of using rigorous statistical and probabilistic techniques in software engineering contexts, it presents opportunities for further research in the statistical sciences and their applications to software engineering. It is intended to motivate and attract new researchers from statistics and the mathematical sciences to attack relevant and pressing problems in the software engineering setting. It describes the "big picture," as this approach provides the context in which statistical methods must be developed. The book's survey nature is directed at the mathematical sciences audience, but software engineers should also find the statistical emphasis refreshing and stimulating. It is hoped that the book will have the effect of seeding the field of statistical software engineering by its indication of opportunities where statistical thinking can help to increase understanding, productivity, and quality of software and software production.

This book presents contemporary empirical methods in software engineering related to the plurality of research methodologies, human factors, data collection and processing, aggregation and synthesis of evidence, and impact of software engineering research. The individual chapters discuss methods that impact the current evolution of empirical software engineering and form the backbone of future research. Following an introductory chapter that outlines the background of and developments in empirical software engineering over the last 50 years and provides an overview of the subsequent contributions, the remainder of the book is divided into four parts: Study Strategies (including e.g. guidelines for surveys or design science); Data Collection, Production, and Analysis (highlighting approaches from e.g. data science, biometric measurement, and simulation-based studies); Knowledge Acquisition and Aggregation (highlighting literature research, threats to validity, and evidence aggregation); and Knowledge Transfer (discussing open science and knowledge transfer with industry). Empirical methods like experimentation have become a powerful means of advancing the field of software engineering by providing scientific evidence on software development, operation, and maintenance, but also by supporting practitioners in their decision-making and learning processes. Thus the book is equally suitable for academics aiming to expand the field and for industrial researchers and practitioners looking for novel ways to check the validity of their assumptions and experiences. Chapter 17 is available open access under a Creative Commons Attribution 4.0 International License via link.springer.com.

Improving your craft is a key skill for product and user experience professionals working in the digital era. There are many established methods of product development to inspire and focus teams—Sprint, Lean, Agile, Kanban—all of which focus on solutions to customer and business problems. Enter XDPD, or Experiment-Driven Product Development—a new approach that turns the spotlight on questions to be answered, rather than on solutions. Within XDPD, discovery is a mindset, not a project phase. In Experiment-Driven Product Development, author Paul Rissen introduces a philosophy of product development that will hone your skills in discovery, research and learning. By guiding you through a practical, immediately applicable framework, you can learn to ask, and answer, questions which will supercharge your product development, making teams smarter and better at developing products and services that deliver for users and businesses alike. When applying the XDPD framework within your organization, the concept of an experiment—a structured way of asking, and answering, questions—becomes the foundation of almost everything you do, instilling a constant sense of discovery that keeps your team inspired. All types of activities, from data analysis to writing software, are seen through the lens of research. Rather than treating research as a separate task from the rest of product development, this book approaches the entire practice as one of research and continuous discovery. Designing successful experiments takes practice. That’s where Rissen’s years of industry expertise come in. In this book, you are given step-by-step tools to ensure that meaningful, efficient progress is made with each experiment. This approach will prove beneficial to your team, your users, and most importantly, to your product’s lasting success. Experiment-Driven Product Development offers a greater appreciation of the craft of experimentation and helps you adapt it in your own context. In our modern age of innovation, XDPD can put you ahead. Go forth and experiment! What You Will LearnKnow how to approach product development in a leaner, more efficient wayUnderstand where and when experiments can be useful, and how they fit into pre-existing organization environments and processesRealize why you should be thinking about the simplest, useful thing rather than the minimum, viable productDiscover how to break down feature and design ideas into the assumptions and the premises that lie behind themAppreciate the importance of designing your experiments, and the statistical concepts that underpin their successMaster the art of communicating the results of experiments back to stakeholders, and help the results guide what happens next Who This Book is For Professionals working in digital product design and development, user experience, and service design. This book is best suited for those who work on digital products every day and want to adopt better approaches to gaining knowledge about their users, what works, and what does not work.