Bad Data Handbook

What is bad data? Some people consider it a technical phenomenon, like missing values or malformed records, but bad data includes a lot more. In this handbook, data expert Q. Ethan McCallum has gathered 19 colleagues from every corner of the data arena to reveal how they’ve recovered from nasty data problems. From cranky storage to poor representation to misguided policy, there are many paths to bad data. Bottom line? Bad data is data that gets in the way. This book explains effective ways to get around it. Among the many topics covered, you’ll discover how to: Test drive your data to see if it’s ready for analysis Work spreadsheet data into a usable form Handle encoding problems that lurk in text data Develop a successful web-scraping effort Use NLP tools to reveal the real sentiment of online reviews Address cloud computing issues that can impact your analysis effort Avoid policies that create data analysis roadblocks Take a systematic approach to data quality analysis

"Mapping the world of data problems"--Cover.

Not all statistics are created equal. Take a look behind the scenes and you'll discover that even most official data isn't the solid bedrock we think it is. It's patchy, inconsistent, full of guesswork and uncertainty - and it's playing an ever-bigger role in policy decisions. BAD DATA takes the reader on that behind-the-scenes journey, guided by House of Commons Library statistician Georgina Sturge. Revealing the secrets of a world that is usually closed off, it will show how governments of the past and present have been led astray by bad data and explain why it is so hard to count and measure things, and how we could better handle these problems. Discover how one Hungarian businessman's bright idea caused half a million people to go missing from UK migration statistics. Find out why it's possible for two politicians to disagree over whether poverty has gone up or down, using the same official numbers, and for both to be right at the same time. And hear about how policies like ID cards, super-casinos and stopping ex-convicts from reoffending failed to live up to their promise because they were based on shaky data.

This book presents a perspective of network analysis as a tool to find and quantify significant structures in the interaction patterns between different types of entities. Moreover, network analysis provides the basic means to relate these structures to properties of the entities. It has proven itself to be useful for the analysis of biological and social networks, but also for networks describing complex systems in economy, psychology, geography, and various other fields. Today, network analysis packages in the open-source platform R and other open-source software projects enable scientists from all fields to quickly apply network analytic methods to their data sets. Altogether, these applications offer such a wealth of network analytic methods that it can be overwhelming for someone just entering this field. This book provides a road map through this jungle of network analytic methods, offers advice on how to pick the best method for a given network analytic project, and how to avoid common pitfalls. It introduces the methods which are most often used to analyze complex networks, e.g., different global network measures, types of random graph models, centrality indices, and networks motifs. In addition to introducing these methods, the central focus is on network analysis literacy – the competence to decide when to use which of these methods for which type of question. Furthermore, the book intends to increase the reader's competence to read original literature on network analysis by providing a glossary and intensive translation of formal notation and mathematical symbols in everyday speech. Different aspects of network analysis literacy – understanding formal definitions, programming tasks, or the analysis of structural measures and their interpretation – are deepened in various exercises with provided solutions. This text is an excellent, if not the best starting point for all scientists who want to harness the power of network analysis for their field of expertise.

This highly practical handbook teaches you how to unlock the value of your existing metadata through cleaning, reconciliation, enrichment and linking and how to streamline the process of new metadata creation. Libraries, archives and museums are facing up to the challenge of providing access to fast growing collections whilst managing cuts to budgets. Key to this is the creation, linking and publishing of good quality metadata as Linked Data that will allow their collections to be discovered, accessed and disseminated in a sustainable manner. This highly practical handbook teaches you how to unlock the value of your existing metadata through cleaning, reconciliation, enrichment and linking and how to streamline the process of new metadata creation. Metadata experts Seth van Hooland and Ruben Verborgh introduce the key concepts of metadata standards and Linked Data and how they can be practically applied to existing metadata, giving readers the tools and understanding to achieve maximum results with limited resources. Readers will learn how to critically assess and use (semi-)automated methods of managing metadata through hands-on exercises within the book and on the accompanying website. Each chapter is built around a case study from institutions around the world, demonstrating how freely available tools are being successfully used in different metadata contexts. This handbook delivers the necessary conceptual and practical understanding to empower practitioners to make the right decisions when making their organisations resources accessible on the Web. Key topics include: - The value of metadata Metadata creation – architecture, data models and standards - Metadata cleaning - Metadata reconciliation - Metadata enrichment through Linked Data and named-entity recognition - Importing and exporting metadata - Ensuring a sustainable publishing model. Readership: This will be an invaluable guide for metadata practitioners and researchers within all cultural heritage contexts, from library cataloguers and archivists to museum curatorial staff. It will also be of interest to students and academics within information science and digital humanities fields. IT managers with responsibility for information systems, as well as strategy heads and budget holders, at cultural heritage organisations, will find this a valuable decision-making aid.

You're sitting on a pile of interesting data. How do you transform that into money? It's easy to focus on the contents of the data itself, and to succumb to the (rather unimaginative) idea of simply collecting and reselling it in raw form. While that's certainly profitable right now, you'd do well to explore other opportunities if you expect to be in the data business long-term. In this paper, we'll share a framework we developed around monetizing data. We'll show you how to think beyond pure collection and storage, to move up the value chain and consider longer-term opportunities.