Modern Data Architectures With Python

Build scalable and reliable data ecosystems using Data Mesh, Databricks Spark, and Kafka Key Features Develop modern data skills used in emerging technologies Learn pragmatic design methodologies such as Data Mesh and data lakehouses Gain a deeper understanding of data governance Purchase of the print or Kindle book includes a free PDF eBook Book DescriptionModern Data Architectures with Python will teach you how to seamlessly incorporate your machine learning and data science work streams into your open data platforms. You’ll learn how to take your data and create open lakehouses that work with any technology using tried-and-true techniques, including the medallion architecture and Delta Lake. Starting with the fundamentals, this book will help you build pipelines on Databricks, an open data platform, using SQL and Python. You’ll gain an understanding of notebooks and applications written in Python using standard software engineering tools such as git, pre-commit, Jenkins, and Github. Next, you’ll delve into streaming and batch-based data processing using Apache Spark and Confluent Kafka. As you advance, you’ll learn how to deploy your resources using infrastructure as code and how to automate your workflows and code development. Since any data platform's ability to handle and work with AI and ML is a vital component, you’ll also explore the basics of ML and how to work with modern MLOps tooling. Finally, you’ll get hands-on experience with Apache Spark, one of the key data technologies in today’s market. By the end of this book, you’ll have amassed a wealth of practical and theoretical knowledge to build, manage, orchestrate, and architect your data ecosystems.What you will learn Understand data patterns including delta architecture Discover how to increase performance with Spark internals Find out how to design critical data diagrams Explore MLOps with tools such as AutoML and MLflow Get to grips with building data products in a data mesh Discover data governance and build confidence in your data Introduce data visualizations and dashboards into your data practice Who this book is forThis book is for developers, analytics engineers, and managers looking to further develop a data ecosystem within their organization. While they’re not prerequisites, basic knowledge of Python and prior experience with data will help you to read and follow along with the examples.

As data management and integration continue to evolve rapidly, storing all your data in one place, such as a data warehouse, is no longer scalable. In the very near future, data will need to be distributed and available for several technological solutions. With this practical book, you’ll learnhow to migrate your enterprise from a complex and tightly coupled data landscape to a more flexible architecture ready for the modern world of data consumption. Executives, data architects, analytics teams, and compliance and governance staff will learn how to build a modern scalable data landscape using the Scaled Architecture, which you can introduce incrementally without a large upfront investment. Author Piethein Strengholt provides blueprints, principles, observations, best practices, and patterns to get you up to speed. Examine data management trends, including technological developments, regulatory requirements, and privacy concerns Go deep into the Scaled Architecture and learn how the pieces fit together Explore data governance and data security, master data management, self-service data marketplaces, and the importance of metadata

Data miner’s survival kit for explainable, effective, and efficient algorithms enabling responsible decision-making KEY FEATURES ● Accessible, and case-based exploration of the most effective data mining techniques in Python. ● An indispensable guide for utilizing AI potential responsibly. ● Actionable insights on modeling techniques, deployment technologies, business needs, and the art of data science, for risk mitigation and better business outcomes. DESCRIPTION "Modern Data Mining with Python" is a guidebook for responsibly implementing data mining techniques that involve collecting, storing, and analyzing large amounts of structured and unstructured data to extract useful insights and patterns. Enter into the world of data mining and machine learning. Use insights from various data sources, from social media to credit card transactions. Master statistical tools, explore data trends, and patterns. Understand decision trees and artificial neural networks (ANNs). Manage high-dimensional data with dimensionality reduction. Explore binary classification with logistic regression. Spot concealed patterns with unsupervised learning. Analyze text with recurrent neural networks (RNNs) and visuals with convolutional neural networks (CNNs). Ensure model compliance with regulatory standards. After reading this book, readers will be equipped with the skills and knowledge necessary to use Python for data mining and analysis in an industry set-up. They will be able to analyze and implement algorithms on large structured and unstructured datasets. WHAT YOU WILL LEARN ● Explore the data mining spectrum ranging from data exploration and statistics. ● Gain hands-on experience applying modern algorithms to real-world problems in the financial industry. ● Develop an understanding of various risks associated with model usage in regulated industries. ● Gain knowledge about best practices and regulatory guidelines to mitigate model usage-related risk in key banking areas. ● Develop and deploy risk-mitigated algorithms on self-serve ModelOps platforms. WHO THIS BOOK IS FOR This book is for a wide range of early career professionals and students interested in data mining or data science with a financial services industry focus. Senior industry professionals, and educators, trying to implement data mining algorithms can benefit as well. TABLE OF CONTENTS 1. Understanding Data Mining in a Nutshell 2. Basic Statistics and Exploratory Data Analysis 3. Digging into Linear Regression 4. Exploring Logistic Regression 5. Decision Trees with Bagging and Boosting 6. Support Vector Machines and K-Nearest Neighbors 7. Putting Dimensionality Reduction into Action 8. Beginning with Unsupervised Models 9. Structured Data Classification using Artificial Neural Networks 10. Language Modeling with Recurrent Neural Networks 11. Image Processing with Convolutional Neural Networks 12. Understanding Model Risk Management for Data Mining Models 13. Adopting ModelOps to Manage Model Risk

Learn a modern approach to data analysis using Python to harness the power of programming and AI across your data. Detailed case studies bring this modern approach to life across visual data, social media, graph algorithms, and time series analysis. Key FeaturesBridge your data analysis with the power of programming, complex algorithms, and AIUse Python and its extensive libraries to power your way to new levels of data insightWork with AI algorithms, TensorFlow, graph algorithms, NLP, and financial time seriesExplore this modern approach across with key industry case studies and hands-on projectsBook Description Data Analysis with Python offers a modern approach to data analysis so that you can work with the latest and most powerful Python tools, AI techniques, and open source libraries. Industry expert David Taieb shows you how to bridge data science with the power of programming and algorithms in Python. You'll be working with complex algorithms, and cutting-edge AI in your data analysis. Learn how to analyze data with hands-on examples using Python-based tools and Jupyter Notebook. You'll find the right balance of theory and practice, with extensive code files that you can integrate right into your own data projects. Explore the power of this approach to data analysis by then working with it across key industry case studies. Four fascinating and full projects connect you to the most critical data analysis challenges you’re likely to meet in today. The first of these is an image recognition application with TensorFlow – embracing the importance today of AI in your data analysis. The second industry project analyses social media trends, exploring big data issues and AI approaches to natural language processing. The third case study is a financial portfolio analysis application that engages you with time series analysis - pivotal to many data science applications today. The fourth industry use case dives you into graph algorithms and the power of programming in modern data science. You'll wrap up with a thoughtful look at the future of data science and how it will harness the power of algorithms and artificial intelligence. What you will learnA new toolset that has been carefully crafted to meet for your data analysis challengesFull and detailed case studies of the toolset across several of today’s key industry contextsBecome super productive with a new toolset across Python and Jupyter NotebookLook into the future of data science and which directions to develop your skills nextWho this book is for This book is for developers wanting to bridge the gap between them and data scientists. Introducing PixieDust from its creator, the book is a great desk companion for the accomplished Data Scientist. Some fluency in data interpretation and visualization is assumed. It will be helpful to have some knowledge of Python, using Python libraries, and some proficiency in web development.

Provides an up-to-date analysis of big data and multi-agent systems The term Big Data refers to the cases, where data sets are too large or too complex for traditional data-processing software. With the spread of new concepts such as Edge Computing or the Internet of Things, production, processing and consumption of this data becomes more and more distributed. As a result, applications increasingly require multiple agents that can work together. A multi-agent system (MAS) is a self-organized computer system that comprises multiple intelligent agents interacting to solve problems that are beyond the capacities of individual agents. Modern Big Data Architectures examines modern concepts and architecture for Big Data processing and analytics. This unique, up-to-date volume provides joint analysis of big data and multi-agent systems, with emphasis on distributed, intelligent processing of very large data sets. Each chapter contains practical examples and detailed solutions suitable for a wide variety of applications. The author, an internationally-recognized expert in Big Data and distributed Artificial Intelligence, demonstrates how base concepts such as agent, actor, and micro-service have reached a point of convergence—enabling next generation systems to be built by incorporating the best aspects of the field. This book: Illustrates how data sets are produced and how they can be utilized in various areas of industry and science Explains how to apply common computational models and state-of-the-art architectures to process Big Data tasks Discusses current and emerging Big Data applications of Artificial Intelligence Modern Big Data Architectures: A Multi-Agent Systems Perspective is a timely and important resource for data science professionals and students involved in Big Data analytics, and machine and artificial learning.

Data fabric, data lakehouse, and data mesh have recently appeared as viable alternatives to the modern data warehouse. These new architectures have solid benefits, but they're also surrounded by a lot of hyperbole and confusion. This practical book provides a guided tour of these architectures to help data professionals understand the pros and cons of each. James Serra, big data and data warehousing solution architect at Microsoft, examines common data architecture concepts, including how data warehouses have had to evolve to work with data lake features. You'll learn what data lakehouses can help you achieve, as well as how to distinguish data mesh hype from reality. Best of all, you'll be able to determine the most appropriate data architecture for your needs. With this book, you'll: Gain a working understanding of several data architectures Learn the strengths and weaknesses of each approach Distinguish data architecture theory from reality Pick the best architecture for your use case Understand the differences between data warehouses and data lakes Learn common data architecture concepts to help you build better solutions Explore the historical evolution and characteristics of data architectures Learn essentials of running an architecture design session, team organization, and project success factors Free from product discussions, this book will serve as a timeless resource for years to come.

Learn the concepts and exercises needed to get certified as a Databricks Associate Developer for Apache Spark 3.0 and validate your skills as a Spark expert with an industry-recognized credential Key Features Understand the fundamentals of Apache Spark to help you design robust and fast Spark applications Delve into various data manipulation components for each phase of your data engineering project Prepare for the certification exam with sample questions and mock exams, and get closer to your goal Purchase of the print or Kindle book includes a free PDF eBook Book DescriptionWith extensive data being collected every second, computing power cannot keep up with this pace of rapid growth. To make use of all the data, Spark has become a de facto standard for big data processing. Migrating data processing to Spark will not only help you save resources that will allow you to focus on your business, but also enable you to modernize your workloads by leveraging the capabilities of Spark and the modern technology stack for creating new business opportunities. This book is a comprehensive guide that lets you explore the core components of Apache Spark, its architecture, and its optimization. You’ll become familiar with the Spark dataframe API and its components needed for data manipulation. Next, you’ll find out what Spark streaming is and why it’s important for modern data stacks, before learning about machine learning in Spark and its different use cases. What’s more, you’ll discover sample questions at the end of each section along with two mock exams to help you prepare for the certification exam. By the end of this book, you’ll know what to expect in the exam and how to pass it with enough understanding of Spark and its tools. You’ll also be able to apply this knowledge in a real-world setting and take your skillset to the next level.What you will learn Create and manipulate SQL queries in Spark Build complex Spark functions using Spark UDFs Architect big data apps with Spark fundamentals for optimal design Apply techniques to manipulate and optimize big data applications Build real-time or near-real-time applications using Spark Streaming Work with Apache Spark for machine learning applications Who this book is for This book is for you if you’re a professional looking to venture into the world of big data and data engineering, a data professional who wants to endorse your knowledge of Spark, or a student. Although working knowledge of Python is required, no prior Spark knowledge is needed. Additionally, experience with Pyspark will be beneficial.

Success in data science depends on the flexible and appropriate use of tools. That includes Python and R, two of the foundational programming languages in the field. This book guides data scientists from the Python and R communities along the path to becoming bilingual. By recognizing the strengths of both languages, you'll discover new ways to accomplish data science tasks and expand your skill set. Authors Rick Scavetta and Boyan Angelov explain the parallel structures of these languages and highlight where each one excels, whether it's their linguistic features or the powers of their open source ecosystems. You'll learn how to use Python and R together in real-world settings and broaden your job opportunities as a bilingual data scientist. Learn Python and R from the perspective of your current language Understand the strengths and weaknesses of each language Identify use cases where one language is better suited than the other Understand the modern open source ecosystem available for both, including packages, frameworks, and workflows Learn how to integrate R and Python in a single workflow Follow a case study that demonstrates ways to use these languages together