Learning.Spark.Lightning-Fast.Big.Data.Analysis.pdf

【实例简介】
Learning Spark， pdf格式, 为数不多的spark著作，值得细看
Learning spark by holden Karau andy Konwins ki, Patrick Wendell, and matei zaharia Copyright 2015 Databricks. All rights reserved Printed in the united states of america Published by o reilly media, Inc, 1005 Gravenstein Highway north, Sebastopol, ca 95472. O'Reilly books may be purchased for educational, business or sales promotional use Onlineeditionsarealsoavailableformosttitles(http://safaribooksonline.com).Formore information, contact our corporate/institutional sales department: 800-998-9938 or corporate@oreilly.com Editors: Ann Spencer and marie beaugureau Production editor: Kara ebrahim Copyeditor: Rachel Monaghan Proofreader: charles roumeliotis Indexer: ellen troutman Interior Designer: David Futato Cover Designer: Ellie Volckhausen Illustrator: Rebecca demarest February 2015: First Edition Revision history for the First Edition 2015-01-26: First Release Seehttp:oreilly.com/catalog/errata.csp?isbn=9781449358624forreleasedetails The O' Reilly logo is a registered trademark of o'Reilly Media, InC. Learning Spark, the cover image of a small-spotted catshark, and related trade dress are trademarks of O Reilly media, Inc While the publisher and the authors have used good faith efforts to ensure that the information and instructions contained in this work are accurate the publisher and the authors disclaim all responsibility for errors or omissions, including without limitation responsibility for damages resulting from the use of or reliance on this work. Use of the nformation and instructions contained in this work is at your own risk. If any code samples or other technology this work contains or describes is subject to open source licenses or the intellectual property rights of others, it is your responsibility to ensure that your use thereof complies with such licenses and/or rights 978-1-449-35862-4 LSI Foreword In a very short time, Apache Spark has emerged as the next generation big data processing engine, and is being applied throughout the industry faster than ever. Spark improves over Hadoop MapReduce, which helped ignite the big data revolution, in several key dimensions: it is much faster, much easier to use due to its rich APIs, and it goes far beyond batch applications to support a variety of workloads, including interactive queries, streaming, machine learning, and graph processing I have been privileged to be closely involved with the development of Spark all the way trom the drawing board to what has become the most active big data open source project today, and one of the most active Apache projects! As such, I,'m particularly delighted to see Matei Zaharia, the creator of Spark, teaming up with other longtime Spark developers Patrick Wendell, Andy Konwinski, and Holden Karau to write this book With Spark's rapid rise in popularity, a major concern has been lack of good reference material. This book goes a long way to address this concern, with 11 chapters and dozens of detailed examples designed for data scientists, students, and developers looking to learn Spark. It is written to be approachable by readers with no background in big data, making it a great place to start learning about the field in general. I hope that many years from now, you and other readers will fondly remember this as the book that introduced you to this exciting new field Ion stoica, ceo of Databricks and Co-director, AMPlab, UC Berkeley Preface As parallel data analysis has grown common, practitioners in many fields have sought easier tools for this task. Apache Spark has quickly emerged as one of the most popular, extending and generalizing MapReduce. Spark offers three main benefits. First, it is easy to use- you can develop applications on your laptop, using a high-level api that lets you focus on the content of your computation. Second, Spark is fast, enabling interactive use and complex algorithms. And third, Spark is a general engine, letting you combine multiple types of computations(e. g SQL queries, text processing, and machine learning) that might previously have required different engines. These features make Spark an excellent starting point to learn about Big data in general This introductory book is meant to get you up and running with Spark quickly. You'll learn how to download and run Spark on your laptop and use it interactively to learn the API. Once there we'l cover the details of available operations and distributed execution Finally, you' ll get a tour of the higher-level libraries built into Spark, including libraries for machine learning, stream processing, and SQL. We hope that this book gives you the tools to quickly tackle data analysis problems, whether you do so on one machine or hundreds Audience This book targets data scientists and engineers. We chose these two groups because they have the most to gain from using Spark to expand the scope of problems they can solve Spark's rich collection of data-focused libraries (like MLlib) makes it easy for data scientists to go beyond problems that fit on a single machine while using their statistical background. Engineers, meanwhile, will learn how to write general-purpose distributed programs in Spark and operate production applications. Engineers and data scientists will both learn different details from this book, but will both be able to apply Spark to solve large distributed problems in their respective fields Data scientists focus on answering questions or building models from data. They often have a statistical or math background and some familiarity with tools like Python, R, and SQL. We have made sure to include Python and, where relevant, SQL examples for all our material, as well as an overview of the machine learning and library in Spark. If you are a data scientist, we hope that after reading this book you will be able to use the same mathematical approaches to solve problems, except much faster and on a much larger scale The second group this book targets is software engineers who have some experience with Java, Python, or another programming language. If you are an engineer, we hope that this book will show you how to set up a Spark cluster, use the Spark shell, and write Spark applications to solve parallel processing problems. If you are familiar with Hadoop, you have a bit of a head start on figuring out how to interact with hdFS and how to manage a cluster, but either way, we will cover basic distributed execution concepts Regardless of whether you are a data scientist or engineer, to get the most out of this book you should have some familiarity with one of Python, Java, Scala, or a similar language We assume that you already have a storage solution for your data and we cover how to load and save data from many common ones, but not how to set them up. If you don't have experience with one of those languages, don' t worry: there are excellent resources available to learn these we call out some of the books available in "Supporting books? How This Book Is Organized The chapters of this book are laid out in such a way that you should be able to go through the material front to back. At the start of each chapter, we will mention which sections we think are most relevant to data scientists and which sections we think are most relevant for engineers. That said, we hope that all the material is accessible to readers of either background The first two chapters will get you started with getting a basic S park installation on your laptop and give you an idea of what you can accomplish with Spark. Once weve got the motivation and setup out of the way, we will dive into the Spark shell, a very useful tool for development and prototyping. Subsequent chapters then cover the Spark programming interface in detail, how applications execute on a cluster, and higher-level libraries available on Spark( such as Spark SQL and MLlib) Supporting Books If you are a data scientist and don' t have much experience with Python, the books Learning Python and Head First Python(both O'Reilly) are excellent introductions. If you have some Python experience and want more, Dive into Python(Apress)is a great book to help you get a deeper understanding of python If you are an engineer and after reading this book you would like to expand your data analysis skills, Machine learning for hackers and doing data Science are excellent books(both O'Reilly) This book is intended to be accessible to beginners. We do intend to release a deep -dive follow-up for those looking to gain a more thorough understanding of spark's internals Conventions used in This book The following typographical conventions are used in this book Italic Indicates new terms URLs. email addresses filenames, and file extensions Constant width Used for program listings, as well as within paragraphs to refer to program elements such as variable or function names, databases, data types, environment variables statements, and keywords Constant width bold Shows commands or other text that should be typed literally by the user Constant width italic Shows text that should be replaced with user-supplied values or by values determined by context P This element signifies a tip or suggestion Warning This element indicates a warning or caution 【实例截图】
【核心代码】