Hadoop: The Definitive Guide, 3rd Edition

Read it now on the O’Reilly learning platform with a 10-day free trial.

O’Reilly members get unlimited access to books, live events, courses curated by job role, and more from O’Reilly and nearly 200 top publishers.

Book description

Ready to unlock the power of your data? With this comprehensive guide, you’ll learn how to build and maintain reliable, scalable, distributed systems with Apache Hadoop. This book is ideal for programmers looking to analyze datasets of any size, and for administrators who want to set up and run Hadoop clusters.

You’ll find illuminating case studies that demonstrate how Hadoop is used to solve specific problems. This third edition covers recent changes to Hadoop, including material on the new MapReduce API, as well as MapReduce 2 and its more flexible execution model (YARN).

• Store large datasets with the Hadoop Distributed File System (HDFS)
• Run distributed computations with MapReduce
• Use Hadoop’s data and I/O building blocks for compression, data integrity, serialization (including Avro), and persistence
• Discover common pitfalls and advanced features for writing real-world MapReduce programs
• Design, build, and administer a dedicated Hadoop cluster—or run Hadoop in the cloud
• Load data from relational databases into HDFS, using Sqoop
• Perform large-scale data processing with the Pig query language
• Analyze datasets with Hive, Hadoop’s data warehousing system
• Take advantage of HBase for structured and semi-structured data, and ZooKeeper for building distributed systems

Publisher resources

Table of contents

1. Hadoop: The Definitive Guide
2. Dedication
3. Foreword
4. Preface
   1. Administrative Notes
   2. What’s in This Book?
   3. What’s New in the Second Edition?
   4. What’s New in the Third Edition?
   5. Conventions Used in This Book
   6. Using Code Examples
   7. Safari® Books Online
   8. How to Contact Us
   9. Acknowledgments
   1. Data!
   2. Data Storage and Analysis
   3. Comparison with Other Systems
      1. Rational Database Management System
      2. Grid Computing
      3. Volunteer Computing
      1. What’s Covered in This Book
         1. Configuration names
         2. MapReduce APIs
         1. A Weather Dataset
            1. Data Format
            1. Map and Reduce
            2. Java MapReduce
               1. A test run
               2. The old and the new Java MapReduce APIs
               1. Data Flow
               2. Combiner Functions
                  1. Specifying a combiner function
                  1. Ruby
                  2. Python
                  1. Compiling and Running
                  1. The Design of HDFS
                  2. HDFS Concepts
                     1. Blocks
                     2. Namenodes and Datanodes
                     3. HDFS Federation
                     4. HDFS High-Availability
                        1. Failover and fencing
                        1. Basic Filesystem Operations
                        1. Interfaces
                           1. HTTP
                           2. C
                           3. FUSE
                           1. Reading Data from a Hadoop URL
                           2. Reading Data Using the FileSystem API
                              1. FSDataInputStream
                              1. FSDataOutputStream
                              1. File metadata: FileStatus
                              2. Listing files
                              3. File patterns
                              4. PathFilter
                              1. Anatomy of a File Read
                              2. Anatomy of a File Write
                              3. Coherency Model
                                 1. Consequences for application design
                                 1. Keeping an HDFS Cluster Balanced
                                 1. Using Hadoop Archives
                                 2. Limitations
                                 1. Data Integrity
                                    1. Data Integrity in HDFS
                                    2. LocalFileSystem
                                    3. ChecksumFileSystem
                                    1. Codecs
                                       1. Compressing and decompressing streams with CompressionCodec
                                       2. Inferring CompressionCodecs using CompressionCodecFactory
                                       3. Native libraries
                                          1. CodecPool
                                          1. Compressing map output
                                          1. The Writable Interface
                                             1. WritableComparable and comparators
                                             1. Writable wrappers for Java primitives
                                             2. Text
                                                1. Indexing
                                                2. Unicode
                                                3. Iteration
                                                4. Mutability
                                                5. Resorting to String
                                                1. Implementing a RawComparator for speed
                                                2. Custom comparators
                                                1. Serialization IDL
                                                1. Avro Data Types and Schemas
                                                2. In-Memory Serialization and Deserialization
                                                   1. The specific API
                                                   1. Python API
                                                   2. C API
                                                   1. SequenceFile
                                                      1. Writing a SequenceFile
                                                      2. Reading a SequenceFile
                                                      3. Displaying a SequenceFile with the command-line interface
                                                      4. Sorting and merging SequenceFiles
                                                      5. The SequenceFile format
                                                      1. Writing a MapFile
                                                      2. Reading a MapFile
                                                      3. MapFile variants
                                                      4. Converting a SequenceFile to a MapFile
                                                      1. The Configuration API
                                                         1. Combining Resources
                                                         2. Variable Expansion
                                                         1. Managing Configuration
                                                         2. GenericOptionsParser, Tool, and ToolRunner
                                                         1. Mapper
                                                         2. Reducer
                                                         1. Running a Job in a Local Job Runner
                                                            1. Fixing the mapper
                                                            1. Packaging a Job
                                                               1. The client classpath
                                                               2. The task classpath
                                                               3. Packaging dependencies
                                                               4. Task classpath precedence
                                                               1. The jobtracker page
                                                               2. The job page
                                                               1. The tasks page
                                                               2. The task details page
                                                               3. Handling malformed data
                                                               1. Profiling Tasks
                                                                  1. The HPROF profiler
                                                                  2. Other profilers
                                                                  1. Decomposing a Problem into MapReduce Jobs
                                                                  2. JobControl
                                                                  3. Apache Oozie
                                                                     1. Defining an Oozie workflow
                                                                     2. Packaging and deploying an Oozie workflow application
                                                                     3. Running an Oozie workflow job
                                                                     1. Anatomy of a MapReduce Job Run
                                                                        1. Classic MapReduce (MapReduce 1)
                                                                           1. Job submission
                                                                           2. Job initialization
                                                                           3. Task assignment
                                                                           4. Task execution
                                                                              1. Streaming and pipes
                                                                              1. Job submission
                                                                              2. Job initialization
                                                                              3. Task assignment
                                                                              4. Task execution
                                                                              5. Progress and status updates
                                                                              6. Job completion
                                                                              1. Failures in Classic MapReduce
                                                                                 1. Task failure
                                                                                 2. Tasktracker failure
                                                                                 3. Jobtracker failure
                                                                                 1. Task failure
                                                                                 2. Application master failure
                                                                                 3. Node manager failure
                                                                                 4. Resource manager failure
                                                                                 1. The Fair Scheduler
                                                                                 2. The Capacity Scheduler
                                                                                 1. The Map Side
                                                                                 2. The Reduce Side
                                                                                 3. Configuration Tuning
                                                                                 1. The Task Execution Environment
                                                                                    1. Streaming environment variables
                                                                                    1. Task side-effect files
                                                                                    1. MapReduce Types
                                                                                       1. The Default MapReduce Job
                                                                                          1. The default Streaming job
                                                                                          2. Keys and values in Streaming
                                                                                          1. Input Splits and Records
                                                                                             1. FileInputFormat
                                                                                             2. FileInputFormat input paths
                                                                                             3. FileInputFormat input splits
                                                                                             4. Small files and CombineFileInputFormat
                                                                                             5. Preventing splitting
                                                                                             6. File information in the mapper
                                                                                             7. Processing a whole file as a record
                                                                                             1. TextInputFormat
                                                                                             2. KeyValueTextInputFormat
                                                                                             3. NLineInputFormat
                                                                                             4. XML
                                                                                             1. SequenceFileInputFormat
                                                                                             2. SequenceFileAsTextInputFormat
                                                                                             3. SequenceFileAsBinaryInputFormat
                                                                                             1. Text Output
                                                                                             2. Binary Output
                                                                                                1. SequenceFileOutputFormat
                                                                                                2. SequenceFileAsBinaryOutputFormat
                                                                                                3. MapFileOutputFormat
                                                                                                1. An example: Partitioning data
                                                                                                2. MultipleOutputs
                                                                                                1. Counters
                                                                                                   1. Built-in Counters
                                                                                                      1. Task counters
                                                                                                      2. Job counters
                                                                                                      1. Dynamic counters
                                                                                                      2. Readable counter names
                                                                                                      3. Retrieving counters
                                                                                                         1. Using the new MapReduce API
                                                                                                         1. Preparation
                                                                                                         2. Partial Sort
                                                                                                            1. An application: Partitioned MapFile lookups
                                                                                                            1. Java code
                                                                                                            2. Streaming
                                                                                                            1. Map-Side Joins
                                                                                                            2. Reduce-Side Joins
                                                                                                            1. Using the Job Configuration
                                                                                                            2. Distributed Cache
                                                                                                               1. Usage
                                                                                                               2. How it works
                                                                                                               3. The distributed cache API
                                                                                                               1. Cluster Specification
                                                                                                                  1. Network Topology
                                                                                                                     1. Rack awareness
                                                                                                                     1. Installing Java
                                                                                                                     2. Creating a Hadoop User
                                                                                                                     3. Installing Hadoop
                                                                                                                     4. Testing the Installation
                                                                                                                     1. Configuration Management
                                                                                                                        1. Control scripts
                                                                                                                        2. Master node scenarios
                                                                                                                        1. Memory
                                                                                                                        2. Java
                                                                                                                        3. System logfiles
                                                                                                                        4. SSH settings
                                                                                                                        1. HDFS
                                                                                                                        2. MapReduce
                                                                                                                        1. Cluster membership
                                                                                                                        2. Buffer size
                                                                                                                        3. HDFS block size
                                                                                                                        4. Reserved storage space
                                                                                                                        5. Trash
                                                                                                                        6. Job scheduler
                                                                                                                        7. Reduce slow start
                                                                                                                        8. Task memory limits
                                                                                                                        1. Important YARN Daemon Properties
                                                                                                                           1. Memory
                                                                                                                           1. Kerberos and Hadoop
                                                                                                                              1. An example
                                                                                                                              1. Hadoop Benchmarks
                                                                                                                                 1. Benchmarking HDFS with TestDFSIO
                                                                                                                                 2. Benchmarking MapReduce with Sort
                                                                                                                                 3. Other benchmarks
                                                                                                                                 1. Apache Whirr
                                                                                                                                    1. Setup
                                                                                                                                    2. Launching a cluster
                                                                                                                                    3. Configuration
                                                                                                                                    4. Running a proxy
                                                                                                                                    5. Running a MapReduce job
                                                                                                                                    6. Shutting down a cluster
                                                                                                                                    1. HDFS
                                                                                                                                       1. Persistent Data Structures
                                                                                                                                          1. Namenode directory structure
                                                                                                                                          2. The filesystem image and edit log
                                                                                                                                          3. Secondary namenode directory structure
                                                                                                                                          4. Datanode directory structure
                                                                                                                                          1. Entering and leaving safe mode
                                                                                                                                          1. dfsadmin
                                                                                                                                          2. Filesystem check (fsck)
                                                                                                                                             1. Finding the blocks for a file
                                                                                                                                             1. Logging
                                                                                                                                                1. Setting log levels
                                                                                                                                                2. Getting stack traces
                                                                                                                                                1. FileContext
                                                                                                                                                2. GangliaContext
                                                                                                                                                3. NullContextWithUpdateThread
                                                                                                                                                4. CompositeContext
                                                                                                                                                1. Routine Administration Procedures
                                                                                                                                                   1. Metadata backups
                                                                                                                                                   2. Data backups
                                                                                                                                                   3. Filesystem check (fsck)
                                                                                                                                                   4. Filesystem balancer
                                                                                                                                                   1. Commissioning new nodes
                                                                                                                                                   2. Decommissioning old nodes
                                                                                                                                                   1. HDFS data and metadata upgrades
                                                                                                                                                      1. Start the upgrade
                                                                                                                                                      2. Wait until the upgrade is complete
                                                                                                                                                      3. Check the upgrade
                                                                                                                                                      4. Roll back the upgrade (optional)
                                                                                                                                                      5. Finalize the upgrade (optional)
                                                                                                                                                      1. Installing and Running Pig
                                                                                                                                                         1. Execution Types
                                                                                                                                                            1. Local mode
                                                                                                                                                            2. MapReduce mode
                                                                                                                                                            1. Generating Examples
                                                                                                                                                            1. Structure
                                                                                                                                                            2. Statements
                                                                                                                                                            3. Expressions
                                                                                                                                                            4. Types
                                                                                                                                                            5. Schemas
                                                                                                                                                               1. Validation and nulls
                                                                                                                                                               2. Schema merging
                                                                                                                                                               1. A Filter UDF
                                                                                                                                                                  1. Leveraging types
                                                                                                                                                                  1. Dynamic invokers
                                                                                                                                                                  1. Using a schema
                                                                                                                                                                  1. Loading and Storing Data
                                                                                                                                                                  2. Filtering Data
                                                                                                                                                                     1. FOREACH. GENERATE
                                                                                                                                                                     2. STREAM
                                                                                                                                                                     1. JOIN
                                                                                                                                                                     2. COGROUP
                                                                                                                                                                     3. CROSS
                                                                                                                                                                     4. GROUP
                                                                                                                                                                     1. Parallelism
                                                                                                                                                                     2. Parameter Substitution
                                                                                                                                                                        1. Dynamic parameters
                                                                                                                                                                        2. Parameter substitution processing
                                                                                                                                                                        1. Installing Hive
                                                                                                                                                                           1. The Hive Shell
                                                                                                                                                                           1. Configuring Hive
                                                                                                                                                                              1. Logging
                                                                                                                                                                              1. Hive clients
                                                                                                                                                                              1. Schema on Read Versus Schema on Write
                                                                                                                                                                              2. Updates, Transactions, and Indexes
                                                                                                                                                                              1. Data Types
                                                                                                                                                                                 1. Primitive types
                                                                                                                                                                                 2. Complex types
                                                                                                                                                                                 1. Conversions
                                                                                                                                                                                 1. Managed Tables and External Tables
                                                                                                                                                                                 2. Partitions and Buckets
                                                                                                                                                                                    1. Partitions
                                                                                                                                                                                    2. Buckets
                                                                                                                                                                                    1. The default storage format: Delimited text
                                                                                                                                                                                    2. Binary storage formats: Sequence files, Avro datafiles and RCFiles
                                                                                                                                                                                    3. An example: RegexSerDe
                                                                                                                                                                                    1. Inserts
                                                                                                                                                                                    2. Multitable insert
                                                                                                                                                                                    3. CREATE TABLE. AS SELECT
                                                                                                                                                                                    1. Sorting and Aggregating
                                                                                                                                                                                    2. MapReduce Scripts
                                                                                                                                                                                    3. Joins
                                                                                                                                                                                       1. Inner joins
                                                                                                                                                                                       2. Outer joins
                                                                                                                                                                                       3. Semi joins
                                                                                                                                                                                       4. Map joins
                                                                                                                                                                                       1. Writing a UDF
                                                                                                                                                                                       2. Writing a UDAF
                                                                                                                                                                                          1. A more complex UDAF
                                                                                                                                                                                          1. HBasics
                                                                                                                                                                                             1. Backdrop
                                                                                                                                                                                             1. Whirlwind Tour of the Data Model
                                                                                                                                                                                                1. Regions
                                                                                                                                                                                                2. Locking
                                                                                                                                                                                                1. HBase in operation
                                                                                                                                                                                                1. Test Drive
                                                                                                                                                                                                1. Java
                                                                                                                                                                                                   1. MapReduce
                                                                                                                                                                                                   1. REST
                                                                                                                                                                                                   2. Thrift
                                                                                                                                                                                                   3. Avro
                                                                                                                                                                                                   1. Schemas
                                                                                                                                                                                                   2. Loading Data
                                                                                                                                                                                                      1. Optimization notes
                                                                                                                                                                                                      1. Successful Service
                                                                                                                                                                                                      2. HBase
                                                                                                                                                                                                      3. Use Case: HBase at Streamy.com
                                                                                                                                                                                                         1. Very large items tables
                                                                                                                                                                                                         2. Very large sort merges
                                                                                                                                                                                                         3. Life with HBase
                                                                                                                                                                                                         1. Versions
                                                                                                                                                                                                         2. HDFS
                                                                                                                                                                                                         3. UI
                                                                                                                                                                                                         4. Metrics
                                                                                                                                                                                                         5. Schema Design
                                                                                                                                                                                                            1. Joins
                                                                                                                                                                                                            2. Row keys
                                                                                                                                                                                                            1. Installing and Running ZooKeeper
                                                                                                                                                                                                            2. An Example
                                                                                                                                                                                                               1. Group Membership in ZooKeeper
                                                                                                                                                                                                               2. Creating the Group
                                                                                                                                                                                                               3. Joining a Group
                                                                                                                                                                                                               4. Listing Members in a Group
                                                                                                                                                                                                                  1. ZooKeeper command-line tools
                                                                                                                                                                                                                  1. Data Model
                                                                                                                                                                                                                     1. Ephemeral znodes
                                                                                                                                                                                                                     2. Sequence numbers
                                                                                                                                                                                                                     3. Watches
                                                                                                                                                                                                                     1. Multiupdate
                                                                                                                                                                                                                     2. APIs
                                                                                                                                                                                                                     3. Watch triggers
                                                                                                                                                                                                                     4. ACLs
                                                                                                                                                                                                                     1. Time
                                                                                                                                                                                                                     1. A Configuration Service
                                                                                                                                                                                                                     2. The Resilient ZooKeeper Application
                                                                                                                                                                                                                        1. InterruptedException
                                                                                                                                                                                                                        2. KeeperException
                                                                                                                                                                                                                           1. State exceptions
                                                                                                                                                                                                                           2. Recoverable exceptions
                                                                                                                                                                                                                           3. Unrecoverable exceptions
                                                                                                                                                                                                                           1. The herd effect
                                                                                                                                                                                                                           2. Recoverable exceptions
                                                                                                                                                                                                                           3. Unrecoverable exceptions
                                                                                                                                                                                                                           4. Implementation
                                                                                                                                                                                                                           1. BookKeeper and Hedwig
                                                                                                                                                                                                                           1. Resilience and Performance
                                                                                                                                                                                                                           2. Configuration
                                                                                                                                                                                                                           1. Getting Sqoop
                                                                                                                                                                                                                           2. Sqoop Connectors
                                                                                                                                                                                                                           3. A Sample Import
                                                                                                                                                                                                                              1. Text and Binary File Formats
                                                                                                                                                                                                                              1. Additional Serialization Systems
                                                                                                                                                                                                                              1. Controlling the Import
                                                                                                                                                                                                                              2. Imports and Consistency
                                                                                                                                                                                                                              3. Direct-mode Imports
                                                                                                                                                                                                                              1. Imported Data and Hive
                                                                                                                                                                                                                              1. Exports and Transactionality
                                                                                                                                                                                                                              2. Exports and SequenceFiles
                                                                                                                                                                                                                              1. Hadoop Usage at Last.fm
                                                                                                                                                                                                                                 1. Last.fm: The Social Music Revolution
                                                                                                                                                                                                                                 2. Hadoop at Last.fm
                                                                                                                                                                                                                                 3. Generating Charts with Hadoop
                                                                                                                                                                                                                                 4. The Track Statistics Program
                                                                                                                                                                                                                                    1. Calculating the number of unique listeners
                                                                                                                                                                                                                                       1. UniqueListenersMapper
                                                                                                                                                                                                                                       2. UniqueListenersReducer
                                                                                                                                                                                                                                       1. SumMapper
                                                                                                                                                                                                                                       2. SumReducer
                                                                                                                                                                                                                                       1. MergeListenersMapper
                                                                                                                                                                                                                                       2. IdentityMapper
                                                                                                                                                                                                                                       3. SumReducer
                                                                                                                                                                                                                                       1. Hadoop at Facebook
                                                                                                                                                                                                                                          1. History
                                                                                                                                                                                                                                          2. Use cases
                                                                                                                                                                                                                                          3. Data architecture
                                                                                                                                                                                                                                          4. Hadoop configuration
                                                                                                                                                                                                                                          1. Advertiser insights and performance
                                                                                                                                                                                                                                          2. Ad hoc analysis and product feedback
                                                                                                                                                                                                                                          3. Data analysis
                                                                                                                                                                                                                                          1. Data organization
                                                                                                                                                                                                                                          2. Query language
                                                                                                                                                                                                                                          3. Data pipelines using Hive
                                                                                                                                                                                                                                          1. Fair sharing
                                                                                                                                                                                                                                          2. Space management
                                                                                                                                                                                                                                          3. Scribe-HDFS integration
                                                                                                                                                                                                                                          4. Improvements to Hive
                                                                                                                                                                                                                                          1. Data Structures
                                                                                                                                                                                                                                             1. CrawlDb
                                                                                                                                                                                                                                             2. LinkDb
                                                                                                                                                                                                                                             3. Segments
                                                                                                                                                                                                                                             1. Link inversion
                                                                                                                                                                                                                                             2. Generation of fetchlists
                                                                                                                                                                                                                                                1. Step 1: Select, sort by score, limit by URL count per host
                                                                                                                                                                                                                                                2. Step 2: Invert, partition by host, sort randomly
                                                                                                                                                                                                                                                1. Requirements/The Problem
                                                                                                                                                                                                                                                   1. Logs
                                                                                                                                                                                                                                                   1. Log collection
                                                                                                                                                                                                                                                   2. Log storage
                                                                                                                                                                                                                                                   1. Processing
                                                                                                                                                                                                                                                      1. Phase 1: Map
                                                                                                                                                                                                                                                      2. Phase 1: Reduce
                                                                                                                                                                                                                                                      3. Phase 2: Map
                                                                                                                                                                                                                                                      4. Phase 2: Reduce
                                                                                                                                                                                                                                                      1. Sharding
                                                                                                                                                                                                                                                      2. Search results
                                                                                                                                                                                                                                                      1. Fields, Tuples, and Pipes
                                                                                                                                                                                                                                                      2. Operations
                                                                                                                                                                                                                                                      3. Taps, Schemes, and Flows
                                                                                                                                                                                                                                                      4. Cascading in Practice
                                                                                                                                                                                                                                                      5. Flexibility
                                                                                                                                                                                                                                                      6. Hadoop and Cascading at ShareThis
                                                                                                                                                                                                                                                      7. Summary
                                                                                                                                                                                                                                                      1. Measuring Community
                                                                                                                                                                                                                                                      2. Everybody’s Talkin’ at Me: The Twitter Reply Graph
                                                                                                                                                                                                                                                         1. Edge pairs versus adjacency list
                                                                                                                                                                                                                                                         2. Degree
                                                                                                                                                                                                                                                         1. Get neighbors
                                                                                                                                                                                                                                                         2. Community metrics and the 1 million × 1 million problem
                                                                                                                                                                                                                                                         3. Local properties at global scale
                                                                                                                                                                                                                                                         1. Prerequisites
                                                                                                                                                                                                                                                         2. Installation
                                                                                                                                                                                                                                                         3. Configuration
                                                                                                                                                                                                                                                            1. Standalone Mode
                                                                                                                                                                                                                                                            2. Pseudodistributed Mode
                                                                                                                                                                                                                                                               1. Configuring SSH
                                                                                                                                                                                                                                                               2. Formatting the HDFS filesystem
                                                                                                                                                                                                                                                               3. Starting and stopping the daemons (MapReduce 1)
                                                                                                                                                                                                                                                               4. Starting and stopping the daemons (MapReduce 2)
                                                                                                                                                                                                                                                               Show and hide more

                                                                                                                                                                                                                                                               Product information

                                                                                                                                                                                                                                                               • Title: Hadoop: The Definitive Guide, 3rd Edition
                                                                                                                                                                                                                                                               • Author(s): Tom White
                                                                                                                                                                                                                                                               • Release date: May 2012
                                                                                                                                                                                                                                                               • Publisher(s): O'Reilly Media, Inc.
                                                                                                                                                                                                                                                               • ISBN: 9781449311520