Learning Elastic Stack 6.0

图书信息

作者	Pranav Shukla,Sharath Kumar M N
出版社	Packt Publishing
ISBN	9781787285866
出版时间	2017-12-22
字数	46.2万
分类	Packt Publishing,进口书,外文原版书,电脑,网络

读书简介

Deliver end-to-end real-time distributed data processing solutions by leveraging the power of Elastic Stack 6.0 About This Book ? Get to grips with the new features introduced in Elastic Stack 6.0 ? Get valuable insights from your data by working with the different components of the Elastic stack such as Elasticsearch, Logstash, Kibana, X-Pack, and Beats ? Includes handy tips and techniques to build, deploy and manage your Elastic applications efficiently on-premise or on the cloud Who This Book Is For This book is for data professionals who want to get amazing insights and business metrics from their data sources. If you want to get a fundamental understanding of the Elastic Stack for distributed, real-time processing of data, this book will help you. A fundamental knowledge of JSON would be useful, but is not mandatory. No previous experience with the Elastic Stack is required. What You Will Learn ? Familiarize yourself with the different components of the Elastic Stack ? Get to know the new functionalities introduced in Elastic Stack 6.0 ? Effectively build your data pipeline to get data from terabytes or petabytes of data into Elasticsearch and Logstash for searching and logging ? Use Kibana to visualize data and tell data stories in real-time ? Secure, monitor, and use the alerting and reporting capabilities of Elastic Stack ? Take your Elastic application to an on-premise or cloud-based production environment In Detail The Elastic Stack is a powerful combination of tools for distributed search, analytics, logging, and visualization of data from medium to massive data sets. The newly released Elastic Stack 6.0 brings new features and capabilities that empower users to find unique, actionable insights through these techniques. This book will give you a fundamental understanding of what the stack is all about, and how to use it efficiently to build powerful real-time data processing applications. After a quick overview of the newly introduced features in Elastic Stack 6.0, you’ll learn how to set up the stack by installing the tools, and see their basic configurations. Then it shows you how to use Elasticsearch for distributed searching and analytics, along with Logstash for logging, and Kibana for data visualization. It also demonstrates the creation of custom plugins using Kibana and Beats. You’ll find out about Elastic X-Pack, a useful extension for effective security and monitoring. We also provide useful tips on how to use the Elastic Cloud and deploy the Elastic Stack in production environments. On completing this book, you’ll have a solid foundational knowledge of the basic Elastic Stack functionalities. You’ll also have a good understanding of the role of each component in the stack to solve different data processing problems. Style and approach This step-by-step guide will show you the Elastic Stack, covering all the components through interactive and easy-to-follow examples. It also includes handy tips.

目录

Title Page

Copyright

Learning Elastic Stack 6.0

Credits

Disclaimer

About the Authors

About the Reviewer

www.PacktPub.com

Why subscribe?

Customer Feedback

Preface

What this book covers

What you need for this book

Who this book is for

Conventions

Reader feedback

Customer support

Downloading the example code

Downloading the color images of this book

Errata

Piracy

Questions

Introducing Elastic Stack

What is Elasticsearch, and why use it?

Schemaless and document-oriented

Searching

Analytics

Rich client library support and the REST API

Easy to operate and easy to scale

Near real time

Lightning fast

Fault tolerant

Exploring the components of Elastic Stack

Elasticsearch

Logstash

Beats

Kibana

X-Pack

Security

Monitoring

Reporting

Alerting

Graph

Elastic Cloud

Use cases of Elastic Stack

Log and security analytics

Product search

Metrics analytics

Web search and website search

Downloading and installing

Installing Elasticsearch

Installing Kibana

Summary

Getting Started with Elasticsearch

Using the Kibana Console UI

Core concepts

Index

Type

Document

Node

Cluster

Shards and replicas

Mappings and data types

Data types

Core datatypes

Complex datatypes

Other datatypes

Mappings

Creating an index with the name catalog

Defining the mappings for the type of product

Inverted index

CRUD operations

Index API

Indexing a document by providing an ID

Indexing a document without providing an ID

Get API

Update API

Delete API

Creating indexes and taking control of mapping

Creating an index

Creating type mapping in an existing index

Updating a mapping

REST API overview

Common API conventions

Formatting the JSON response

Dealing with multiple indices

Searching all documents in one index

Searching all documents in multiple indexes

Searching all documents of a particular type in all indices

Summary

Searching-What is Relevant

Basics of text analysis

Understanding Elasticsearch analyzers

Character filters

Tokenizer

Standard Tokenizer

Token filters

Using built-in analyzers

Standard Analyzer

Implementing autocomplete with a custom analyzer

Searching from structured data

Range query

Range query on numeric types

Range query with score boosting

Range query on dates

Exists query

Term query

Searching from full text

Match query

Operator

minimum_should_match

Fuzziness

Match phrase query

Multi match query

Querying multiple fields with defaults

Boosting one or more fields

With types of multi match queries

Writing compound queries

Constant score query

Bool query

Combining OR conditions

Combining conditions AND and OR conditions

Adding NOT conditions

Summary

Analytics with Elasticsearch

The basics of aggregations

Bucket aggregations

Metric aggregations

Matrix aggregations

Pipeline aggregations

Preparing data for analysis

Understanding the structure of data

Loading the data using Logstash

Metric aggregations

Sum, average, min, and max aggregations

Sum aggregation

Average aggregation

Min aggregation

Max aggregation

Stats and extended stats aggregations

Stats aggregation

Extended stats Aggregation

Cardinality aggregation

Bucket aggregations

Bucketing on string data

Terms aggregation

Bucketing on numeric data

Histogram aggregation

Range aggregation

Aggregations on filtered data

Nesting aggregations

Bucketing on custom conditions

Filter aggregation

Filters aggregation

Bucketing on date/time data

Date Histogram aggregation

Creating buckets across time

Using a different time zone

Computing other metrics within sliced time intervals

Focusing on a specific day and changing intervals

Bucketing on geo-spatial data

Geo distance aggregation

GeoHash grid aggregation

Pipeline aggregations

Calculating the cumulative sum of usage over time

Summary

Analyzing Log Data

Log analysis challenges

Logstash

Installation and configuration

Prerequisites

Downloading and installing Logstash

Installing on Windows

Installing on Linux

Running Logstash

Logstash architecture

Overview of Logstash plugins

Installing or updating plugins

Input plugins

Output plugins

Filter plugins

Codec plugins

Exploring plugins

Exploring Input plugins

File

Beats

JDBC

IMAP

Output plugins

Elasticsearch

CSV

Kafka

PagerDuty

Codec plugins

JSON

Rubydebug

Multiline

Filter plugins

Ingest node

Defining a pipeline

Ingest APIs

Put pipeline API

Get Pipeline API

Delete pipeline API

Simulate pipeline API

Summary

Building Data Pipelines with Logstash

Parsing and enriching logs using Logstash

Filter plugins

CSV filter

Mutate filter

Grok filter

Date filter

Geoip filter

Useragent filter

Introducing Beats

Beats by Elastic.co

Filebeat

Metricbeat

Packetbeat

Heartbeat

Winlogbeat

Auditbeat

Community Beats

Logstash versus Beats

Filebeat

Downloading and installing Filebeat

Installing on Windows

Installing on Linux

Architecture

Configuring Filebeat

Filebeat prospectors

Filebeat global options

Filebeat general options

Output configuration

Filebeat modules

Summary

Visualizing data with Kibana

Downloading and installing Kibana

Installing on Windows

Installing on Linux

Configuring Kibana

Data preparation

Kibana UI

User interaction

Configuring the index pattern

Discover

Elasticsearch query string

Elasticsearch DSL query

Visualize

Kibana aggregations

Bucket aggregations

Metric

Creating a visualization

Visualization types

Line, area, and bar charts

Data table

MarkDown widget

Metric

Goal

Gauge

Pie charts

Co-ordinate maps

Region maps

Tag cloud

Visualizations in action

Response codes over time

Top 10 URLs requested

Bandwidth usage of top five countries over time

Web traffic originating from different countries

Most used user agent

Dashboards

Creating a dashboard

Saving the dashboard

Cloning the dashboard

Sharing the dashboard

Timelion

Timelion UI

Timelion expressions

Using plugins

Installing plugins

Removing plugins

Summary

Elastic X-Pack

Installing X-Pack

Installing X-Pack on Elasticsearch

Installing X-Pack on Kibana

Uninstalling X-Pack

Configuring X-Pack

Security

User authentication

User authorization

Security in action

New user creation

Deleting a user

Changing the password

New role creation

How to Delete/Edit a role

Document-level security or field-level security

X-Pack security APIs

User management APIs

Role management APIs

Monitoring Elasticsearch

Monitoring UI

Elasticsearch metrics

Overview tab

Nodes tab

The Indices tab

Alerting

Anatomy of a watch

Alerting in action

Create a new alert

Threshold Alert

Advanced Watch

How to Delete/Deactivate/Edit a Watch

Summary

Running Elastic Stack in Production

Hosting Elastic Stack on a managed cloud

Getting up and running on Elastic Cloud

Using Kibana

Overriding configuration

Recovering from a snapshot

Hosting Elastic Stack on your own

Selecting hardware

Selecting an operating system

Configuring Elasticsearch nodes

JVM heap size

Disable swapping

File descriptors

Thread pools and garbage collector

Managing and monitoring Elasticsearch

Running in Docker containers

Special considerations while deploying to a cloud

Choosing instance type

Changing default ports; do not expose ports!

Proxy requests

Binding HTTP to local addresses

Installing EC2 discovery plugin

Installing S3 repository plugin

Setting up periodic snapshots

Backing up and restoring

Setting up a repository for snapshots

Shared filesystem

Cloud or distributed filesystems

Taking snapshots

Restoring a specific snapshot

Setting up index aliases

Understanding index aliases

How index aliases can help

Setting up index templates

Defining an index template

Creating indexes on the fly

Modeling time series data

Scaling the index with unpredictable volume over time

Unit of parallelism in Elasticsearch

The effect of the number of shards on the relevance score

The effect of the number of shards on the accuracy of aggregations

Changing the mapping over time

New fields get added

Existing fields get removed

Automatically deleting older documents

How index-per-timeframe solves these issues

Scaling with index-per-timeframe

Changing the mapping over time

Automatically deleting older documents

Summary

Building a Sensor Data Analytics Application

Introduction to the application

Understanding the sensor-generated data

Understanding the sensor metadata

Understanding the final stored data

Modeling data in Elasticsearch

Defining an index template

Understanding the mapping

Setting up the metadata database

Building the Logstash data pipeline

Accept JSON requests over the web

Enrich the JSON with the metadata we have in the MySQL database

The jdbc_streaming plugin

The mutate plugin

Move the looked-up fields that are under lookupResult directly in JSON

Combine the latitude and longitude fields under lookupResult as a location field

Remove the unnecessary fields

Store the resulting documents in Elasticsearch

Sending data to Logstash over HTTP

Visualizing the data in Kibana

Set up an index pattern in Kibana

Build visualizations

How does the average temperature change over time?

How does the average humidity change over time?

How do temperature and humidity change at each location over time?

Can I visualize temperature and humidity over a map?

How are the sensors distributed across departments?

Create a dashboard

Summary

Monitoring Server Infrastructure

Metricbeat

Downloading and installing Metricbeat

Installing on Windows

Installing on Linux

Architecture

Event structure

Configuring Metricbeat

Module configuration

Enabling module configs in the modules.d directory

Enabling module config in the metricbeat.yml file

General settings

Output configuration

Logging

Capturing system metrics

Running Metricbeat with the system module

Specifying aliases

Visualizing system metrics using Kibana

Deployment architecture

Summary