Module 02 · Introduction to the Hadoop Ecosystem
Module 2 Glossary: Introduction to the Hadoop Ecosystem模块 2 词汇表：Hadoop 生态系统简介
Introduction to Big Data with Spark and Hadoop 大数据的 Spark 与 Hadoop 入门
Welcome! This alphabetized glossary contains many of the terms in this course. This comprehensive glossary also includes additional industry-recognized terms not used in course videos. These terms are essential to recognize when working in the industry, participating in user groups, and other professional certificate programs.
欢迎！这个按字母顺序排列的词汇表包含了本课程中的许多术语。这个全面的词汇表还包括了课程视频中未使用的行业公认术语。在行业工作、参加用户组以及其他专业证书项目时，认识这些术语至关重要。
Estimated reading time: 12 minutes
预计阅读时间：12 分钟
| Term 术语 | Definition 定义 |
|---|---|
| Anomaly detection 异常检测 | A process in machine learning that identifies data points, events, and observations that deviate from a data set's normal behavior. Detecting anomalies from time series data is a pain point that is critical to address for industrial applications. 机器学习中的一种过程，用于识别数据点、事件和观察结果，这些与数据集的正常行为存在偏差。从时间序列数据中检测异常是一个对工业应用至关重要的关键问题。 |
| Apache | This open-source HTTP server implements current HTTP standards to be highly secure, easily configurable, and highly extendible. The Apache Software License by the Apache Software Foundation builds and distributes it. 这是一个开源的 HTTP 服务器，实现了当前的 HTTP 标准，具有高度安全性、易于配置和高度可扩展性。它由 Apache 软件基金会使用 Apache 软件许可证进行构建和分发。 |
| Apache Cassandra | It is a scalable, NoSQL database specifically designed not to have a single point of failure. 它是一个可扩展的 NoSQL 数据库，专门设计为没有单点故障。 |
| Apache Nutch | An extensible and scalable web crawler software product to aggregate data from the web. 一个可扩展和可伸缩的网页爬虫软件产品，用于从网络上聚合数据。 |
| Apache ZooKeeper | A centralized service for maintaining configuration information to maintain healthy links between nodes. It provides synchronization across distributed applications. It also tracks server failure and network partitions by triggering an error message and then repairing the failed nodes. 一个集中式服务，用于维护配置信息，以保持节点之间的健康连接。它提供跨分布式应用的同步。它还通过触发错误消息来跟踪服务器故障和网络分区，然后修复故障节点。 |
| Big data 大数据 | Data sets whose type or size supersedes the ability of traditional relational databases to manage, capture, and process the data with low latency. Big data characteristics include high volume, velocity, and variety. 数据集的类型或规模超过了传统关系型数据库管理、捕获和处理数据的能力，且具有低延迟。大数据的特征包括高容量、高速度和高多样性。 |
| Big data analytics 大数据分析 | Uses advanced analytic techniques against large, diverse big data sets, including structured, semi-structured, and unstructured data, from varied sources and sizes, from terabytes to zettabytes. 对大型、多样化的海量数据集使用高级分析技术，包括结构化、半结构化和非结构化数据，这些数据来自不同的来源和规模，从 TB 级到 ZB 级。 |
| Block 区块 | Minimum amount of data written or read, and also offers fault tolerance. The default block size can be 64 or 128 MB, depending on the user's system configuration. Each file stored need not take up the storage of the preconfigured block size. 写入或读取的最小数据量，并提供容错功能。默认块大小可以是 64 或 128 MB，具体取决于用户的系统配置。存储的每个文件不必占用预配置的块大小。 |
| Clusters 集群 | These servers are managed and participate in workload management. They allow enterprise applications to supersede the throughput achieved with a single application server. 这些服务器被管理和参与工作负载管理。它们允许企业应用程序超越单个应用程序服务器所达到的吞吐量。 |
| Command-line interface (CLI) 命令行界面（CLI） |
Used to enter commands that enable users to manage the system. 用于输入命令，使用户能够管理系统。 |
| Commodity hardware 商用硬件 | Consists of low-cost workstations or desktop computers that are IBM-compatible and run multiple operating systems such as Microsoft Windows, Linux, and DOS without additional adaptations or software. 由低成本的 IBM 兼容工作站或台式计算机组成，可运行 Microsoft Windows、Linux 和 DOS 等多种操作系统，无需额外的适配或软件。 |
| Data ingestion 数据摄取 | The first stage of big data processing. It is a process of importing and loading data into IBM® WatsonX.data. You can use the Ingestion jobs tab from the Data manager page to load data securely and easily into WatsonX.data console. 大数据处理的第一阶段。它是一个将数据导入并加载到 IBM® WatsonX.data 的过程。您可以使用数据管理页面上的摄取作业选项卡，安全且轻松地将数据加载到 WatsonX.data 控制台。 |
| Data sets 数据集 | Created by extracting data from packages or data modules. They gather a customized collection of items that you use frequently. As users update their data set, dashboards and stories are also updated. 从软件包或数据模块中提取数据创建而成。它们收集您经常使用的定制项目集合。当用户更新其数据集时，仪表板和故事也会更新。 |
| Data warehouse 数据仓库 | Stores historical data from many different sources so users can analyze and extract insights from it. 存储来自多个不同来源的历史数据，以便用户可以分析并从中提取见解。 |
| Distributed computing 分布式计算 | A system or machine with multiple components on different machines. Each component has its own job, but the components communicate with each other to run as one system for the end user. 一个由分布在不同机器上的多个组件组成的系统或机器。每个组件都有其自身的任务，但组件之间相互通信，以作为一个系统运行，为最终用户提供服务。 |
| Driver 驱动器 | Receives query statements submitted through the command line and sends the query to the compiler after initiating a session. 接收通过命令行提交的查询语句，并在启动会话后将查询发送给编译器。 |
| Executor 执行者 | Executes tasks after the optimizer has split the tasks. 在优化器分割任务后执行任务。 |
| Extended Hadoop Ecosystem 扩展的 Hadoop 生态系统 |
Consists of libraries or software packages commonly used with or installed on top of the Hadoop core. 由通常与 Hadoop 核心一起使用或安装在 Hadoop 核心之上的库或软件包组成。 |
| Fault tolerance 容错性 | A system is fault-tolerant if it can continue performing despite parts failing. Fault tolerance helps to make your remote-boot infrastructure more robust. In the case of OS deployment servers, the whole system is fault-tolerant if the servers back up each other. 一个系统具有容错能力，如果它能够在部分组件失效的情况下继续运行。容错能力有助于使您的远程引导基础设施更加健壮。在操作系统部署服务器的案例中，如果服务器之间相互备份，则整个系统具有容错能力。 |
| File system 文件系统 | An all-comprehensive directory structure with a root ( / ) directory and other directories and files under a logical volume. The complete information about the file system centralized in the /etc/filesystems file. 一个全面的目录结构，包含根（/）目录以及逻辑卷下方的其他目录和文件。关于文件系统的完整信息集中存储在/etc/filesystems 文件中。 |
| Flume | A distributed service that collects, aggregates, and transfers big data to the storage system. Offers a simple yet flexible architecture that streams data flows and uses an extensible data model, allowing online analytic applications. 一个分布式服务，用于收集、聚合和将大数据传输到存储系统。提供简单而灵活的架构，用于流式传输数据流，并使用可扩展的数据模型，支持在线分析应用。 |
| Hadoop | An open-source software framework offering reliable distributed processing of large data sets using simplified programming models. 一个开源软件框架，提供使用简化的编程模型对大型数据集进行可靠分布式处理。 |
| Hadoop Common | Fundamental part of the Apache Hadoop framework. It refers to a collection of primary utilities and libraries that support other Hadoop modules. Apache Hadoop 框架的基础部分。它指的是支持其他 Hadoop 模块的一组主要实用程序和库。 |
| Hadoop Distributed File System (HDFS) Hadoop 分布式文件系统（HDFS） |
A file system distributed on multiple file servers, allowing programmers to access or store files from any network or computer. It is the storage layer of Hadoop. It works by splitting the files into blocks, creating replicas of the blocks, and storing them on different machines. It can access streaming data seamlessly. It uses a command-line interface to interact with Hadoop. 一个分布在多个文件服务器上的文件系统，允许程序员从任何网络或计算机访问或存储文件。它是 Hadoop 的存储层。它通过将文件分割成块、创建块的副本并将它们存储在不同的机器上工作。它可以无缝访问流数据。它使用命令行界面与 Hadoop 交互。 |
| Hadoop Ecosystem Hadoop 生态系统 | It splits big data analytics processing tasks into smaller tasks. The small tasks are performed in conjunction using an algorithm (MapReduce) and then distributed across a Hadoop cluster (nodes that perform parallel computations on big data sets). 它将大数据分析处理任务分解成更小的任务。这些小任务使用算法（MapReduce）协同执行，然后分布到 Hadoop 集群（在大型数据集上进行并行计算的节点）中。 |
| Hadoop Ecosystem stages Hadoop 生态系统阶段 | The four main stages are: Ingest, store, process, analyze, and access. 主要有四个阶段：摄取、存储、处理、分析和访问。 |
| HBase | A column-oriented, non-relational database system that runs on top of the Hadoop Distributed File System (HDFS). It provides real-time wrangling access to the Hadoop file system. It uses hash tables to store data in indexes and allow for random data access, making lookups faster. 一种列式、非关系型数据库系统，运行在 Hadoop 分布式文件系统（HDFS）之上。它提供对 Hadoop 文件系统的实时数据处理访问。它使用哈希表在索引中存储数据，并允许随机数据访问，从而加快查找速度。 |
| High-throughput 高通量 | Throughput quantifies the data processed in a timeframe. The target system needs robust throughput for heavy workloads with substantial data changes from the source database to prevent latency spikes. Performance objectives are frequently outlined with throughput targets. High throughput is achieved when most messages are delivered successfully, whereas low successful delivery rates indicate poor throughput and network performance. 高通量量化在特定时间段内处理的数据量。目标系统需要具备强大的高通量，以应对大量数据变化的工作负载，从而防止延迟峰值。性能目标通常以高通量指标来设定。当大多数消息成功传递时，即可实现高通量；而低成功率则表明高通量差，网络性能不佳。 |
| Hive | It is a data warehouse infrastructure used in data query and analysis with an SQL-like interface. It helps in generating and creating reports. It is a declarative programming language allowing users to express which data they wish to receive. 它是一种用于数据查询和分析的数据仓库基础设施，具有类似 SQL 的接口。它有助于生成和创建报告。它是一种声明式编程语言，允许用户表达他们希望接收的数据。 |
| Hive client Hive 客户端 | Hive provides different communication drivers depending on the application type. For example, Java-based applications use JDBC drivers, and other applications use ODBC drivers. These drivers communicate with the servers. Hive 根据应用类型提供不同的通信驱动程序。例如，基于 Java 的应用程序使用 JDBC 驱动程序，而其他应用程序使用 ODBC 驱动程序。这些驱动程序与服务器进行通信。 |
| Hive server Hive 服务器 | Used to execute queries and enable multiple clients to submit requests. It can support JDBC and ODBC clients. 用于执行查询并允许多个客户端提交请求。它可以支持 JDBC 和 ODBC 客户端。 |
| Hive services Hive 服务 | Client interactions and query operations are done through the Hive services. The command-line interface acts as an interface for the Hive service. The driver takes in query statements, monitors each session's progress and life cycle, and stores metadata generated from the query statements. 客户端交互和查询操作通过 Hive 服务进行。命令行界面作为 Hive 服务接口。驱动程序接收查询语句，监控每个会话的进度和生命周期，并存储由查询语句生成的元数据。 |
| Hive Web Interface Hive Web 界面 | A web-based user interface that interacts with Hive through a web browser. It offers a graphical user interface (GUI) to browse tables, execute Hive queries, and manage Hive resources. 一种基于网络的用户界面，通过网页浏览器与 Hive 交互。它提供图形用户界面（GUI）来浏览表、执行 Hive 查询和管理 Hive 资源。 |
| HMaster | The master server that monitors the region server instances. It assigns regions to region servers and distributes services to different region servers. It also manages any changes to the schema and metadata operations. 负责监控区域服务器实例的主服务器。它将区域分配给区域服务器，并将服务分配到不同的区域服务器。它还管理模式更改和元数据操作。 |
| Hue | An acronym for Hadoop user experience. It allows you to upload, browse, and query data. Users can run Pig jobs and workflow in Hue. It also provides an SQL editor for several query languages, like Hive and MySQL. Hadoop 用户体验的缩写。它允许您上传、浏览和查询数据。用户可以在 Hue 中运行 Pig 作业和工作流。它还提供用于 Hive 和 MySQL 等多种查询语言的 SQL 编辑器。 |
| Impala | A scalable system that allows nontechnical users to search for and access the data in Hadoop. 一个可扩展的系统，允许非技术人员搜索和访问 Hadoop 中的数据。 |
| InputSplits 数据分割 | Created by the logical division of data. They serve as an input to a single Mapper job. 由数据的逻辑划分创建。它们作为单个 Mapper 作业的输入。 |
| JDBC client JDBC 客户端 | Component in the Hive client allows Java-based applications to connect to Hive. Hive 客户端中的组件允许基于 Java 的应用程序连接到 Hive。 |
| Low latency data access 低延迟数据访问 |
A type of data access allowing minimal delays, not noticeable to humans, between an input processed and corresponding output offering real-time characteristics. It is crucial for internet connections using trading, online gaming, and Voice over IP. 一种允许输入处理与相应输出之间延迟最小，对人类而言不可察觉的数据访问类型，具有实时特性。这对于使用交易、在线游戏和 VoIP 的互联网连接至关重要。 |
| Map 映射 | Job in MapReduce converts a set of data into another set of data. The elements fragment into tuples (key/value pairs). MapReduce 中的任务将一组数据转换为另一组数据。元素被分割成元组（键/值对）。 |
| MapReduce | A program model and processing technique used in distributed computing based on Java. It splits the data into smaller units and processes big data. It is the first method used to query data stored in HDFS. It allows massive scalability across hundreds or thousands of servers in a Hadoop cluster. 一种基于 Java 的分布式计算中使用的程序模型和处理技术。它将数据分割成更小的单元，并处理大数据。它是查询存储在 HDFS 中数据的第一种方法。它允许在 Hadoop 集群中的数百或数千台服务器上实现大规模扩展。 |
| Meta store 元存储 | Stores the metadata, the data, and information about each table, such as the location and schema. In turn, the meta store, file system, and job client communicate with Hive storage and computing to perform the following: Metadata information from tables store in some databases and query results, and data loaded from the tables store in a Hadoop cluster on HDFS. 存储元数据、数据和每个表的信息，例如位置和模式。反过来，元存储、文件系统和作业客户端与 Hive 存储和计算通信以执行以下操作：存储在数据库中的表元数据信息以及查询结果，以及从表中加载的数据存储在 Hadoop 集群上的 HDFS。 |
| Node 节点 | A single independent system for storing and processing big data. HDFS follows the primary/secondary concept. 一个用于存储和处理大数据的独立系统。HDFS 遵循主/次概念。 |
| ODBC (Open Database Connectivity) Client ODBC（开放数据库连接）客户端 |
Component in the Hive client, which allows applications based on the ODBC protocol to connect to Hive. Hive 客户端中的组件，允许基于 ODBC 协议的应用程序连接到 Hive。 |
| Optimizer 优化器 | Performs transformations on the execution and splits the tasks to help speed up and improve efficiency. 对执行进行转换，并将任务拆分以帮助加快速度和提高效率。 |
| Parallel computing 并行计算 | Workload for each job is distributed across several processors on one or more computers, called compute nodes. 每个作业的工作负载分布在多个计算机上（一个或多个）的多个处理器上，这些计算机称为计算节点。 |
| Parser 解析器 | A program that interprets the physical bit stream of an incoming message and creates an internal logical representation of the message in a tree structure. The parser also regenerates a bit stream from the internal message tree representation for an outgoing message. 一个解释传入消息的物理比特流并创建消息内部逻辑表示的程序，该表示以树形结构呈现。解析器还可以根据内部消息树形表示为传出消息重新生成比特流。 |
| Partitioning 分区 | This implies dividing the table into parts depending on the values of a specific column, such as date or city. 这意味着根据特定列的值（如日期或城市）将表格划分为不同的部分。 |
| Pig Hadoop component 猪 Hadoop 组件 | Famous for its multi-query approach, it analyzes large amounts of data. It is a procedural data flow and programming language that follows an order and set of commands. 以其多查询方法而闻名，它能够分析大量数据。它是一种过程式数据流和编程语言，遵循特定的顺序和命令集。 |
| Primary node 主节点 | Also known as the name node, it regulates client file access and maintains, manages, and assigns tasks to the secondary node. The architecture is such that per cluster, there is one name node and multiple data nodes, the secondary nodes. 也称为名称节点，它管理客户端文件访问，并维护、管理和分配任务给辅助节点。该架构规定每个集群有一个名称节点和多个数据节点，即辅助节点。 |
| Rack 机架 | The collection of about forty to fifty data nodes using the same network switch. 使用同一个网络交换机连接的约四十到五十个数据节点的集合。 |
| Rack awareness 机架感知 | When performing operations such as read and write, the name node maximizes performance by choosing the data nodes closest to themselves. Developers can select data nodes on the same rack or nearby racks. It reduces network traffic and improve cluster performance. The name node keeps the rack ID information to achieve rack awareness. 在进行读和写等操作时，名称节点通过选择离自己最近的数据节点来最大化性能。开发者可以选择同一机架或附近机架上的数据节点。这减少了网络流量并提高了集群性能。名称节点保留机架 ID 信息以实现机架感知。 |
| Read 读 | In this operation, the client will request the primary node to acquire the location of the data nodes containing blocks. The client will read files closest to the data nodes. 在此操作中，客户端将请求主节点获取包含数据块的数据节点的位置。客户端将读取离数据节点最近的数据。 |
| Reduce 减少 | Job in MapReduce that uses output from a map as an input and combines data tuples into small sets of tuples. MapReduce 中的一个作业，它使用 map 的输出作为输入，并将数据元组组合成小的元组集。 |
| Region 区域 | The basic building element and most negligible unit of the HBase cluster, consisting of column families. It contains multiple stores, one for each column family, and has two components: HFile and MemStore. HBase 集群的基本构建单元和最微不足道的单元，由列族组成。它包含多个存储，每个列族一个，具有两个组件：HFile 和 MemStore。 |
| Region servers 区域服务器 | These servers receive read and write requests from the client. They assign the request to a region where the column family resides. They serve and manage regions present in a distributed cluster. The region servers can communicate directly with the client to facilitate requests. 这些服务器接收来自客户端的读写请求。它们将请求分配到包含列族的区域。它们为分布式集群中的区域提供服务和管理。区域服务器可以直接与客户端通信以促进请求。 |
| Relational database 关系型数据库 | Data is organized into rows and columns collectively, forming a table. The data is structured across tables, joined by a primary or a foreign key. 数据按行和列组织，形成表。数据在表之间结构化，通过主键或外键进行连接。 |
| Relational Database Management System (RDBMS) 关系数据库管理系统（RDBMS） |
Traditional RDBMS maintains a database and uses the structured query language, SQL. It is suited for real-time data analysis, like data from sensors. It allows for as many read-and-write operations as a user may require. It can handle up to terabytes of data. It enforces that the schema must verify loading data before it can proceed. It may not always have built-in support for data partitioning. 传统 RDBMS 维护数据库，并使用结构化查询语言 SQL。它适用于实时数据分析，例如来自传感器的数据。它允许用户进行任意数量的读写操作。它可以处理高达 TB 级别的数据。它强制要求模式在加载数据之前进行验证。它可能没有内置的数据分区支持。 |
| Replication 复制 | The process of creating a copy of the data block. It is performed by rack awareness as well. It is done by ensuring data node replicas are in different racks. So, if a rack is down, users can obtain the data from another rack. 创建数据块副本的过程。它也由机架感知来执行。它是通过确保数据节点副本位于不同的机架上完成的。因此，如果一个机架宕机，用户可以从另一个机架获取数据。 |
| Replication factor 复制因子 | Defined as the number of times you make a copy of the data block. Users can set the number of copies they want, depending on their configuration. 定义为数据块副本的次数。用户可以根据其配置设置所需的副本数量。 |
| Schema 模式 | It is a collection of named objects. It provides a way to group those objects logically. A schema is also a name qualifier; it provides a way to use the same natural name for several objects and prevent ambiguous references. 它是一组命名对象的集合。它提供了一种逻辑分组这些对象的方法。模式也是一个名称限定符；它提供了一种使用相同的自然名称来表示多个对象的方法，并防止歧义引用。 |
| Secondary node 次级节点 | This node is also known as a data node. There can be hundreds of data nodes in the HDFS that manage the storage system. They perform read and write requests at the instructions of the name node. They also create, replicate, and delete file blocks based on instructions from the name node. 该节点也称为数据节点。在 HDFS 中可以有数百个数据节点来管理存储系统。它们根据名称节点的指令执行读写请求。它们还根据名称节点的指令创建、复制和删除文件块。 |
| Semi-structured data 半结构化数据 | Semi-structured data (JSON, CSV, XML) is the "bridge" between structured and unstructured data. It does not have a predefined data model and is more complex than structured data, yet easier to store than unstructured data. 半结构化数据（JSON、CSV、XML）是结构化数据和非结构化数据之间的"桥梁"。它没有预定义的数据模型，比结构化数据更复杂，但比非结构化数据更容易存储。 |
| Shuffle 洗牌 | Phase in which interim map output from mappers transfers to reducers. Every reducer fetches interim results for all values associated with the same key from multiple nodes. This is a network-intensive operation within the Hadoop cluster nodes. 一个阶段，其中从映射器传输到归约器的中间映射输出。每个归约器从多个节点获取与相同键关联的所有值的中间结果。这是一个在 Hadoop 集群节点中网络密集型的操作。 |
| Sqoop | An open-source product designed to transfer bulk data between relational database systems and Hadoop. It looks at the relational database and summarizes the schema. It generates MapReduce code to import and export data. It helps develop other MapReduce applications that use the records stored in HDFS. 一个开源产品，设计用于在关系型数据库系统与 Hadoop 之间传输大量数据。它分析关系型数据库并总结其模式。它生成 MapReduce 代码以导入和导出数据。它有助于开发其他使用 HDFS 中存储记录的 MapReduce 应用程序。 |
| Streaming 流式传输 | Implies HDFS provides a constant bitrate when transferring data rather than having the data transferred in waves. 意味着 HDFS 在传输数据时提供恒定的比特率，而不是以波浪形式传输数据。 |
| Structured data 结构化数据 | Structured data, typically categorized as quantitative data, is highly organized and easily decipherable by machine learning algorithms. Developed by IBM in 1974, structured query language (SQL) is the programming language used to manage structured data. 结构化数据，通常被归类为定量数据，具有高度的组织性，并且易于机器学习算法解析。结构化查询语言（SQL）是由 IBM 于 1974 年开发的，是用于管理结构化数据的编程语言。 |
| Unstructured data 非结构化数据 | Information lacking a predefined data model or not fitting into relational tables. 缺乏预定义数据模型或无法放入关系表中的信息。 |
| Write 编写 | In this operation, the Name node ensures that the file does not exist. If the file exists, the client gets an IO Exception message. If the file does not exist, the client is given access to start writing files. 在此操作中，Name 节点确保文件不存在。如果文件存在，客户端将收到一个 IO 异常消息。如果文件不存在，客户端将获得写入文件的权限。 |
| Yet Another Resource Negotiator (YARN) | Prepares Hadoop for batch, stream, interactive, and graph processing. 为批处理、流处理、交互式和图处理准备 Hadoop。 |