Module 03 · Apache Spark
Module 3 Glossary: Apache Spark模块 3 术语表：Apache Spark
Welcome! This alphabetized glossary contains many of the terms in this course. This comprehensive glossary also includes additional industry-recognized terms not used in course videos. These terms are essential for you to recognize when working in the industry, participating in user groups, and in other professional certificate programs.
欢迎！这个按字母顺序排列的词汇表包含了本课程中的许多术语。这个全面的词汇表还包括了课程视频中没有使用的行业认可术语。这些术语对于您在行业内工作、参与用户组以及在其他专业证书课程中都是必不可少的。
Estimated reading time: 10 minutes
预计阅读时间：10 分钟
| Term 术语 | Definition 定义 |
|---|---|
| Amazon Simple Storage Service (Amazon S3) 亚马逊简单存储服务（Amazon S3） |
An object store interface protocol that Amazon invented. It is a Hadoop component that understands the S3 protocol. S3 provides an interface for Hadoop services, such as IBM Db2 Big SQL, to consume S3-hosted data. 亚马逊发明的一种对象存储接口协议。它是一个理解 S3 协议的 Hadoop 组件。S3 为 Hadoop 服务（如 IBM Db2 Big SQL）提供接口，以消费 S3 托管的数据。 |
| Apache Spark | An in-memory and open-source application framework for distributed data processing and iterative analysis of enormous data volumes. 一个内存中且开源的应用框架，用于分布式数据处理和海量数据的迭代分析。 |
| Application programming interface (API) 应用程序编程接口（API） |
Set of well-defined rules that help applications communicate with each other. It functions as an intermediary layer for processing data transfer between systems, allowing companies to open their application data and functionality to business partners, third-party developers, and other internal departments. 一套定义良好的规则，帮助应用程序相互通信。它作为系统间数据传输处理的中介层，允许公司向商业伙伴、第三方开发者和其他内部部门开放其应用程序数据和功能。 |
| Big data 大数据 | Data sets whose type or size supersedes the ability of traditional relational databases to manage, capture, and process the data with low latency. Big data characteristics include high volume, velocity, and variety. 数据集的类型或规模超过了传统关系型数据库管理、捕获和处理数据的能力，且具有低延迟。大数据的特征包括高容量、高速度和高多样性。 |
| Classification algorithms 分类算法 |
A type of machine learning algorithm that helps computers learn how to categorize things into different groups based on patterns they find in data. 一种机器学习算法，帮助计算机根据数据中发现的模式将事物分类到不同的组中。 |
| Cluster management framework 集群管理框架 |
It handles the distributed computing aspects of Spark. It can exist as a stand-alone server, Apache Mesos, or Yet Another Resource Network (YARN). A cluster management framework is essential for scaling big data. 它处理 Spark 的分布式计算方面。它可以作为独立服务器、Apache Mesos 或另一个资源网络（YARN）存在。集群管理框架对于大数据扩展至关重要。 |
| Commodity hardware 商用硬件 | Consists of low-cost workstations or desktop computers that are IBM-compatible and run multiple operating systems such as Microsoft Windows, Linux, and DOS without additional adaptations or software. 由低成本的 IBM 兼容工作站或台式计算机组成，可运行 Microsoft Windows、Linux 和 DOS 等多种操作系统，无需额外的适配或软件。 |
| Compute interface 计算接口 | A shared boundary in computing against which two or more different computer system components exchange information. 在计算中，两个或多个不同的计算机系统组件之间交换信息的共享边界。 |
| Data engineering 数据工程 | A prominent practice that entails designing and building systems for collecting, storing, and analyzing data at scale. It is a discipline with applications in different industries. Data engineers use Spark tools, including the core Spark engine, clusters, executors and their management, Spark SQL, and DataFrames. 一种突出的实践，涉及设计和构建用于大规模收集、存储和分析数据的系统。它是一个在不同行业有应用领域的学科。数据工程师使用 Spark 工具，包括核心 Spark 引擎、集群、执行器和它们的管理，Spark SQL 和 DataFrames。 |
| Data science 数据科学 | Discipline that combines math and statistics, specialized programming, advanced analytics, artificial intelligence (AI), and machine learning with specific subject matter expertise to unveil actionable insights hidden in the organization's data. These insights can be used in decision-making and strategic planning. 一门结合数学与统计学、专业编程、高级分析、人工智能（AI）和机器学习，并运用特定领域专业知识来揭示组织中隐藏的可操作洞察力的学科。这些洞察力可用于决策和战略规划。 |
| DataFrames 数据帧 | Data collection categorically organized into named columns. DataFrames are conceptually equivalent to a table in a relational database and similar to a dataframe in R or Python, but with greater optimizations. They are built on top of the Spark SQL RDD API. They use RDDs to perform relational queries. Also, they are highly scalable and support many data formats and storage systems. They are developer-friendly, offering integration with most big data tools via Spark and APIs for Python, Java, Scala, and R. 将数据按命名列进行分类组织的数据集合。数据帧在概念上等同于关系数据库中的表，类似于 R 或 Python 中的数据帧，但具有更高的优化。它们基于 Spark SQL 的 RDD API 构建。它们使用 RDD 来执行关系查询。此外，它们具有高度的可扩展性，并支持多种数据格式和存储系统。它们对开发者友好，通过 Spark 和 Python、Java、Scala 和 R 的 API，与大多数大数据工具集成。 |
| Declarative programming 声明式编程 | A programming paradigm that a programmer uses to define the program's accomplishment without defining how it needs to be implemented. The approach primarily focuses on what needs to be achieved, rather than advocating how to achieve it. 一种编程范式，程序员通过它来定义程序要完成什么，而不需要定义如何实现它。这种方法主要关注需要实现什么，而不是提倡如何实现。 |
| Distributed computing 分布式计算 | A group of computers or processors working together behind the scenes. It is often used interchangeably with parallel computing. Each processor accesses its own memory. 一组在幕后协同工作的计算机或处理器。它通常与并行计算互换使用。每个处理器访问自己的内存。 |
| Fault tolerance 容错性 | A system is fault-tolerant if it can continue performing despite parts failing. Fault tolerance helps to make your remote-boot infrastructure more robust. In the case of OS deployment servers, the whole system is fault-tolerant if the OS deployment servers back up each other. 如果一个系统在部分失效时仍能继续运行，则称该系统具有容错性。容错性有助于使远程启动基础设施更加健壮。在操作系统部署服务器的案例中，如果操作系统部署服务器之间相互备份，则整个系统具有容错性。 |
| For-loop For 循环 | Extends from a FOR statement to an END FOR statement and executes for a specified number of iterations, defined in the FOR statement. 从 FOR 语句到 END FOR 语句扩展，并执行指定次数的迭代，该次数在 FOR 语句中定义。 |
| Functional programming (FP) 函数式编程（FP） |
A style of programming that follows the mathematical function format. Declarative implies that the emphasis of the code or program is on the "what" of the solution as opposed to the "how to" of the solution. Declarative syntax abstracts out the implementation details and only emphasizes the final output, restating "the what." We use expressions in functional programming, such as the expression f of x, as mentioned earlier. 一种遵循数学函数格式的编程风格。声明式意味着代码或程序的重点在于解决方案的"什么"，而不是解决方案的"如何"。声明式语法抽象出实现细节，只强调最终输出，重申"什么"。我们在函数式编程中使用表达式，例如前面提到的表达式 f(x)。 |
| Hadoop | An open-source software framework offering reliable distributed processing of large data sets by using simplified programming models. 一个开源软件框架，通过使用简化的编程模型提供可靠的大数据集分布式处理。 |
| Hadoop Common | Fundamental part of the Apache Hadoop framework. It refers to a collection of primary utilities and libraries that support other Hadoop modules. Apache Hadoop 框架的基本组成部分。它指的是支持其他 Hadoop 模块的一组主要工具和库。 |
| Hadoop Distributed File System (HDFS) Hadoop 分布式文件系统（HDFS） |
A file system distributed on multiple file servers, allowing programmers to access or store files from any network or computer. It is the storage layer of Hadoop. It works by splitting the files into blocks, creating replicas of the blocks, and storing them on different machines. It is built to access streaming data seamlessly. It uses a command-line interface to interact with Hadoop. 一个分布在多个文件服务器上的文件系统，允许程序员从任何网络或计算机访问或存储文件。它是 Hadoop 的存储层。它通过将文件分割成块、创建块的副本并将它们存储在不同的机器上来工作。它是为无缝访问流数据而构建的。它使用命令行界面与 Hadoop 进行交互。 |
| HBase | A column-oriented, non-relational database system that runs on top of Hadoop Distributed File System (HDFS). It provides real-time wrangling access to the Hadoop file system. It uses hash tables to store data in indexes, allowing for random data access and making lookups faster. 一种列式、非关系型数据库系统，运行在 Hadoop 分布式文件系统（HDFS）之上。它提供对 Hadoop 文件系统的实时处理访问。它使用哈希表在索引中存储数据，允许随机数据访问，并使查找速度更快。 |
| Immutable 不可变 | This type of object storage allows users to set indefinite retention on the object if they are unsure of the final duration of the retention period or want to use event-based retention. Once set to indefinite, user applications can change the object retention to a finite value. 这种对象存储允许用户在不确定保留期的最终持续时间或希望使用基于事件的保留时，对对象设置无限期保留。一旦设置为无限期，用户应用程序可以将对象的保留更改为有限值。 |
| Imperative programming paradigm 命令式编程范式 |
In this software development paradigm, functions are implicitly coded in every step used in solving a problem. Every operation is coded, specifying how the problem will be solved. This implies that pre-coded models are not called on. 在这种软件开发范式中，函数在解决问题的每一步中都被隐式编码。每项操作都被编码，指定了问题将如何被解决。这意味着不会调用预编码的模型。 |
| In-memory processing 内存内处理 | The practice of storing and manipulating data directly in a computer's main memory (RAM), allowing for faster and more efficient data operations compared to traditional disk-based storage. 将数据直接存储和处理在计算机的主内存（RAM）中的实践，与传统的基于磁盘的存储相比，可以实现更快、更高效的数据操作。 |
| Iterative process 迭代过程 | An approach to continuously improving a concept, design, or product. Creators produce a prototype, test it, tweak it, and repeat the cycle to get closer to the solution. 一种持续改进概念、设计或产品的方法。创造者制作原型，测试它，调整它，并重复循环以更接近解决方案。 |
| Java | A technology equipped with a programming language and a software platform. 一种配备编程语言和软件平台的技术。 |
| Java virtual machines (JVMs) Java 虚拟机 (JVM) |
The platform-specific component that runs a Java program. At runtime, the VM interprets the Java bytecode compiled by the Java compiler. The VM is a translator between the language and the underlying operating system and hardware. 一种特定于平台的组件，用于运行 Java 程序。在运行时，JVM 解释由 Java 编译器编译的 Java 字节码。JVM 是语言与底层操作系统和硬件之间的翻译器。 |
| JavaScript Object Notation (JSON) JavaScript 对象表示法 (JSON) |
A simplified data-interchange format based on a subset of the JavaScript programming language. IBM Integration Bus provides support for a JSON domain. The JSON parser and serializer process messages in the JSON domain. 一种基于 JavaScript 编程语言子集的简化数据交换格式。IBM 集成总线为 JSON 域提供支持。JSON 解析器和序列化器处理 JSON 域中的消息。 |
| Lambda calculus Lambda 演算 | A mathematical concept that implies every computation can be expressed as an anonymous function that is applied to a data set. 一种数学概念，意味着所有计算都可以表示为应用于数据集的匿名函数。 |
| Lambda functions Lambda 函数 | Calculus functions, or operators. These are anonymous functions that enable functional programming. They are used to write functional programming code. 演算函数，或运算符。这些是支持函数式编程的匿名函数，用于编写函数式编程代码。 |
| List processing language (Lisp) 列表处理语言（Lisp） |
The functional programming language that was initially used in the 1950s. Today, there are many functional programming language options, including Scala, Python, R, and Java. 一种最初在 20 世纪 50 年代使用的函数式编程语言。如今，有许多函数式编程语言可供选择，包括 Scala、Python、R 和 Java。 |
| Machine learning 机器学习 | A full-service cloud offering that allows developers and data scientists to collaborate and integrate predictive capabilities with their applications. 一项全面云服务，允许开发者和数据科学家协作，并将预测能力集成到他们的应用程序中。 |
| MapReduce | A program model and processing technique used in distributed computing based on Java. It splits the data into smaller units and processes big data. It is the first method used to query data stored in HDFS. It allows massive scalability across hundreds or thousands of servers in a Hadoop cluster. 一种基于 Java 的分布式计算中使用的程序模型和处理技术。它将数据分割成更小的单元，并处理大数据。它是查询存储在 HDFS 中数据的第一种方法。它允许在 Hadoop 集群中的数百或数千台服务器上实现大规模扩展。 |
| Modular development 模块化开发 | Techniques used in job designs to maximize the reuse of parallel jobs and components and save user time. 在任务设计中用于最大化并行任务和组件的复用，并节省用户时间的技巧。 |
| Parallel computing 并行计算 | A computing architecture in which multiple processors execute different small calculations fragmented from a large, complex problem simultaneously. 一种计算架构，其中多个处理器同时执行从大型复杂问题中分解出来的不同小计算。 |
| Parallel programming 并行编程 | It resembles distributed programming. It is the simultaneous use of multiple compute resources to solve a computational task. Parallel programming parses tasks into discrete parts solved concurrently using multiple processors. The processors access a shared pool of memory, which has control and coordination mechanisms in place. 它类似于分布式编程。它是同时使用多个计算资源来求解一个计算任务。并行编程将任务解析为离散的部分，使用多个处理器并发求解。处理器访问一个共享的内存池，该内存池具有控制和协调机制。 |
| Parallelization 并行化 | Parallel regions of program code executed by multiple threads, possibly running on multiple processors. Environment variables determine the number of threads created and calls to library functions. 由多个线程执行的程序代码的并行区域，这些线程可能运行在多个处理器上。环境变量决定了创建的线程数量，以及对库函数的调用。 |
| Persistent cache 持久缓存 | Information is stored in "permanent" memory. Therefore, data is not lost after a system crash or restart, as if it were stored in cache memory. 信息存储在"永久"存储器中。因此，在系统崩溃或重启后，数据不会丢失，就像存储在缓存存储器中一样。 |
| Python | Easy-to-learn, high-level, interpreted, and general-purpose dynamic programming language focusing on code readability. It provides a robust framework for building fast and scalable applications for z/OS, with a rich ecosystem of modules to develop new applications like any other platform. 一种易于学习、高级、解释型、通用的动态编程语言，注重代码可读性。它为构建适用于 z/OS 的快速可扩展应用程序提供了一个强大的框架，并拥有丰富的模块生态系统，可以像其他平台一样开发新应用程序。 |
| R | An open-source, optimized programming language for statistical analysis and data visualization. Developed in 1992, it has a rich ecosystem with complex data models and elegant tools for data reporting. 一种用于统计分析和数据可视化的开源、优化型编程语言。它于 1992 年开发，拥有丰富的生态系统，具有复杂的数据模型和优雅的数据报告工具。 |
| Redundancy 冗余 | Duplication of data across multiple partitions or nodes in a cluster. This duplication is implemented to enhance fault tolerance and reliability. If one partition or node fails, the duplicated data on other partitions or nodes can still be used to ensure that the computation continues without interruption. Redundancy is critical in maintaining data availability and preventing data loss in distributed computing environments like Spark clusters. 在集群的多个分区或节点之间对数据进行复制。这种复制是为了提高容错性和可靠性。如果一个分区或节点发生故障，其他分区或节点上的复制数据仍然可以使用，以确保计算能够无中断地继续进行。冗余对于在 Spark 集群等分布式计算环境中维护数据可用性和防止数据丢失至关重要。 |
| Resilient Distributed Datasets (RDDs) 弹性分布式数据集（RDD） |
A fundamental abstraction in Apache Spark that represents distributed collections of data. RDDs allow you to perform parallel and fault-tolerant data processing across a cluster of computers. RDDs can be created from existing data in storage systems (like HDFS), and they can undergo various transformations and actions to perform operations like filtering, mapping, and aggregating. The "resilient" aspect refers to resilient distributed datasets (RDDs) ability to recover from node failures, and the "distributed" aspect highlights their distribution across multiple machines in a cluster, enabling parallel processing. Apache Spark 中的一个基本抽象，表示分布式数据集合。RDD 允许您在计算机集群上执行并行和容错的数据处理。RDD 可以由存储系统（如 HDFS）中的现有数据创建，并且可以经历各种转换和操作以执行过滤、映射和聚合等操作。"弹性"方面指的是弹性分布式数据集（RDD）从节点故障中恢复的能力，而"分布式"方面则突出了它们在集群中的多台机器上的分布，从而实现并行处理。 |
| Scala | A general-purpose programming language that supports both object-oriented and functional programming. The most recent representative in the family of programming languages. Apache Spark is written mainly in Scala, which treats functions as first-class citizens. Functions in Scala can be passed as arguments to other functions, returned by other functions, and used as variables. 一种通用编程语言，支持面向对象和函数式编程。它是编程语言家族中最新的代表。Apache Spark 主要用 Scala 编写，该语言将函数视为一等公民。Scala 中的函数可以作为参数传递给其他函数，由其他函数返回，并用作变量。 |
| Scalability 可扩展性 | The ability of a system to take advantage of additional resources, such as database servers, processors, memory, or disk space. It aims at minimizing the impact on maintenance. It is the ability to maintain all servers efficiently and quickly with minimal impact on user applications. 系统利用额外资源（如数据库服务器、处理器、内存或磁盘空间）的能力。其目标是最小化维护影响。这是指能够以最小的用户应用程序影响高效快速地维护所有服务器的能力。 |
| Spark applications Spark 应用程序 | Include a driver program and executors that run the user's multiple primary functions and different parallel operations in a cluster. 包含一个驱动程序和执行器，用于在集群中运行用户的多个主要函数和不同的并行操作。 |
| Spark Core | Often popularly referred to as "Spark." The fault-tolerant Spark Core is the base engine for large-scale parallel and distributed data processing. It manages memory and task scheduling. It also contains the APIs used to define RDDs and other datatypes. It parallelizes a distributed collection of elements across the cluster. 通常被称为"Spark。" 容错性强的 Spark Core 是大规模并行和分布式数据处理的基础引擎。它管理内存和任务调度。它还包含用于定义 RDDs 和其他数据类型的 API。它将分布式元素集合并行化到集群中。 |
| Spark ML | Spark's machine learning library for creating and using machine learning models on large data sets across distributed clusters. Spark 的机器学习库，用于在分布式集群上创建和使用大规模数据集的机器学习模型。 |
| Spark SQL | A Spark module for structured data processing. Users can interact with Spark SQL using SQL queries and the DataFrame API. Spark SQL supports Java, Scala, Python, and R APIs. Spark SQL uses the same execution engine to compute the result independently of the API or language used for computation. Developers can use the API to help express a given transformation. Unlike the basic Spark RDD API, Spark SQL includes a cost-based optimizer, columnar storage, and code generation to perform optimizations that equip Spark with information about the structure of data and the computation in process. 一个用于结构化数据处理 Spark 模块。用户可以使用 SQL 查询和 DataFrame API 与 Spark SQL 进行交互。Spark SQL 支持 Java、Scala、Python 和 R API。Spark SQL 使用相同的执行引擎来计算结果，而与用于计算的 API 或语言无关。开发者可以使用 API 来帮助表达给定的转换。与基本的 Spark RDD API 不同，Spark SQL 包括基于成本的优化器、列式存储和代码生成，以执行优化，使 Spark 具备有关数据结构和处理中计算的信息，从而进行优化。 |
| SQL Procedural code SQL 过程性代码 | A set of instructions written in a programming language within an SQL database environment. This code allows users to perform more complex tasks and create custom functions, procedures, and control structures, enabling them to manipulate and manage data in a more controlled and structured manner. 一个在 SQL 数据库环境中以编程语言编写的指令集。该代码允许用户执行更复杂的任务，并创建自定义函数、过程和控制结构，使他们能够以更受控和结构化的方式操作和管理数据。 |
| Streaming analytics 流式分析 | Help leverage streams to ingest, analyze, monitor, and correlate data from real-time data sources. They also help to view information and events as they unfold. 帮助利用流来摄取、分析、监控和关联来自实时数据源的数据。它们还有助于实时查看信息和事件。 |
| Worker node 工作节点 | A unit in a distributed system that performs tasks and processes data according to instructions from a central coordinator. 分布式系统中根据中央协调器的指令执行任务和处理数据的单元。 |