Summary and Highlights总结与要点

In this lesson, you have learned:
在本课程中，您学习了：

A Data Engineer’s ecosystem includes the infrastructure, tools, frameworks, and processes for extracting data, architecting and managing data pipelines and data repositories, managing workflows, developing applications, and managing BI and Reporting tools.
数据工程师的生态系统包括用于提取数据、架构和管理数据管道及数据存储库的基础设施、工具、框架和流程，以及管理工作流、开发应用程序和管理 BI 和报告工具。

Based on how well-defined the structure of the data is, data can be categorized as
根据数据结构的定义程度，数据可以分为

• Structured data, that is data which is well organized in formats that can be stored in databases.
  结构化数据，即数据以可存储在数据库中的格式良好组织的数据。
• Semi-structured data, that is data which is partially organized and partially free-form.
  半结构化数据，即部分组织化和部分自由格式化的数据。
• Unstructured data, that is data which can not be organized conventionally into rows and columns.
  非结构化数据，即无法按照传统方式组织成行和列的数据。

Data comes in a wide-ranging variety of file formats, such as, delimited text files, spreadsheets, XML, PDF, and JSON, each with its own list of benefits and limitations of use.
数据以多种多样的文件格式存在，例如分隔符文本文件、电子表格、XML、PDF 和 JSON，每种格式都有其自身的优缺点。

Data is extracted from multiple data sources, ranging from relational and non-relational databases, to APIs, web services, data streams, social platforms, and sensor devices.
数据从多个数据源中提取，范围从关系型和非关系型数据库，到 API、Web 服务、数据流、社交平台和传感器设备。

Once the data is identified and gathered from different sources, it needs to be staged in a data repository so that it can be prepared for analysis. The type, format, and sources of data influence the type of data repository that can be used.
一旦从不同来源识别并收集了数据，就需要将其存放在数据存储库中，以便进行分析。数据的类型、格式和来源会影响可使用的数据存储库类型。

Data professionals need a host of languages that can help them extract, prepare, and analyse data. These can be classified as:
数据专业人员需要一系列语言来帮助他们提取、准备和分析数据。这些语言可以分为：

• Querying languages, such as SQL, used for accessing and manipulating data from databases.
  查询语言，如 SQL，用于访问和操作数据库中的数据。
• Programming languages such as Python, R, and Java, for developing applications and controlling application behavior.
  编程语言，如 Python、R 和 Java，用于开发应用程序和控制应用程序行为。
• Shell and Scripting languages, such as Unix/Linux Shell, and PowerShell, for automating repetitive operational tasks.
  Shell 和脚本语言，如 Unix/Linux Shell 和 PowerShell，用于自动化重复的运维任务。