Coursera 笔记

Module 02 · The Data Engineering Ecosystem数据工程生态系统

Summary and Highlights总结与要点

Coursera ↗

In this lesson, you have learned:
在本课程中,您学习了:

A Data Repository is a general term that refers to data that has been collected, organized, and isolated so that it can be used for reporting, analytics, and also for archival purposes.
数据仓库是一个通用术语,指的是经过收集、组织和隔离的数据,以便用于报告、分析和存档目的。

The different types of Data Repositories include:
数据仓库的不同类型包括:

  • Databases, which can be relational or non-relational, each following a set of organizational principles, the types of data they can store, and the tools that can be used to query, organize, and retrieve data.
    数据库,可以是关系型或非关系型,每种数据库都遵循一套组织原则、可存储的数据类型以及用于查询、组织和检索数据的工具。
  • Data Warehouses, that consolidate incoming data into one comprehensive store house.
    数据仓库,将传入数据整合到一个综合存储库中。
  • Data Marts, that are essentially sub-sections of a data warehouse, built to isolate data for a particular business function or use case.
    数据集市,本质上是数据仓库的子部分,用于隔离特定业务功能或用例的数据。
  • Data Lakes, that serve as storage repositories for large amounts of structured, semi-structured, and unstructured data in their native format.
    数据湖,作为存储大量结构化、半结构化和非结构化数据的存储库,保留其原始格式。
  • Big Data Stores, that provide distributed computational and storage infrastructure to store, scale, and process very large data sets.
    大数据存储,提供分布式计算和存储基础设施,用于存储、扩展和处理非常大的数据集。

The ETL, or Extract Transform and Load, Process is an automated process that converts raw data into analysis-ready data by:
ETL(提取、转换和加载)过程是一个自动化过程,通过以下方式将原始数据转换为分析就绪的数据:

  • Extracting data from source locations.
    从源位置提取数据。
  • Transforming raw data by cleaning, enriching, standardizing, and validating it.
    通过清洗、丰富、标准化和验证来转换原始数据。
  • Loading the processed data into a destination system or data repository.
    将处理后的数据加载到目标系统或数据存储库中。

The ELT, or Extract Load and Transform, Process is a variation of the ETL Process. In this process, extracted data is loaded into the target system before the transformations are applied. This process is ideal for Data Lakes and working with Big Data.
ELT(提取、加载和转换)过程是 ETL 过程的一种变体。在此过程中,提取的数据在应用转换之前被加载到目标系统中。该过程非常适合数据湖和大数据处理。

Data Pipeline, sometimes used interchangeably with ETL and ELT, encompasses the entire journey of moving data from its source to a destination data lake or application, using the ETL or ELT process.
数据管道,有时与 ETL 和 ELT 互换使用,涵盖了将数据从其源移动到目标数据湖或应用程序的整个过程,使用 ETL 或 ELT 流程。

Data Integration Platforms combine disparate sources of data, physically or logically, to provide a unified view of the data for analytics purposes.
数据集成平台将不同的数据源物理或逻辑上组合起来,为分析目的提供统一的数据视图。