Coursera 笔记

Module 03 · Data Engineering Lifecycle数据工程生命周期

Summary and Highlights总结与要点

Coursera ↗

In this lesson, you have learned:
在本课程中,您已学习:

The architecture of a data platform can be seen as a set of layers, or functional components, each one performing a set of specific tasks. These layers include:
数据平台的架构可以看作是一组层或功能组件,每个组件都执行一组特定的任务。这些层包括:

  • Data Ingestion or Data Collection Layer, responsible for bringing data from source systems into the data platform.
    数据摄取或数据收集层,负责将数据从源系统引入数据平台。
  • Data Storage and Integration Layer, responsible for storing and merging extracted data.
    数据存储和集成层,负责存储和合并提取的数据。
  • Data Processing Layer, responsible for validating, transforming, and applying business rules to data.
    数据处理层,负责验证、转换数据并对数据应用业务规则。
  • Analysis and User Interface Layer, responsible for delivering processed data to data consumers.
    分析和用户界面层,负责将处理后的数据交付给数据消费者。
  • Data Pipeline Layer, responsible for implementing and maintaining a continuously flowing data pipeline.
    数据管道层,负责实施和维护持续流动的数据管道。

A well-designed data repository is essential for building a system that is scalable and capable of performing during high workloads. The choice or design of a data store is influenced by the type and volume of data that needs to be stored, the intended use of data, and storage considerations. The privacy, security, and governance needs of your organization also influence this choice. The CIA, or Confidentiality, Integrity, and Availability triad are three key components of an effective strategy for information security. The CIA triad is applicable to all facets of security, be it infrastructure, network, application, or data security.
设计良好的数据存储库对于构建可扩展且在高工作负载下仍能正常运行的系统至关重要。 数据存储的选择或设计受需要存储的数据类型和数据量、数据的预期用途以及存储注意事项的影响。您组织的隐私、安全和治理需求也会影响此选择。 CIA,即保密性(Confidentiality)、完整性(Integrity)和可用性(Availability)三要素,是有效信息安全策略的三个关键组成部分。CIA 三要素适用于安全的所有方面,无论是基础设施、网络、应用程序还是数据安全。