Module 03 · Data Engineering Lifecycle数据工程生命周期
Summary and Highlights总结与要点
In this lesson, you have learned,
在本课程中,您已学习到:
- In order for raw data to become analytics-ready, a number of transformation and cleansing tasks need to be performed on raw data. And that requires you to understand your dataset from multiple perspectives. One of the ways in which you can explore your dataset is to query it.
为了使原始数据能够用于分析,需要对原始数据执行多项转换和清洗任务。这要求您从多个角度理解您的数据集。探索数据集的方法之一是查询它。 - Basic querying techniques can help you explore your data, such as, counting and aggregating a dataset, identifying extreme values, slicing data, sorting data, filtering patterns, and grouping data.
基本的查询技术可以帮助您探索数据,例如:计数和聚合数据集、识别极端值、切片数据、排序数据、过滤模式和分组数据。 - In a data engineering lifecycle, the performance of data pipelines, platforms, databases, applications, tools, queries, and scheduled jobs, need to be constantly monitored for performance and availability.
在数据工程生命周期中,数据管道、平台、数据库、应用程序、工具、查询和计划作业的性能和可用性需要持续监控。 - The performance of a data pipeline can get impacted if the workload increases significantly, or there are application failures, or a scheduled job does not work as expected, or some of the tools in the pipeline run into compatibility issues.
如果工作负载显著增加、应用程序出现故障、计划作业未按预期运行,或者管道中的某些工具出现兼容性问题,数据管道的性能可能会受到影响。 - Databases are susceptible to outages, capacity overutilization, application slowdown, and conflicting activities and queries being executed simultaneously.
数据库容易出现中断、容量过度利用、应用程序运行缓慢以及同时执行的冲突活动和查询。 - Monitoring and alerting systems collect quantitative data in real time to give visibility into the performance of data pipelines, platforms, databases, applications, tools, queries, scheduled jobs, and more.
监控和警报系统实时收集定量数据,以提供数据管道、平台、数据库、应用程序、工具、查询、计划作业等的性能可见性。 - Time-based and condition-based maintenance schedules generate data that helps identify systems and procedures responsible for faults and low availability.
基于时间和基于条件的维护计划会生成数据,有助于识别导致故障和低可用性的系统和程序。