Optional: Overview of the DataOps Methodology可选：DataOps 方法论概述

Gartner defines DataOps as a collaborative data management practice focused on improving the communication, integration, and automation of data flows between data managers and consumers across an organization. DataOps aims to create predictable delivery and change management of data, data models, and related artifacts. DataOps uses technology to automate data delivery with the appropriate levels of security, quality, and metadata to improve the use and value of data in a dynamic environment.” (Source: https://blogs.gartner.com/nick-heudecker/hyping-dataops/) A small team working on a simpler or limited number of use cases can meet business requirements efficiently. As data pipelines and data infrastructures get more complex, and data teams and consumers grow in size, you need development processes and efficient collaboration between teams to govern the data and analytics lifecycle. From data ingestion and data processing to analytics and reporting, you need to reduce data defects, ensure shorter cycle times, and ensure 360-degree access to quality data for all stakeholders. DataOps helps you achieve this through metadata management, workflow and test automation, code repositories, collaboration tools, and orchestration to help manage complex tasks and workflows. Using the DataOps methodology ensures all activities occur in the right order the right security permissions. It helps set in a continual process that allows you to cut wastages, streamline steps, automate processes, increase throughput, and improve continually. Several DataOps Platforms are available in the market, some of the popular ones being IBM DataOps, Nexla, Switchboard, Streamsets, and Infoworks.
Gartner 将 DataOps 定义为一种协作式数据管理实践，其重点是改进组织内数据管理者和消费者之间数据流的沟通、集成和自动化。DataOps 旨在实现数据、数据模型和相关工件的可预测交付和变更管理。DataOps 利用技术通过适当的安全级别、质量和元数据实现数据交付自动化，以提高数据在动态环境中的使用和价值。 （来源：https://blogs.gartner.com/nick-heudecker/hyping-dataops/） 一个处理较简单或数量有限用例的小团队可以高效地满足业务需求。随着数据管道和数据基础设施变得越来越复杂，以及数据团队和消费者规模的增长，您需要开发流程和团队之间的高效协作来管理数据和分析生命周期。从数据摄取和数据处理到分析和报告，您需要减少数据缺陷，确保更短的周期时间，并确保所有利益相关者都能 360 度访问高质量数据。 DataOps 通过元数据管理、工作流和测试自动化、代码存储库、协作工具和编排来帮助您实现这一目标，从而管理复杂的任务和工作流。使用 DataOps 方法可确保所有活动以正确的顺序和正确的安全权限进行。它有助于建立一个持续的过程，让您能够减少浪费、简化步骤、自动化流程、提高吞吐量并持续改进。 市面上有多种 DataOps 平台，其中一些比较流行的包括 IBM DataOps、Nexla、Switchboard、Streamsets 和 Infoworks。

DataOps Methodology: DataOps 方法论

The purpose of the DataOps Methodology is to enable an organization to utilize a repeatable process to build and deploy analytics and data pipelines. Successful implementation of this methodology allows an organization to know, trust, and use data to drive value.
DataOps 方法论的目的是使组织能够利用可重复的流程来构建和部署分析和数据管道。成功实施此方法论可使组织了解、信任和使用数据来创造价值。

It ensures that the data used in problem-solving and decision making is relevant, reliable, and traceable and improves the probability of achieving desired business outcomes. And it does so by tackling the challenges associated with inefficiencies in accessing, preparing, integrating, and making data available.
它确保用于解决问题和决策的数据是相关的、可靠的和可追溯的，并提高了实现预期业务成果的可能性。它通过解决与数据访问、准备、集成和可用性效率低下相关的挑战来实现这一点。

In a nutshell, the DataOps Methodology consists of three main phases:
简而言之，DataOps 方法论包含三个主要阶段：

• The Establish DataOps Phase provides guidance on how to set up the organization for success in managing data.
  建立 DataOps 阶段提供关于如何设置组织以成功管理数据的指导。
• The Iterate DataOps Phase delivers the data for one defined sprint.
  迭代 DataOps 阶段为一次定义的冲刺交付数据。
• The Improve DataOps Phase ensures learnings from each sprint is channeled back to continually improve the DataOps process.
  改进 DataOps 阶段确保每次冲刺的学习成果被反馈回来，以持续改进 DataOps 流程。

The figure below presents a high-level overview of these phases and the key activities within each of these phases.
下图概述了这些阶段以及每个阶段中的关键活动。

Benefits of using the DataOps methodology: 使用 DataOps 方法的好处

Adopting the DataOps methodology helps organizations to organize their data and make it more trusted and secure. Using the DataOps methodology, organizations can:
采用 DataOps 方法有助于组织整理数据，使其更值得信赖和安全。通过使用 DataOps 方法，组织可以：

• Automate metadata management and catalog data assets, making them easy to access.
  自动化元数据管理和编目数据资产，使其易于访问。
• Trace data lineage to establish its credibility and for compliance and audit purposes.
  追踪数据沿袭，以建立其可信度，并用于合规和审计目的。
• Automate workflows and jobs in the data lifecycle to ensure data integrity, relevancy, and security.
  自动化数据生命周期中的工作流和作业，以确保数据的完整性、相关性和安全性。
• Streamline the workflow and processes to ensure data access and delivery needs can be met at optimal speed.
  简化工作流和流程，以确保能以最佳速度满足数据访问和交付需求。
• Ensure a business-ready data pipeline that is always available for all data consumers and business stakeholders.
  确保业务就绪的数据管道始终可供所有数据消费者和业务利益相关者使用。
• Build a data-driven culture in the organization through automation, data quality, and governance.
  通过自动化、数据质量和治理，在组织内建立数据驱动的文化。

As a data practitioner, using the methodology can help you reduce development time, cut wastages and duplication of effort, increase your productivity and throughput, and ensure that your actions produce the best possible quality of data.
作为数据从业者，使用该方法可以帮助您减少开发时间，减少浪费和重复工作，提高您的生产力和吞吐量，并确保您的操作产生尽可能高质量的数据。

With DataOps, data professionals, consumers, and stakeholders can collaborate more effectively towards the shared goal of creating valuable insights for business. While implementing the methodology will require systemic change, time, and resources, but in the end, it makes data and analytics more efficient and reliable.
通过 DataOps，数据专业人员、消费者和利益相关者可以更有效地协作，共同实现为业务创造有价值洞察的目标。虽然实施该方法需要系统性变革、时间和资源，但最终它会使数据和分析更加高效和可靠。

Interestingly, it also opens up additional career opportunities for you as a data engineer. DataOps Engineers are technical professionals that focus on the development and deployment lifecycle rather than the product itself. And as you grow in experience, you can move into more specialist roles within DataOps, contributing to defining the data strategy, developing and deploying business processes, establishing performance metrics, and measuring performance.
有趣的是，它还为您作为数据工程师开辟了额外的职业机会。DataOps 工程师是专注于开发和部署生命周期而非产品本身的技术专业人员。随着经验的增长，您可以在 DataOps 中担任更专业的角色，为定义数据策略、开发和部署业务流程、建立绩效指标以及衡量绩效做出贡献。