How DataOps is Transforming How Business Handles Data

You are building a mission-critical big data infrastructure. You have a team of talented software engineers who are dragged into internal meetings with various stakeholders and customers as data and product Subject Matter Experts. You have a team of Data Scientists who are interested in analyzing new data samples but spend a ton of time working out data collection and implementation issues with different internal teams. Your relationship with customers is reactive and they come to you with data volume or quality issues before you are able to identify them. Your Customer Support Team still uses engineering resources and time to help navigate data and a complex product.

If the above issues sound all too familiar to you, keep calm as you’re not the only one in this situation. Many companies face the same challenges, and to help them, Data Operations (DataOps) steps in as the latest agile operations methodology.

Why Data Ops?

Want to design better business strategies based on data-driven decisions? Want to reduce churn by building an awesome recommender engine? Are your engineers or data scientists flooded with work by maintaining projects instead of creating new ones? Customers noticing data outages before your own team?

Data is something that impacts so many different touch points of your business, which is why Data Ops makes life easier for everyone.

It’s About Enabling the Business

DataOps came along not only to avoid customer complaints and to allow other teams to focus on their areas but most importantly to introduce methodologies that favor collaboration between departments, use automation whenever possible, as well as build defined processes and documentation.  

At its heart, the basic idea of DataOps is that it helps bridge the gap between product development, engineers and data science to ensure that maintenance on existing solutions benefits from the same data-driven processes as product development. 

DataOps is an early emerging trend and hasn’t been completely defined yet across various industries where DataOps is present– in fact like most new specialist departments, every organization seems to have their own idea of what DataOps does. But here at BitSight, this is what our team focuses on:

• Data Feed Metrics & Monitoring
• Data Feed Maintenance
• Data-related Customer Support
• Data Partnerships Success

We’re focused on those things because that’s where we see the biggest return for the critical audiences we mentioned above.

Customers know they are getting a robust and reliable product as a dedicated team is assigned to Data Quality Monitoring. They will also be getting more frequent and better product capabilities, due to data being ready to serve Data Scientists and Engineers.

The Product Team will also benefit from this team, as it will be able to focus on innovation and answering customer needs. By having more and better data available, more efficient development processes will ultimately help your company to reduce churn.

DataOps will also enable data democratization, which will allow everyone to visualize data in a simple and intuitive way. This will benefit any member of any team in the company, as they will be able to pull whatever data needed on demand.

For example, if you are a Director or Executive preparing to present your department’s roadmap in the next big board meeting, you will be able to do so independently and with confidence, since you will have easy access to the resources to make data-driven decisions.

Data Feed Metrics & Monitoring

As projects are initiated and developed by numerous people, it is common to have different tools and metrics in place to analyze data before the launch of a given product or feature. According to your goal, you may analyze specific metrics that you will maintain (or not) after the development phase.

In a modern world where databases are huge, the demand for better and bigger data is enormous, and businesses have to make data-driven decisions, having key datasets and metrics spread across the business is no longer sustainable, especially in a high-level industry.

DataOps brings a unified approach, as they will collaborate with different teams to integrate all data metrics into a “built-for-purpose” infrastructure. These database engines can deal with large quantities of data in a quick and sustainable way. Better yet, they also come with reporting and alerting capabilities, so monitoring and visualizing data feeds and custom alerts becomes a piece of cake.

How does the BitSight team do this?

• We have metrics in place for different projects and phases.
• We analyze file presence and size from the Data Partner into our own storage. This helps us assess Data Partners’ performance.
• We take that raw data and run specific metrics that will help determine data quality and coverage. 
• Other specific metrics that are focused on a Product or new feature before its release.
• We report on abnormal behavior patterns. 

BitSight’s Approach to DataOps

Data Feed Maintenance

By standardizing the processes and tools used to manage data, our team can rapidly operationalize data across to accelerate project timelines and increase collaboration within different teams. This leads to faster and better data available to everyone.

• We love automation! Part of this team’s responsibilities is to automate as many steps of the data flow as possible - from the data partner into our own database, from our database to our product. This will increase productivity and produce easier and faster metrics.
• We also love documentation. When automation is not possible or not justified, this team will design its own manual process and write appropriate documentation. 

Data-Related Customer Support

BitSight’s approach to Data Operations may not fit the standard definition of DataOps, but we view that a good thing. Here’s how we do things a little differently:

• We are a customer-facing team. Unlike other DataOps teams, we communicate with customers when necessary and mostly over calls. This happens when other teams such as Customer Support or Customer Success need an extra hand on data-related topics.

• We address the data-related Customer Escalations. Customer appeals that stem from data-related issues are handled by BitSight’s DataOps. We act in the form of a “Tier 4” for our own Customer Support team.
• We identify data-related improvements. BitSight’s DataOps works closely with Product to identify product limitations and other improvements that derive from escalations.

Data Partnerships Success

• We care about our Data Partners. Relationship health is one of the most important priorities and defined goals for BitSight’s Data Operations Team. Regular catch up calls and meetings are taking place to make sure both sides of the relationship are engaged.
• We reach out. For data-related escalations or a possible outage situation, this team is responsible for reaching out to Data Partners when necessary. We also file feature requests for our Partners based on Customer feedback. 

So why would a DataOps team take on data-related escalations aside from other existing responsibilities? 

Having a specialized team that has independent monitoring and metrics in place allows the business to add another layer of customer support to handle escalations. By centralizing the handling of data in one team, DataOps can glean unique insights and proactively identify possible data issues or outages — and also understand the context and impacts of those issues. This not only allows for better support within the product, but frees Customer Support to focus on working with the customer instead of chasing down data.

Conclusion

Nowadays when engineers and data scientists have strict project deadlines and release schedules, they are unable to spend time maintaining existing implementations. Data Operations teams are a new solution which helps address the emerging gap between engineering teams and product in conjunction with product support.

As described above, the most prominent benefits of the DataOps are:

• Engineers do not have to support their data projects and areas of responsibility. They can concentrate on building new solutions.
• Various teams have instant access to data-driven statistics and are able to make decisions faster. 
• Data outages and quality issues are identified faster and addressed more efficiently. 
• Engineering and data project handoff process is well-defined and documented.
• Data Partnership success guarantees healthy vendor relationships and renewals.
• Product Support is enhanced by having access to internal expertise and unique route to subject matter experts.

This post was written by Catarina Neves, Liliana Gomes and Igor Potapov.