DevOps is a set of practices and tools that has revolutionised the software engineering, regardless of choices about on-premise vs. multi-tenant Cloud, the DevOps approach is undeniably demonstrating how quickly one can develop new features and functions, test it and deploy it.
Broadly speaking, DevOps is all about combining software engineering, quality assurance and technology operations. DevOps emerged because traditional systems management wasn’t remotely adequate to meet the needs of modern, agile application development and deployment.
There is a similar need being felt now in the data engineering/data science practices. While the data science kind of a job is looking attractive every passing day, a significant set of activities leading to data science is pure grunt work of just dealing with data – acquire data, cleanse the data, integrate the data, enrich the data, transform the data and so on and so forth. On the other hand, there are business users who want to quickly access data, explore data, validate their hypothesis and so on and so forth. But they are not able to do so because the distinct disciplines of ‘data engineering’ and ‘insight generation’ are at loggerheads with each other.
Together these conflicting and most often siloed working practices are creating “pressure at both ends of the value chain.” From the top of the value chain, more users want access to more data to create more values for their business functions. And from the bottom of the value chain, more data is available than ever before – some structured and well-defined, much of it unstructured and loosely defined.
I believe that it’s time for data engineers and data scientists to embrace a similar new discipline like DevOps – let’s call it “DataOps” — that at its core, it institutionalizes a new approach to managing data that blends engineering and operations in a highly collaborative way, and deliver data from many sources to many users reliably with flexibility and simplicity, so that people focus more on business problem solving than technology, architecture and myriad of tools.
Three compelling trends that are creating the need for DataOps:
1. Analytics-as-a-Service, which is all about giving more individuals access to cutting-edge visualization, data modelling, machine learning, etc in a much more simplified way. This way they can experiment more, ask more questions and go after the unknown-unknowns, solve problems and generate meaning from data in a way that has never before been possible.
2. Data-as-a-Service, which is all about radically improve the performance and accessibility of large quantities of data at unprecedented velocities without bothering about departmental silos and draconian data governance rules/processes.
3. Infra to Insights all-in-one platform, which is all about integrating and orchestrating a platform that can scale, deliver different kind of data engineering/data science workloads, and thus, taking the architecture/platform build-up complexity out of the picture.
Mosaic.DS delivering accelerated DataOps and simplified Data Engineering
It will be wrong to trivialize the specialisation associated with data science, it is an exceptionally important discipline, but data science is only useful insofar, as it can be efficiently and reliably executed. And for that to happen, you need DataOps.
At LTI, we focused on how to make ‘DataOps’ real, and developed an offering that delivers the value ‘Simplified Analytics, Accelerated Outcomes’. The “ops” in DataOps is very intentional. The platform required to support the volume, velocity and variety of data available in the enterprise today is radically different than what traditional data management approaches have assumed. The nature of DataOps embraces the need to manage a variety of data sources and huge number of data pipelines, with a wide variety of transformations. With features like data platform, integrated ML canvas, data discovery, self-service BI and powerful visualization capabilities, MOSAIC.DS provides far reaching capabilities for all data to be used to its full capacity by extracting the ‘goodness’ from every data interaction.
Given the scope of DataOps and aspirational goals, for many it may be tempting to stick to ‘Business As Usual’. In our view, this is not an option. If an organisation is already using Business Intelligence, this will normally drive an appetite for DataOps, because today’s information consumers are hungry to explore more and will demand more value from the data.
The Security Operations Center, or SOC, is often the first solution that comes to the mind…
At the onset of the current Covid-driven pandemic, no one visualized either the timeframe it…
In the era, where there is an exponential increase in a number of breaches per year and decrease…
The world has come a long way from the era of dial up internet connections that offered humble…