“Enterprisey” Data Engineering Is Now Cool!

Data engineering is becoming more “enterprisey”. This may make you violently cringe. The term “enterprise” conjures up nightmares of faceless committees dressed in overly starched blue shirts and khakis, endless red tape, waterfall development, and the place where innovation goes to die. This image is certainly disturbing, and also not what I’m talking about. When I discuss “enterprisey”, I’m referring to some of the good things that larger companies might do with data — management, operations, governance, and other “boring” stuff. I think data engineering becoming “enterprisey” is a great thing, and welcome it with arms wide open!

Once upon a time, data engineers largely focused on maintaining the lower level details of complex “big data” tools. These tools often had a lot of moving parts, and data engineers didn’t have time for much else except maintenance, fire fighting, and other heroics. As a result, a lot of “enterprisey” things fell by the wayside — data governance, data discovery, data quality, and a slew of other critical data management and operational practices.

Nowadays, data tools are abstracting much of the heavy lifting of “big data” tools. Things that were once complicated, like data pipelines and data lakes/warehouses, are commoditized to the point where they are largely “plug and play”, and “set it and forget it”. Think of companies like AWS, Google Cloud, Azure, Snowflake, Fivetran, and countless others who are simplifying the data stack from end to end. While a data engineer will still engineer systems, the engineering will be focused on creating high-value systems that lead to competitive advantage and differentiation.

Because of widespread data tool abstraction and simplification, a data engineer now has the bandwidth to start working higher up on the value chain — data management, DataOps, among others. While these were practices once reserved for large enterprises, they’re becoming mainstream for companies of all sizes and maturity. Just like there are countless companies simplifying the “big data” stacks of yore, a new crop of best practices, tools and companies are now tackling once “enterprisey” areas such as data governance, data discovery, data quality, and a slew of other critical data management and operational practices. Think Great Expectations (data quality), DataHub (data catalog), and many other projects currently working to solve once ignored problems in data engineering.

With more attention on “enterprisey” problems in data engineering, data engineers will move up the value chain and tackle different types of problems than those from several years ago. I’m excited to imagine the types of problems the next generation of data engineers will be solving in a few years. “Enterprisey” data engineering is now cool. Get used to it.