Here’s a talk the Ternary Data crew did at Machine Learning Utah’s February event last week at Weave (Lehi). We got lots of great audience questions, and we plan to do this talk again in the near future. Enjoy.
It’s January, which means you’ll see a ton of predictions about what’s going to happen in data this year. It’s also 2020, which means a few daring people will take a stab at what the next decade will look like. Predictions are a funny thing – you need to be correct about what and when things will happen. Yearly predictions are mostly trivia and novelty. They’re fun, but kind of miss the point of macro themes that underlay many predictions.
At Ternary, we tend to think in terms of themes, not predictions. Last year, we identified some themes woven across various aspects of the data engineering world. Let’s first look at our themes for 2019, and see how they’re doing. Then we’ll take the bold (some would say foolish) move of laying out some themes for the ‘20s.
A look back at our 2019 themes
Our themes from 2019 were mostly on target. The themes were broadly in a few buckets – data is difficult, technology will continue simplifying and stripping away operational tedium, and privacy and ethics will continue putting checks and balances on AI progress.
- Data will still be hard for most companies – Definitely correct. Data will continue to be difficult. This is a near softball of a theme for data. Despite the press about the looming adoption of AI, we find that most companies are barely doing proper analytics or have robust data architecture. No surprise, we consider the difficulty of data success a continued theme not just in 2020, but also throughout the decade.
- Serverless will gain stronger adoption – 2019 saw advances in serverless such as AWS Lambda EventBridge and Google Cloud Run. Less managed infrastructure is usually a good thing, and IT managers will continue to migrate applications to serverless as it makes sense. We are curious to see where serverless data pipelines and infrastructure goes, as these tend to lag behind serverless application advancements. Especially in light of multi/hybrid cloud, it also makes sense that serverless will be around to keep cloud platforms sticky.
- Containerization will accelerate multi/hybrid cloud – AWS Outposts and Google Cloud Anthos make it clear that clouds realize that customers want options. So, if you can’t beat your competition, make yourself ubiquitous and capitalize where you can.
- DataOps will become more widely adopted – Still in the buzzword stage, but we see DevOps principles becoming more and more common in Data Engineering and Data Science. The AI Manifesto encapsulates our thoughts in this area.
- Data lineage will grow in importance – Anecdotally, this is a growing trend. The market for affordable or open source data lineage tools is still early. We think this market will continue to develop, especially in light of GDPR, CCPA, and a general recognition of data privacy.
- Ethics issues from data science will become a data engineering problem – AI ethics was the big topic of 2019. It’s just getting started. It’s incredibly difficult to have a conversation about AI where ethics isn’t part of the discussion. We think ethical uses of data will not just be a big topic for 2020, but a big topic for the decade as well.
What about 2020?
It’s pretty safe to say our 2019 themes in data will continue being big themes in 2020 as well, and probably beyond.
Here are some additional themes.
- Data Scientists will begin fragmenting into more specific roles and titles. The term ‘data scientist’ is incredibly vague and is losing its meaning. Expect this title to become more meaningful. Since 85%+ of ML models don’t make it to production, it’s going to be harder for companies to justify paying data scientists a lot of money unless they’re seeing results. We think data science will become much more engineering focused out of business necessity.
- Companies will get back to basics with their data. Anecdotally, we see many companies attempting to do AI/ML when they barely have functioning analytics. The challenges and importance of analytics will force companies to scale back their efforts and build a solid data foundation. There will be an analytics renaissance. Companies failing at AI/ML will realize they are suffering from poor analytics, and focus more on building up analytical clout in the form of better data pipelines, warehousing, and business/operational intelligence.
- AutoML will continue supplanting traditional data science practices. Mostly because many problems can be decently solved using off-the-shelf solutions. For harder problems that require domain expertise, data scientists will be forced to focus on the “science” in data science.
- Pipelines, pipelines, pipelines. This is the hard part. What’s new in pipelines is that we can think of numerous interconnected services in the cloud instead of one or a handful of systems.
- Data sharing – within and across companies – will become more popular as companies realize their data is valuable currency.
Our stab at the data themes for the next 10 years…
- All companies will be data companies. It will be impossible to survive without the ability to use data in every process, most of which will be automated.
- Smartphones – and most screens – will be quaint antiques. The internet will simply be…everywhere. 5G (and 6G and beyond) will bring AR/VR and ambient computing to the masses. If you think the smartphone revolution was interesting, you haven’t seen anything yet.
- China vs Western AI – this is the elephant in the room that will shape the role of how AI and data are used in society and business. If you think the ethical AI debates are interesting now, just wait.
- Data transformation will still be difficult for most companies. Although digital transformation came on the scene in the late 1990’s, look at how difficult digital transformation has been for most companies to this day (most fail – cite stat). Data is arguably more difficult, and companies that succeed will see great gains.
- We’ll start seeing the effects of automation on the workforce, and society at large. If you think the recent election was interesting, you haven’t seen anything yet.
- Blockchain and data will converge into a useful utility that will seem incredibly obvious in hindsight. This is the case with nearly every revolutionary advance in technology.
Will we be right? Who knows. A lot can happen between now and 2030 (heck, a lot can happen between now and 2021!). As Bill Gates said, “Most people overestimate what they can do in one year and underestimate what they can do in ten years.”
Last month, the Ternary crew chatted with Great Data Minds about differences between AWS and GCP, as it pertains to data and machine learning. It was a lot of fun, and a lot of great questions were answered. The video interview is finally up. Check it out below.
Did you know that Ternary Data works from a cafe…in a climbing gym?! It’s true. One day after a climbing session at The Front Climbing Club, we camped out in their cafe to knock out a bunch of work. One day became many, and the cafe became our de facto office.
We’re very happy with this arrangement. Instead of going to the gym after work, why not bring the gym to your work? We get to climb – and work out of – one of the world’s best climbing gyms. It’s also a nice place to bring clients, and the coffee and food are excellent.
Thanks to the Front for this write up.
Ternary’s thoughts on digital transformation, and why people matter in the process
Digital transformation is insanely hard. Especially for older companies. Yet, it can be done. And the path probably isn’t what you expect.
The Wall Street Journal has an interesting article about how older companies can digitally transform in one piece, without destroying themselves in the process.
It’s still early days for digital transformation in big companies. Another interesting tidbit – “Big, established companies in particular have been slow to embrace digital organizational practices, a recent McKinsey & Co. study found. Only 26% of world-wide sales are made through digital channels and only 30% of internal operations have been digitally automated, according to the study.”
This echoes what we see in the wild. Most bigger companies are in the middle of changing direction with their digital practices, and it will take many years for this transition to yield change. Again, the biggest barrier to change are the embedded habits and culture. This means CIO/CTOs in big companies need to focus as much or more on getting visible wins that will get the support of laggards within various parts of the company.
Most critically, take time to focus on the culture and the organization. Just as important – don’t mistake new tools and technology for digital transformation. We’ve seen this happen quite often – a hot technology is seen as the cure-all for a business. Several months after introduction, this once hot technology is highly unpopular, and the person who introduced it is either equally unpopular, or already working at another company. Political and organizational capital must be grown carefully and wisely invested.
The punchline – Move Slowly. Solve the biggest pain points first. Don’t try to do everything at once.
If you like this…
Setup your free 30 minute data consultation with Ternary Data. Register here.