How To Hire A Data Engineer In A Hot Job Market

On a weekly basis, we field questions from a variety of companies asking us to link them with data engineering talent. Our typical reply - “maybe”. The reality is the supply of data engineers is insanely tight nowadays. The job market for data talent is red hot right now*.

Ternary’s Joe Reis even posted about this on LinkedIn last week, and got quite a bit of feedback.

What can you do to increase your odds of hiring a data engineer in a hot job market? This is really a question of “build vs buy”, meaning you can choose to cultivate data talent from within your company (build), or search externally for data talent (buy). Let’s walk through the pros and cons of each approach.

Option 1 - Build - Hire internally and build data engineering skills

Pros

• The people being trained have existing domain expertise with your company.

• You understand the reasons why you’re hiring, and can pick people who you trust to level up their skills.

• This is potentially easier and cheaper than finding an external hire.

Cons

• Backfilling the position may be challenging, especially if you’re training a software engineer to become a data engineer. The job market is equally hot for software engineers.

• Watch out for bad patterns inherited from other disciplines. This is a matter of training and experience.

Option 2 - Buy - Hire externally (employees or contractors) 

Pros

• You can build a team for the long term.

• In the long run, employees are way cheaper than contractors.

Cons

• Employees are hard to find.

• Contractors are expensive.

Considerations with both approaches

• Why not try both build AND buy? Consider hiring experienced data people who can lead internal training and team building.

• Relentlessly automate. As you’re building your team’s skills, find ways to automate mundane data engineering tasks. Focus your team on mission critical tasks that are core to your business objectives.

• Avoid undifferentiated heavy lifting. If a vendor has a tool that accomplishes a task, consider investing in this tool instead of building it yourself. Build when you encounter problems that are unique to your business.

Here are some ways to make your company more attractive to external data talent

• Build your network of candidates before hiring. Be active in your local tech community, especially in the data domain. Host and attend meetups, and have your team give talks at tech events and conferences. Focus on making contributions.

• Positive Glassdoor ratings are essential. Like it or not, Glassdoor is here to stay. Make sure your company is accurately and positively portrayed. If you see an alarming number of negative reviews from current or former employees, you need to ask some hard questions.

• Keep your reputation stellar within the data community. The data world is incredibly small.

• Set clear priorities, map out initiatives and establish expectations. Treating machine learning like a magic bullet is likely to lead to failed projects and dashed hopes. What do you actually intend to achieve with data? What is the value for your business?

• Internally, keep data front and center. Keep momentum going to build data literacy within your organization. A track record of wins helps convince execs to provide more resources for hiring.

Good luck building your data team!

* A quick note - We’re still looking for concrete data about the 2021 hiring market for data engineers. The most recent data we can find (so far) is from early 2020, which is when COVID landed like an asteroid on the job market. If anything, we noticed a pickup in demand for cloud and data related work (we’re data consultants). Also, we run a few tech meetups, and run a “who’s hiring, who’s looking” at the start of our meetup. Consistently, several companies are hiring, and nobody is looking for work. Based on the calls we’re getting, we expect that many companies are searching for data talent, often to little avail.