The data landscape evolves at an astonishing pace. New tech, new best practices, and new roles are constantly emerging and changing the way teams create value from data.
Over the last few years, tools and platforms in the data space have been removing the burden of managing and maintaining outdated architectures and codebases. As the technology matures, there’s a constant pattern of “moving up the stack” and bringing data teams closer to business exposure and decisions. And as more businesses invest in leveraging data to make decisions and power products, there’s a real need for technically skilled, business-savvy specialists to create as much value as possible from data.
One recent example is a relatively new role that sits one step “up the stack” from data engineers: the analytics engineer.
What is an analytics engineer?
Analytics engineers are specialists that bridge the gap between highly technical data engineering and outcome-driven analytics. As Michael Kaminsky described the emerging role way back in 2019, “They bring a formal and rigorous software engineering practice to the efforts of analysts and data scientists, and they bring an analytical and business-outcomes mindset to the efforts of data engineering.”
So what does an analytics engineer actually do?
While the job descriptions can vary widely across companies, the core responsibility of an analytics engineer is maintaining analysis-ready data to make a meaningful impact. That usually means a) understanding the business use cases for data; and b) ensuring the right data is ingested, transformed, and available for the relevant analysis to support those use cases.
For analytics engineers, understanding the business needs and transforming the data to meet them are two key steps. As most experienced data teams can tell you, simply connecting raw data sources to BI tools doesn’t get the job done. In fact, it can lead to more confusion and less reliable decision-making because stakeholders end up referencing different datasets and finding different answers to the same questions.
When stakeholders ask what appear to be simple questions (such as What are our best-selling products? Or What’s the average purchase amount on our website?), getting a clear and consistent answer requires both a nuanced understanding of the question and complex data transformations.
“[Analytics engineers are] the ones who can build you a sessions table out of raw clickstream events, or stitch together data from three different payment systems into a single fact table,” Eli Kastelein writes on the Brooklyn Data Co. blog. “It’s easy to overlook the complexity in data transformation, which is why I think there will continue to be a growing demand for analytics engineers. Because when you don’t have your data transformations under control, everything else becomes really really difficult.”
This is how analytics engineers add tremendous value to organizations: ensuring data analysts and business users are accessing the right data in the right places to answer meaningful questions and enable accurate decision-making.
Differences between analytics engineers, data engineers, and data analysts
Since analytics engineers sit between data engineers and data analysts, the lines between these roles can sometimes blur. Here’s a quick breakdown of how they work together, and how their responsibilities are distinct:
- Data engineers are responsible for the infrastructure that supports the ingestion and storage of data. They build and maintain the pipelines that move data, but usually don’t work with the data itself. Data engineers typically use Python and/or Spark to write and deploy code.
- Analytics engineers make sure the data flowing into BI tools or self-serve data products is ready for analysis and decision-making. They deal with both the movement of data and the data itself. Analytics engineers tend to use SQL and/or Python to transform data and deliver automated data models.
- Data analysts focus on deriving insights from the data and reporting out to the business. They usually only work with the data itself, not with the infrastructure. Data analysts typically use SQL to create dashboards and reports.
Analytics engineers work closely with data engineers, but their primary stakeholders are the data analysts and business users they support. And whereas data engineers tend to interact with scrum masters or other developers, analytics engineers need to communicate with business users to understand the use cases and requirements the data needs to meet to facilitate useful analysis. These technically skilled, business-savvy specialists truly bridge the gap between tech teams and business users.
But, keep in mind, the role can look different across companies, especially depending on the size of the data team. For example, if an organization doesn’t have any data engineers, an analytics engineer may be responsible for creating and maintaining data pipelines (which require Python).
As analytics engineer and author Madison Schott writes, “A data engineer at one company could be an analytics engineer at another. A data analyst at one company could be a BI engineer at another.”
When did analytics engineering become a discipline?
If analytics engineering is new to you, you aren’t alone. Kaminsky’s aforementioned January 2019 blog post was one of the first widely read instances of the data community discussing the new role of “analytics engineer”. The role became mainstream as more companies invested in data infrastructure and analytics, and sought to drive the greatest possible value from their data.
In the mid-2010s, cloud-based warehouses, data pipeline services, and BI tools made it much easier to ingest, store, and move raw data. This led to a shift in pipeline management from ETL (extract, transform, load) to ELT (extract, load, transform) — meaning it became much easier to transform raw data after it entered the warehouse, instead of before. And as data engineering platforms introduced automation, transformation logic became less intertwined with infrastructure code. Once solely the domain of highly technical data engineers, transformation logic was now accessible to people closer to the business, who could use write SQL and manipulate data.
But that wasn’t the core skillset (or interest) of the average data analyst or data scientist. And now, transformation didn’t make the best use of the average data engineer’s time, either. There was a gap on modern data teams.
As Kaminsky described it, “While data scientists and analysts are writing a lot of code, being great software engineers isn’t what they’ve been trained for and it often isn’t their first priority. Similarly, while data engineers are great software engineers, they don’t have training in how the data are actually used and so can’t always partner effectively with analysts and data scientists.”
As analytics engineer Keaten Fox told us, the new role “…allowed data engineers to focus solely on creating solutions for ingesting and maintaining source systems and third-party data, while at the same time, analysts gained more capacity to partner with key business partners to help inform and drive critical business decisions.”
How to decide if analytics engineering is right for you
If you’re considering becoming an analytics engineer — or hiring one for your organization — here are a few things to keep in mind.
The more intelligent structure and organization of data meshes have made them a more approachable self-service tool, with speed and quality in mind, where data is domain-oriented, and pipeline ownership is more clearly maintained.
Let’s apply data mesh principles to our Pandora example: one domain stores user-specific data. Another stores programming data. Ads’ data is in yet another domain. And so on. Much more manageable, right? To be clear, this doesn’t mean that data will become fragmented and siloed, which would work against you. Instead, a data mesh sits between a centralized data platform and an excessively fragmented one.
Required skills for analytics engineers
Analytics engineering requires a mix of technical expertise, business acumen, and “soft skills” to create the most value for an organization.
Your technical skills should include:
- Languages: SQL (must-have) and Python (nice-to-have)
- Skills: data modeling (must-have) and data visualization (nice-to-have)
Your business acumen and soft skills should include:
- Communication: The best analytics engineers are strong communicators. They need to possess the business savvy and contextual understanding to develop use cases for data, and collaborate with engineering teams to solve complex pipeline problems. That ability to speak fluently with technical and business stakeholders will make or break an analytics engineer’s ability to succeed.
- Curiosity: Most analytics engineers cite innate curiosity as essential trait that makes them good at their jobs. According to Raphaela Abramson, analytics engineer at Brooklyn Data Co., “You want to know where the data is coming from and why it’s formatted that way, and you also (to the annoyance of your peers) frequently want to tinker with it… A good analytics engineer can think about data from a user’s perspective, an analyst’s perspective, and an engineer’s perspective.”
- Proactive problem solving: Analytics engineers also tend to be self-motivated and proactive problem solvers. Unlike traditional engineering roles, they don’t typically have a scrum master or product owner scoping out work and setting a roadmap. Instead, analytics engineers partner closely with their business teams to identify opportunities to make data more accessible and useful for analytics.
It’s quite common for data engineers with a business mindset or data analysts with technical inclinations to make the switch to analytics engineering. So if you’re considering hiring an analytics engineer, start by looking at your existing teams to see if anyone is already doing this kind of work — or interested in expanding their skills.
The future of analytics engineering
Analytics engineers have quickly become an integral part of modern data teams, and are helping businesses create more value at a faster pace than ever before.
What’s next for analytics engineering? We know that tools will continue to evolve, and new specializations and must-have roles will emerge. But as long as companies continue to invest in creating data products powered by data pipeline automation, the need for specialists who can use data to make a meaningful impact on the business will only increase.