Download: 2023 DataAware Pulse Survey
Pattern 1: Data Teams Are at Capacity
- Persistent Capacity Strain: For four consecutive years, an overwhelming 95% of data teams have reported operating at or above their work capacity. While 57% indicate they’re either somewhat above or significantly over their capacity.
- Disproportionate Burden on Data Engineers: Delving deeper, it’s evident that data engineers are feeling this strain most acutely. They’re three times more likely to report feeling significantly overburdened compared to their counterparts in data architecture and data analytics.
- Hiring Isn’t Catching Up: The gap between organizational demands and team growth is widening, especially for data engineers. They’re 27% more likely than others to report a stark disparity between the growth of their team and the escalating company demands. Simply put, the demand for data is outpacing team growth at an alarming rate.
Pattern 2: Data Teams Are Stuck in Reactive Mode
- Maintenance Over Innovation: Instead of pioneering new solutions, the majority of data engineers, alarmingly, allocate 50% or more of their time to merely maintaining existing programs and infrastructure.
- Reactive Metrics: Data teams are stuck in a reactive mode. They primarily gauge their impact by the number of errors fixed — with 42% saying this metric is their top performance indicator.
- The Engineer’s Dilemma: Taking a closer look, engineers are the most impacted by this trend. 46% cite “number of errors fixed” as the most common metric used to measure their success, underscoring a prevalent culture of squashing bugs rather than creating value.
Pattern 3: Productivity Is Stagnant
- Tool Sprawl: Data teams now use five tools on average in their data stack and 68% plan to maintain or increase this number in the next year. With so many tools to jump between just to do basic tasks, teams are falling further behind. Individual contributors were 2.1x more likely than their managers to say they want to cut tools out of their stack.
- Rising Inefficiencies: Surprisingly, respondents are twice as likely to highlight a decline in their productivity this year when compared to previous years.
- Engineers at the Forefront: Among the roles, data engineers feel the dip most acutely, reporting a 2.2x drop in their productivity year-over-year.
- Tools or Hindrances?: Data engineers, in particular, identify two core issues stymieing their productivity: constraints within the tools they use and a glaring absence of effective automation.
Pattern 4: Data Automation as the Answer
- Surge in Automation Interest: 91% of survey participants are either currently employing, or have intentions of leveraging data automation technologies in the next 12 months.
- Momentum Grows, Yet Challenges Persist: There’s a marked 20% uptick in respondents who are ‘very likely’ to integrate data automation technologies within the coming year. Remarkably, a 110% increase was observed in the current use of these technologies compared to 2022. However, a disparity emerges when considering past year’s projections: 40% professed a strong inclination to adopt automation, yet a mere 5% followed through. This stark difference suggests that, despite the genuine aspiration, there may be a gap in resources or expertise without the aid of supporting automation platforms.
- Top-Down Approach and Disparity: Data team leadership is 74% more inclined to prioritize and fund automation projects. This automation imperative is felt acutely by larger enterprises and teams, with such entities being 1.3 times more likely to experience the setbacks from an automation deficit.
- Engineers and Architects Champion Automation: Data engineers and architects are twice as likely to pivot to automation for augmenting team capabilities, as compared to their data analyst counterparts.
Pattern 5: Managers Bridge the Gap for Data Alignment
- Disparate Perspectives on KPIs: A striking difference emerges when considering top impact measures. Executives gravitate towards tangible outputs, being 3x more inclined to highlight presentations created and 1.5x more drawn to new dashboards as the primary impact measure for their role. Contrastingly, individual contributors prioritize error resolutions and ticket closures.
- Perceptions of Time Efficiency: Executive frustration is palpable, with many feeling basic data tasks are simply taking too long. This sentiment is twice as pronounced among them than the individual contributors on their teams. Further exacerbating their concerns, executives invest an average of 17.8 hours weekly looking for data to do their jobs — 5.6 hours more than individual contributors.
- Strategic Dissonance: Disagreements on major initiatives surface. For instance, individual contributors are five times more likely to dismiss plans to adopt data mesh/fabric, while executives are enthusiastic about its imminent implementation. Skepticism around data mesh benefits is also prevalent among team leads, being 2.8 times more widespread than among executives.
- Tooling Preferences: A tug-of-war ensues over data stack preferences. Individual contributors are up to 2.1x more likely to favor cutting tools. Team leads favor consolidating to platforms while retaining all functions. On the flip side, executives are up to 1.6x more likely to favor adding more tools to the stack.
- AI’s Role: The data team sees high potential for Generative AI. The top use cases include test automation, code generation, and documentation. However, individual contributors are 200% less likely to report believing that Generative AI can increase their impact on the business.
Pattern 6: Expenses Keep Growing
- Proliferation of Data Pipelines: A staggering 89% of respondents anticipate a growth in the number of data pipelines over the next year, underscoring the continual expansion of data infrastructure.
- Cost-Saving Strategies Emerge: As expenses rise, teams are strategizing to curtail them. A prominent 48% of teams are leaning towards optimizing data pipelines as their top tactic to mitigate cloud computing costs, seeking efficiency in operations.
- Consolidation Over Volume: Meanwhile, 38% are setting their sights on consolidating data sources. By reducing data volumes, they aim to achieve a dual goal of streamlining operations and saving costs.
- Divergence in Strategies: Interestingly, while data team executives, at a rate 1.9 times higher than others, feel confident about consolidating data sources, the broader technical team expresses a preference for pipeline optimizations as their cost-control measure.