Earlier this year, in the LinkedIn 2020 Emerging Jobs Report, the role of the data engineer was listed as number eight in a ranking of emerging jobs in the U.S. — with a 33% annual hiring growth rate. More recently, the Robert Half Technology’s Salary Guide ranked data engineering as the highest-paying IT job in 2021.
According to Robert Half, data engineering talent will continue to be compensated well above average as organizations will heavily rely on “individuals who can transform large amounts of raw data into actionable information for strategy-setting, decision making and innovation.”
As the demand for data engineers increases, data teams must be prepared with the right tools and resources to ensure the role maintains a productive and effective workflow — instead of spending a majority of their time on maintenance of existing systems, as cited by a recent survey.
It’s important to acknowledge that the data engineering space is evolving quickly — there are new open-source projects and tools released all the time, “best practices” are constantly changing, and data engineers are stretched thinner than ever before as businesses increase their data demands. Not only that, but data engineers have recently seen firsthand the hybridization of the data lake and data warehouse, with modern warehouses blurring the distinction by separating out compute from storage. This hybridization has evolved basic “ETL” pipelines of the past into more complex orchestrations, oftentimes both reading from and writing to warehouses. Additionally, as data scientist and data analyst colleagues are operationalizing their work, data engineers need to work more collaboratively with these functions and further empower these roles. With all of this pressure and constant change, it can be difficult to keep up with the space (or even enter it!).
From podcasts to blogs, below is a roundup of resources — both for new data engineers and seasoned professionals — looking to stay up to date with the latest in the world of data engineering.
The Data Engineering Podcast
It’s hard to beat the Data Engineering Podcast for relevancy in the space. With a new episode each week, the pace is easy to follow. Keeping up with the subject matter, however, is another thing. Topics range from companies discussing their data architecture and technical challenges to deep dives into open-source and paid offerings.
I’ve highlighted a few of the fascinating episodes below:
Presto Distributed SQL Engine: We’ve gotten so used to aggregating data to get it ready for querying, it’s illuminating to think about the pattern from the other side — federating the query execution to query/combine data where it resides. Hearing the pros and cons of this approach helps to add another tool to a data engineer’s toolbox.
Jepsen: The legendary Jepsen project has been diving deep on how distributed systems were designed, built, and perhaps most importantly, how they deal with failure conditions. Although deeper in the stack than a normal day-to-day stack, getting more familiar with the distributed systems that we rely on and are building (as we tie together multiple disparate systems) is a boon.
Non-Profit Data Professional: The guest on this episode, a director of data infrastructure at a non-profit, shares how challenging it can be to work with a multitude of data sources and compliance changes all while on a strict budget — something many data professionals at today’s startups (or non-profits in this case) can relate to and learn something from.
Subscribing to this podcast can be a great way to dedicate some time each week to stay ahead of the curve.
Big Data at InfoQ
Maybe podcasts aren’t so much your style, or maybe you crave more than one a week. If you’re looking for more resources (in multiple formats including podcasts, articles, presentations, and more), InfoQ’s Big Data section can be a great place to dig in further. The company has been making a name for itself by creating an editorial community with engineers and practitioners as opposed to journalists, and the content reflects this choice. The Big Data section covers a wide range of topics including AI, machine learning, and data engineering. While it’s not entirely focused on data engineering, it’s an informative way to learn more about adjacent spaces.
Awesome Big Data
I’m a big fan of the software community’s trend of creating GitHub repositories with curated lists of resources focused on a specific area (aka the awesome list). In my Clojure days, awesome-clojure was a great resource for looking up different database adapters or linters. Luckily, there is an “awesome” for Big Data, which includes subsections for Data Engineering and Public Datasets (those always seem to come in handy). The curation seemed a bit tighter before the project exploded in popularity; fret not, even having a giant list of public data sets — in, for example, the energy space — is incredibly valuable.
The data engineering space is moving quickly, whether measured by hiring growth rate, business needs, or toolset evolution. The resources available are fortunately keeping pace. Although it’s easy to get lost in the day-to-day grind of projects, I’ve been able to rekindle enthusiasm and keep up with the space by taking a step back to learn about newer paradigms/technologies. With this list of resources, I hope you all find the jumping-off point to do the same!
Share on facebookShare on twitterShare on linkedinShare on redditShare on email
def fetch_commit_history(
repos: Union[str, List[str], pathlib.Path],
timeout_seconds: int = 120,
since_date: Optional[str] = None,
from_ref: Optional[str] = None,
to_ref: Optional[str] = None,
) -> Dict[str, List[Dict[str, Any]]]:
"""
Fetches commit history from one or multiple GitHub repositories using the GitHub CLI.
Works with both public and private repositories, provided the authenticated user has access.
"""
# Check GitHub CLI is installed
subprocess.run(
["gh", "--version"],
capture_output=True,
check=True,
timeout=timeout_seconds,
)
# Process the repos input to handle various formats
if isinstance(repos, pathlib.Path) or (isinstance(repos, str) and os.path.exists(repos) and repos.endswith(".json")):
with open(repos, "r") as f:
repos = json.load(f)
elif isinstance(repos, str):
repos = [repo.strip() for repo in repos.split(",")]
results = {}
for repo in repos:
# Get repository info and default branch
default_branch_cmd = subprocess.run(
["gh", "api", f"/repos/{repo}"],
capture_output=True,
text=True,
check=True,
timeout=timeout_seconds,
)
repo_info = json.loads(default_branch_cmd.stdout)
default_branch = repo_info.get("default_branch", "main")
# Build API query with parameters
api_path = f"/repos/{repo}/commits"
query_params = ["per_page=100"]
if since_date:
query_params.append(f"since={since_date}T00:00:00Z")
target_ref = to_ref or default_branch
query_params.append(f"sha={target_ref}")
api_url = f"{api_path}?{'&'.join(query_params)}"
# Fetch commits using GitHub CLI
result = subprocess.run(
["gh", "api", api_url],
capture_output=True,
text=True,
check=True,
timeout=timeout_seconds,
)
commits = json.loads(result.stdout)
results[repo] = commits
return results
Key implementation details:
- GitHub CLI integration: Uses the `
gh` command-line tool for authenticated API access to both public and private repositories
- Flexible input handling: Accepts single repos, comma-separated lists, or JSON files containing repository lists
- Robust error handling: Validates GitHub CLI installation and repository access before attempting to fetch commits
- Configurable date filtering: Supports both date-based and ref-based commit filtering
AI-Powered Summarization
def summarize_text(content: str, api_key: Optional[str] = None) -> str:
"""
Summarize provided text content (e.g., commit messages) using OpenAI API.
"""
if not content.strip():
return "No commit data found to summarize"
# Get API key from parameter or environment
api_key = api_key or os.getenv("OPENAI_API_KEY")
if not api_key:
raise RuntimeError("OpenAI API key not found. Set the OPENAI_API_KEY environment variable.")
client = OpenAI(api_key=api_key)
response = client.chat.completions.create(
model="gpt-4o",
messages=[
{"role": "system", "content": SYSTEM_PROMPT},
{"role": "user", "content": content},
],
temperature=0.1,
max_tokens=1000,
)
return response.choices[0].message.content.strip()
def summarize_commits(content: str, add_date_header: bool = True) -> str:
"""
Summarize commit content and optionally add a date header.
"""
summary_body = summarize_text(content)
if add_date_header:
# Add header with week date
now_iso = datetime.utcnow().strftime("%Y-%m-%dT%H:%M:%SZ")
monday = get_monday_of_week(now_iso)
return f"## 🗓️ Week of {monday}\n\n{summary_body}"
return summary_body
Our initial system prompt for consistent categorization:
You are a commit message organizer. Analyze the commit messages and organize them into a clear summary.
Group similar commits and format as bullet points under these categories:
- 🚀 Features
- ⚠️ Breaking changes
- 🌟 Improvements
- 🛠️ Bug fixes
- 📝 Additional changes
...
Within the Improvements section, do not simply say "Improved X" or "Fixed Y" or "Added Z" or "Removed W".
Instead, provide a more detailed and user-relevant description of the improvement or fix.
Convert technical commit messages to user-friendly descriptions and remove PR numbers and other technical IDs.
Focus on changes that would be relevant to users and skip internal technical changes.
Format specifications:
- Format entries as bullet points: "- [Feature description]"
- Use clear, user-friendly language while preserving technical terms
- For each item, convert technical commit messages to user-friendly descriptions:
- "add line" → "New line functionality has been added"
- "fix css overflow" → "CSS overflow issue has been fixed"
- Capitalize Ascend-specific terms in bullet points such as "Components"
Strictly exclude the following from your output:
- Any mentions of branches (main, master, develop, feature, etc.)
- Any mentions of AI rules such as "Added the ability to specify keywords for rules"
- Any references to branch integration or merges
- Any language about "added to branch" or "integrated into branch"
- Dependency upgrades and version bumps
…
Prompt engineering:
- Structured categorization: Our prompt enforces specific emoji-categorized sections for consistent output formatting
- User-focused translation: Explicitly instructs the AI to convert technical commits into user-friendly language
- Content filtering: Automatically excludes dependency updates, test changes, and internal technical modifications
- Low temperature setting: Uses 0.1 temperature for consistent, factual output rather than creative interpretation
Content Integration and File Management
def get_monday_of_week(date_str: str) -> str:
"""
Get the Monday of the week containing the given date, except for Sunday which returns the next Monday.
"""
date = datetime.strptime(date_str, "%Y-%m-%dT%H:%M:%SZ")
# For Sunday (weekday 6), get the following Monday
if date.weekday() == 6: # Sunday
days_ahead = 1
else: # For all other days, get the Monday of the current week
days_behind = date.weekday()
days_ahead = -days_behind
target_monday = date + timedelta(days=days_ahead)
return target_monday.strftime("%Y-%m-%d")
File handling considerations:
- Consistent date formatting: Automatically calculates the Monday of the current week for consistent release note headers
- Encoding safety: Properly handles Unicode characters in commit messages from international contributors
- Atomic file operations: Uses temporary files during processing to prevent corruption if the process is interrupted
GitHub Actions: Orchestrating the Automation
Our workflow ties everything together with robust automation that handles the complexities of CI/CD environments.
Workflow Triggers and Inputs
name: Weekly Release Notes Update
on:
workflow_dispatch:
inputs:
year:
description: 'Year (YYYY) of date to start collecting releases from'
default: '2025'
month:
description: 'Month (MM) of date to start collecting releases from'
default: '01'
day:
description: 'Day (DD) of date to start collecting releases from'
default: '01'
repo_filters:
description: 'JSON string defining filters for specific repos'
required: false
timeout_seconds:
description: 'Timeout in seconds for API calls'
default: '45'
Flexible triggering options:
- Manual dispatch with granular date control: Separate year, month, day inputs for precise date filtering
- Repository-specific filtering: JSON configuration allows different filtering strategies per repository
- Configurable timeouts: Adjustable API timeout settings for different network conditions
Secure Authentication Flow
- uses: actions/create-github-app-token@v2
id: app-token
with:
app-id: <YOUR-APP-ID>
private-key: ${{ secrets.GHA_DOCS_PRIVATE_KEY }}
owner: ascend-io
repositories: ascend-docs,ascend-core,ascend-ui,ascend-backend
Security best practices:
- GitHub App with specific repository access: Explicitly lists only the repositories that need access
- Scoped permissions: App configured with minimal necessary permissions for the specific repositories
- Secret management: Private key stored securely in GitHub Secrets
Repository Configuration Processing
- name: Prepare repository filter configuration
run: |
CONFIG_FILE=$(mktemp)
echo "{}" > "$CONFIG_FILE"
if [ -n "${{ github.event.inputs.repo_filters }}" ]; then
echo '${{ github.event.inputs.repo_filters }}' > "$CONFIG_FILE"
else
DATE_STRING="${{ github.event.inputs.year }}-${{ github.event.inputs.month }}-${{ github.event.inputs.day }}"
jq -r '.[]' bin/release_notes/input_repos.json | while read -r REPO; do
FILTER="since:$DATE_STRING"
jq --arg repo "$REPO" --arg filter "$FILTER" '. + {($repo): $filter}' "$CONFIG_FILE" > "${CONFIG_FILE}.tmp" && mv "${CONFIG_FILE}.tmp" "$CONFIG_FILE"
done
fi
CONFIG_JSON=$(cat "$CONFIG_FILE")
echo "config_json<<EOF" >> $GITHUB_OUTPUT
echo "$CONFIG_JSON" >> $GITHUB_OUTPUT
echo "EOF" >> $GITHUB_OUTPUT
Data Processing and File Management
- name: Generate release notes
env:
OPENAI_API_KEY: ${{ secrets.OPENAI_API_KEY }}
GITHUB_TOKEN: ${{ steps.app-token.outputs.token }}
run: |
CONFIG_JSON='${{ steps.repo_config.outputs.config_json }}'
CONFIG_FILE=$(mktemp)
echo "$CONFIG_JSON" > "$CONFIG_FILE"
RAW_OUTPUT=$(python bin/release_notes/generate_release_notes.py \
--repo-config-string "$(cat "$CONFIG_FILE")" \
--timeout "${{ github.event.inputs.timeout_seconds }}")
# Split summary and commits using delimiter
SUMMARY=$(echo "$RAW_OUTPUT" | sed -n '1,/^### END SUMMARY ###$/p' | sed '$d')
MONDAY_DATE=$(echo "$SUMMARY" | head -n1 | grep -oE "[0-9]{4}-[0-9]{2}-[0-9]{2}")
echo "monday_date=$MONDAY_DATE" >> $GITHUB_OUTPUT
echo 'summary<<EOF' >> $GITHUB_OUTPUT
echo "$SUMMARY" >> $GITHUB_OUTPUT
echo 'EOF' >> $GITHUB_OUTPUT
Key implementation lessons:
- Temporary file strategy: We learned the hard way that GitHub Actions environments can lose data between steps. Writing to temporary files solved reliability issues where data would appear blank in subsequent steps.
- Complex JSON handling: Uses `
jq` for safe JSON manipulation and temporary files to avoid shell quoting issues with complex JSON strings
- Output parsing: Logic to split AI-generated summaries from raw commit data using delimiter markers
- Robust error handling: `
set -euo pipefail` ensures the script fails fast on any error, preventing silent failures
File Integration and Pull Request Creation
- name: Update whats-new.mdx with release notes
run: |
FILE="website/docs/whats-new.mdx"
BRANCH_NAME="notes-${{ steps.generate_notes.outputs.monday_date }}"
git branch $BRANCH_NAME main
git switch $BRANCH_NAME
TEMP_SUMMARY_FILE=$(mktemp)
echo '${{ steps.generate_notes.outputs.summary }}' > "$TEMP_SUMMARY_FILE"
cat "$TEMP_SUMMARY_FILE" "$FILE" > "${FILE}.new"
mv "${FILE}.new" "$FILE"
rm -f "$TEMP_SUMMARY_FILE"
File management features:
- Atomic file operations: Uses temporary files and atomic moves to prevent file corruption
- Branch management: Creates date-based branches for organized PR tracking
- Content preservation: Carefully prepends new content while preserving existing documentation structure
Lessons Learned and Best Practices
Building this pipeline taught us valuable lessons about documentation automation that go beyond the technical implementation.
Technical Insights
File persistence matters in CI/CD environments. GitHub Actions environments can be unpredictable—always write important data to files rather than relying on environment variables or memory. We learned this the hard way when release notes would mysteriously appear blank in PRs.
API reliability requires defensive programming. Build retry logic and fallbacks for external API calls (OpenAI, GitHub). Network issues and rate limits are inevitable, especially as your usage scales.
Prompt engineering is crucial for consistent output. Spend time crafting prompts that consistently produce the format and tone you want. Small changes in wording can dramatically affect AI output quality and consistency.
Human review is essential, even with AI generation. Having team members review PRs catches edge cases, ensures quality, and builds confidence in the automated system. The goal isn't to eliminate human oversight—it's to make it more efficient and focused.
Historical tracking and product evolution insights. Automated generation creates a consistent record of product evolution that's valuable for retrospectives, planning, and onboarding new team members.
Results and Impact
The automation has fundamentally transformed our release process and team dynamics:
Quantifiable Improvements
Dramatic time savings: Reduced release note creation from 2-3 hours of writing time to 15 minutes of review time. That's a 90% reduction in effort while improving quality and consistency.
Perfect consistency: Every release now has properly formatted, comprehensive notes. No more missed releases or inconsistent formatting across different team members.
Increased frequency: We can now generate release notes weekly, providing users with more timely updates about product improvements.
Complete coverage: Captures changes across all repositories without manual coordination, eliminating the risk of missing important updates.
Next Steps and Future Enhancements
We're continuously improving the pipeline based on team feedback and evolving needs:
Immediate Roadmap
Slack integration: Building a Slackbot to automatically share release notes with our community channels, extending the reach beyond just documentation updates.
Repository tracing: Categorize the raw commits by repository and add links so it's easy to (literally) double-click into each PR for additional context.
Future Possibilities
Multi-language support: Generating release notes in different languages for global audiences as we expand internationally.
Ready to automate your own release notes? Start with the requirements above and build incrementally. Begin with a single repository, get the basic workflow running, then expand to multiple repos and add advanced features. Your future self (and your team) will thank you for eliminating this manual drudgery and creating a more consistent, professional release process.
The investment in automation pays dividends immediately—not just in time saved, but in the improved quality and consistency of your user communication. In a time where software moves fast, automated release notes ensure your documentation keeps pace.
.png)
.png)
.png)
