Data Engineering, Unified
More Architecting, Less Plumbing.
The Ascend Unified Data Engineering Platform gives data teams 10x faster build velocity and automated maintenance for modern data pipelines. Generate autonomous data pipelines that dynamically adapt to any changes in data, code or environment. Evolve beyond traditional ETL and data orchestration tools to ingest, build, run, integrate, and govern advanced data pipelines with 95% less code.
Ascend’s DataAware™ intelligence observes and maintains detailed records of data movement and processing, code changes and user activity, enabling data pipelines to run at optimal efficiency with integrated lineage tracking, auditability and governance.
Deploy Ascend on top of existing Apache Spark clusters, or as a fully managed solution.
One platform to get from prototype to production
Ascend fits easily into your existing data ecosystem, whether you are just starting your pipeline journey, or have thousands already running. Quickly connect to any system to start building. Autonomous data pipelines directly fuel downstream applications, analytics, and machine learning models.
ASCEND INGEST
Tap into your data sources with little to no code from any data lake, warehouse, database, stream or API, simply by describing the inputs. Ascend automatically monitors for new data, format conversions, data profiling, and incremental processing.
Any Data, Anywhere, Any Format
Automated Change Detection
Detect and ingest new, updated, and deleted data automatically, and efficiently. Ascend’s DataAware keeps tabs on your data connectors, keeping track of not only where your data is located, but how often it changes, as well as what has already been ingested.
Automated Data Profiling
Automatically profile every piece of data. Analyze the mins and maxes of every column, of every partition of data. See how values change over time, and keep an eye on things like data anomalies.
Automated Data Reformatting
Not all data comes in ready for big data processing. Whether it is too many small files that need to be aggregated, too few of large files that should be partitioned, or simply GZIP’ed CSVs that should be converted to Snappy compressed Parquet files, we have you covered.
Declarative Data Pipelines
Design your pipelines with declarative definitions that require 95% less code and result in far less maintenance. Specify inputs, outputs, and data logic in Python, SQL, and YAML specs.
Interactive Pipeline Builder
Visually navigate your live data pipelines, from source to sink, and everything in between. Trace lineage, preview data, and prototype changes in minutes instead of days.
Queryable Pipelines
Treat any stage, of any data pipeline, as a queryable table. Quickly prototype new pipeline stages, or run ad-hoc queries against existing pipeline stages, all in a matter of seconds. Ascend’s DataAware will notify you when the underlying data has changed, and makes converting your queries to pipelines in just a few clicks.
Git & CICD Integration
Connect your data engineering platform to your favorite CICD tool, from Jenkins to CircleCI. Ascend’s Declarative Pipelines makes it easy to track changes, and Ascend’s DataAware capabilities make pipeline replication, roll-forward, and roll-back safe, fast, and efficient.
ASCEND BUILD
Architect data pipelines using 95% less code with Declarative Pipeline Definitions written in Python, SQL and YAML. Ascend’s Interactive Pipeline Builder visually connects code and data with data exploration, rapid prototyping, and lineage navigation.
ASCEND RUN
Leveraging the Autonomous Pipeline Engine, data teams can optimize data pipelines and infrastructure with Adaptive Spark Cluster Management, dynamic resourcing, proactive data persistence, and automatic backfills.
Intelligent Persistence
Thanks to Ascend’s DataAware intelligence, incremental data processing can be safely stored in your private object store, providing fast access to incremental processing stages, and ensuring you never have to repeat yourself.
Data & Job Deduplication
Never run the same piece of code on the same piece of data again. Ascend’s DataAware intelligence ensures that all work across pipelines is safely deduplicated, ensuring your pipelines run fast, efficiently, and cost effectively. Branching and merging data pipelines is finally as easy as it is with code.
Dynamic Partitioning
Whether you’re performing ETL-style map operations, aggregations, or complex window functions, Ascend has you covered. Our DataAware parses all of your code, compares it with upstream data profiles, dynamically partitions your data to optimize propagation of incremental changes in data.
Automated Backfill
Find that bug in your pipeline that was producing problematic data for the past month, and now need to deploy the fix and backfill old calculations? Ascend’s DataAware automates the entire process, detecting code changes, tracing the lineage of your previous version, and updating everything for you, automatically.
Automated Spark Management
Ascend dynamically creates, manages, and deletes your Spark clusters based on workload, optimizing resource efficiency while enforcing appropriate security boundaries. Ascend also optimizes Spark parameters for every job, based on data and code profiles.
Notebook Connectors
Connect Jupyter, Zeppelin, and more directly to Ascend for fast, efficient access to data as it moves through your data pipelines.
BI & Data Visualization
Feed data directly from Ascend to your BI and Data Visualization tools via Ascend’s High Performance Records API.
Structured Data Lake
Get direct access to Ascend’s internal storage (.snappy.parquet) files for efficient processing by other big data systems. Ascend’s SDL offers fully transaction reads across multiple files, and guarantees that the data available is always linked directly to an active data pipeline.
Record APIs & SDKs
Read records from any stage of any data pipeline via Ascend’s high throughput records API. Connect Ascend directly to your applications, BI, and Viz tools with one easy API.
ASCEND INTEGRATE
Connect to data pipelines at any stage of the data lifecycle. Ascend’s Structured Data Lake enables notebooks and big data tools to directly access pipeline optimized files, while Ascend’s connectors ensure data connectivity and consistency to any downstream data system.
ASCEND GOVERN
With a suite of tracking, reporting, and security capabilities, Ascend records and permanently maintains an in-depth understanding of the linkages between code, data and users, with a level of visibility and auditability never before possible.
Automated Cataloging
Ascend provides organized and searchable access to all code and data under the platform’s management, with automated registration of new data sets and code.
Data Lineage Tracking
For any piece of data managed within the Ascend platform, data teams can track down to a column level where it came from, the entire series of code that ran on it, and where it went.
Resource and Cost Reporting
For every piece of data in the system, Ascend reports the resources required, historically and at present, to produce and maintain it.
Activity Monitoring and Reporting
Tracking of all user, data, and system events, with integration into external platforms such as Splunk and PagerDuty.
Secure Data Feeds
The safest and easiest way to connect data pipelines across organizational and security boundaries to limit data access while preserving metadata lineage and integrity.
Data Garbage Collection
Data teams can now have Ascend crawl data storage systems, automatically deleting data that has been abandoned and is no longer associated with active data pipelines.