As a part of the overall feature and functionality updates to the Ascend Platform, we’ve released a number of improvements and features.

:rocket: FEATURES :rocket:
  • MS SQL Server Connection (read + write) uses the newly released MS SQL Spark connector to boost performance
  • Delta Lake write connection
  • New Connection Type called “Ascend System Usage”. Creating this connection in your Ascend environment will allow for accessing three raw data streams: Node, Pod, and Jobs. The “Node” stream is a log of what nodes (instances) exist in the environment at minute granularity. The “Pod” stream is a log of what pods are running on the nodes at minute granularity. The “Job” stream is a log of all the jobs that Ascend has scheduled, when it was scheduled, completed, and the resources consumed. Some customers may have previously had a “DFU stream” enabled by Ascend, which these streams serve to replace in a unified way whilst enabling self-service.

 

:sparkles: ENHANCEMENTS :sparkles:

  • Polish test connection error messages for S3 and GCS
  • Only use whole or half cores for Spark and pin requests and limits
  • Snowflake write connections add a new option to keep column case; this option defaults to convert the column names to upper case per Snowflake’s convention
  • Turn on MySQL rewriteBatchedStatements to improve write performance to MySQL 4 – 40x
  • Date partitions in BigQuery now support column types of Date and DateTime in addition to Timestamp
  • Polish error message when user-supplied schema does not match schema from the source (adding more detail for the nested schema case)
  • Mongo CDC connector can resume from cold start by switching from resume_token to the oplog timestamps
  • Switch MongoDB Read Connector to ingest with the user-defined explicit schema and avoid auto inference from the collection (which can lead to an inconsistent schema)
  • Update Azure blob Go library to v0.13.0
  • Increase limits for image pulls in EKS
  • UI improvements:
    • Autocomplete column names in SQL editor for tables without aliases
    • Improve error message with empty schema on S3 Read Connector
    • Update PySpark template to build transforms more easily
    • Add “Edit Component” button to the context menu (the menu displayed when right-clicking a component or clicking the 3 “dots” in the upper right of a component)
    • Add “Refresh Now” button to the context menu (the menu displayed when right-clicking a component or clicking the 3 “dots” in the upper right of a component)
    • default new transforms to a “Running” state to avoid needing to explicitly select between “Running” and “Paused”
  • Delete parent directories during cleanup in Azure Blob Store
  • SDK: Add in preview_connection function that returns the predicted schema and some sample records for a connection. This can be used to bootstrap the creation of programmatic connections or as part of scripts to re-sync schemas according to their sources.
  • Retry tasks on transient Azure auth errors
  • Enable semantic IDs for connections and credentials. These ID’s now work the same way as Data Service, Dataflow, and Component IDs where re-naming in the UI changes the “slug” format that you can use to refer to these, which also makes it simpler to deploy code from one Data Service to another without having to change references. With this change, existing references to connections / credentials through UUID will continue to work. With the latest SDK (0.2.27), you can now also use these semantic IDs.
  • Hide Requires Credentials checkbox unless Connection Type is selected in Site Admin form
  • Allow an empty dataset to process through a “Repartition” transform as a single partition rather than raising an error that repartitioning is not possible on an empty data set.
  • Order Connection Browser items by name

 

:wrench: BUGFIXES :wrench:

  • Google Sheets Read Connector fixes for handling the “snapshot” configuration
  • Delete Azure blobs in 256 sub batches
  • Stabilize queries from the Ascend UI by downgrading Spark version
  • Fix MySQL non-SSL JDBC connection to not show SSL errors
  • Fix BigQuery Write Connection conflicting user-provided staging credentials with system credentials in cases of Ascend GCP deployments
  • Fix Spark SQL making the page unresponsive when converting “{{” syntax
  • Fix partition display not consistent when components are paused
  • Fix partition switching does not load new limit
  • Fix Intercom button overlapping other UI elements
  • Fix double X to close panel on Create Data Service and Create Data Flow panels
  • Fix Export button in Build/Browse panel
  • Fix intermittent loading of the connection/connector information on existing connectors
  • Improve stability and responsiveness of Ascend API by decreasing network connection hangs (enabling smoother SDK workflows)
  • Fix case in SFTP Read Connection where files and directories would appear in connection browser and then not be ingested
  • Sync Dataflow view in UI across browsers to ensure that deleted components are not shown (allowing both users and browsers to see the same state)

If you need support, have questions, or would like to get started on the platform, schedule time here!