Release Notes: 7/28/2021

 FEATURES 

• Kinesis Read Connector
• Nested type support for Read Connectors in the UI
• Input partition configuration for SparkSQL, PySpark, and Scala transforms (available under Advanced Settings). The partition configuration allows for adding in metadata columns as well as filtering down input partitions for non-primary inputs.

 ENHANCEMENTS 

• Improve performance of records preview
• Support shared files in Google Sheets Read Connector
• Jupyter notebook (early access): Improve performance consistency in component DataFrame loading, update interface to limit record count instead of partition count
• Spark SQL (early access): Improve Spark SQL performance by re-using Spark SQL clusters across transforms
• Show detailed schema information in the UI for nested types (Array, Map, Struct)
• Read Connectors: When a binary column is coerced to a string schema type, coerce with hex encoding
• Return reason for spark jobs that fail to start because of unavailable insufficient node capacity
• Replace “Blocked” state with “Waiting” when a component’s upstream is still running schema inference
• Include the error message when losing a Spark worker
• SDK: Downloading Dataflow + Data Service templates sort Dict keys to keep ordering the same between different downloads

 BUGFIXES 

• JDBC Read Connector: Use load table with filter to avoid column name with invalid string
• Correctly populate component names in schema inference errors from Spark SQL Transforms
• Update MongoDB Read Connector to avoid double json encoding of records
• Fix test connection for S3 ACL checks
• Fix hostname lookup for MongoDB Test Connection
• Improve Loki query cache to avoid “Connection Refused” and “Remote End Disconnected” errors in Read Connectors