Data Transformation is essential to transform raw data into a form that can be used by advanced data applications, such as Dataiku ML models, or Flywheel Customer Activation. Raw data from multiple sources can be disorganized and difficult to leverage. Data transformation organizes and prepares the data, creating mature data warehouse models where data can be trusted and used.
Data Rocket now includes a dbt cloud™ integration module that governs and fully automates data processing in the Snowflake Data Cloud. With the Data Rocket dbt Integration Framework, organizations can use Talend and dbt cloud to create data pipelines at scale.
Starting with Talend’s Data Ingestion and Data Quality tools, Data Rocket imports process data from a variety of sources – including flat files, databases, or APIs – while cleaning and preprocessing it to remove errors or inconsistencies.
Once cleansed data is in the persistent layer of the Snowflake Data Cloud, the dbt integration module enables dbt job orchestration using Talend, creating an automated and highly observable data workflow. dbt data transformation jobs can be triggered automatically once the Data Rocket ingestion batch loads are completed.
As part of the Framework, all dbt jobs will be incorporated into Data Rocket’s Audit and Control Dashboard, giving data stewards easy access to critical insights into data throughout its lifecycle. With the Audit and Control Dashboard, Data Rocket augments dbt logs with hourly, daily and monthly run-time descriptions, comparative run times, and data loading trends.
Data Rocket’s dbt Integration Framework eliminates the need for data engineers to manually create workflows from the lake persistent staging layer to the data warehouse, saving time upfront and creating a scalable, predictable model for data transformation.
Interested in learning more? We’d love the chance to hear about how Talend + dbt cloud integration could help you. Contact us today to request set up a call.