Project Details
Challenges
RI EOHHS recognized the need for a more robust and efficient data infrastructure to support data-driven decision-making across its state agencies. RI EOHHS must contend with large volumes of highly-sensitive data, shared into and out of the organization with other state departments, community stakeholders and research organizations. While COVID-19 pandemic response funds helped fund its initial cloud data migration initiatives, RI EOHHS is building a scalable modern data stack that promotes accessibility and governance throughout RI EOHHS.
Solutions
In 2021, the RI EOHHS began exploring the Snowflake Data Cloud with Passerelle, looking to improve performance, reduce server management, and reduce costly AWS queries. RI EOHHS eventually landed on Data Rocket, Passerelle’s modern data stack built on Talend and Snowflake, as a solution to accelerate migration to the Snowflake Data Cloud and build data governance into its data stack.
For Steve Raymond, Database Architect at RI EOHHS, bringing on new technology necessitated a thorough review.
“I feel that technology adoption should always have an eye toward, are we really, truly supporting the goal and mission of our department,” Steve said. “The goal and mission of our department is to generate great analytics, and I only adopt technology that directly supports or enhances our Ecosystem.”
Steve and his team were drawn to Snowflake’s performance and database management.
“Instead of using our brainpower to manage the warehousing space, disk drives and indexes, our brainpower can be better used to understand our data and support analytics.”
RI EOHHS’s transition to Snowflake is supported by Data Rocket, Passerelle’s modern data stack built on Talend and Snowflake. As RI EOHHS migrates data from SQL Server databases into Snowflake, Data Rocket will facilitate the replacement of brittle SSIS connectors with Talend jobs as part of Data Rocket’s Dynamic Ingestion Engine.
One of the first areas RI EOHHS hopes to use Snowflake is with external data shares to research organizations such as the Federal Reserve or Brown University, which is made complex with wieldy data share usage agreements. Today, when RI EOHHS wants to share large datasets, data engineers must back the database and send enormous files via SFTP. Snowflake will make this type of data share easier through Snowflake data sharing that grants access to specific data sets or tables within the RI EOHHS Snowflake Data Cloud to other Snowflake users or accounts.
RI EOHHS will also use Snowflake as the center for its analytics dashboards, which has helped convince other state agencies to join EOHHS’s data estate modernization initiative.
“Having Snowflake has appealed to a lot of other agencies,” Steve said. With the complexities of data management, data sharing and data licensing, a modern analytics platform provides state agencies with the ability to create scalable data governance protocols.
“With Data Rocket, you have the ability to clone a database to create a new virtual warehouse that can be done in seconds versus spinning up a new database in SQL Server and copying the data.”
While RI EOHHS is continuing to ramp up the utilization of Data Rocket, Passerelle’s consultants are working with the state’s technical team to build a unified resident view and assemble facts and dimensions in an AWS data warehouse. The RI EOHHS currently has fifty distinct data sets that can be leveraged for reporting, including vital record statistics, Medicaid enrollment and claims, social services referrals, and program participant outcomes. These data sets are used by the State to create an extensive collection of public-facing reports, available here, that include public health reports, integrated demographic information and snapshots of program efficacy.
Steve said the EOHHS data ecosystem has become the place state agencies go to build reports with data from across the state.
“One of the things that we can do that others can’t do is to integrate data across agencies,” Steve said.
Steve said Passerelle engineers have been critical contributors in the drive toward data estate modernization and in building a roadmap for the future.
“We’ve always been very pleased with the quality of the work, the timeliness, and the professionalism,” he said.
