Datasets
Import from Data Warehouse
Refuel makes it a breeze to import your data residing in Data Warehouses such as Snowflake, Databricks (more coming soon).
Refuel is committed to keeping your data safe by following industry-standard practices for securing physical deployments, setting access policies, and leveraging the security features of leading Cloud providers.
Generally, the import is a 2-step process:
If you want to automatically run a task on new data as it lands in your Data Warehouse, you can setup a Scheduled Task Run on your dataset.
- Setting up the Data Warehouse Integration
- Telling Refuel the location of the data
Snowflake
Prerequisite: Setup the snowflake integration1
Choose snowflake from the list of Data Sources
2
Specify the table name
Add the name of the table that you want to import in the format schema_name.table_name.The user/role specified during snowflake integration should have read access to the table
3
Setup the dataset
Select the set of primary keys that can uniquely identify a row in the provided table. You also need to select
a timestamp column that will act as a checkpoint so we only read new data as it arrives in your Snowflake table.If your table does not have a primary key or timestamp column, please drop us a message.
Databricks
Prerequisite: Setup the databricks integration1
Choose databricks from the list of Data Sources
2
Specify the table name
Add the name of the table that you want to import in the format schema_name.table_name.
3
Setup the dataset
Select the set of primary keys that can uniquely identify a row in the provided table. You also need to select
a timestamp column that will act as a checkpoint so we only read new data as it arrives in your Databricks table.If your table does not have a primary key or timestamp column, please drop us a message.
