diff --git a/TOC-tidb-cloud-lake.md b/TOC-tidb-cloud-lake.md index 5afecca43656e..5526015b4aa2c 100644 --- a/TOC-tidb-cloud-lake.md +++ b/TOC-tidb-cloud-lake.md @@ -64,6 +64,8 @@ - [Load from Bucket](/tidb-cloud-lake/guides/load-from-bucket.md) - [Load from Local File](/tidb-cloud-lake/guides/load-from-local-file.md) - [Load from Remote File](/tidb-cloud-lake/guides/load-from-remote-file.md) + - Load with Platforms + - [Load with dbt](/tidb-cloud-lake/guides/load-with-dbt.md) - Load Semi-structured Data - [Overview](/tidb-cloud-lake/guides/load-semi-structured-data.md) - [Load Parquet](/tidb-cloud-lake/guides/load-parquet.md) diff --git a/tidb-cloud-lake/guides/load-with-dbt.md b/tidb-cloud-lake/guides/load-with-dbt.md index 282be6d2b85b8..b36bce981df65 100644 --- a/tidb-cloud-lake/guides/load-with-dbt.md +++ b/tidb-cloud-lake/guides/load-with-dbt.md @@ -1,13 +1,13 @@ --- -title: dbt +title: Load Data with dbt summary: dbt is a transformation workflow that helps you get more work done while producing higher quality results. You can use dbt to modularize and centralize your analytics code, while also providing your data team with guardrails typically found in software engineering workflows. Collaborate on data models, version them, and test and document your queries before safely deploying them to production, with monitoring and visibility. --- -# dbt +# Load Data with dbt [dbt](https://www.getdbt.com/) is a transformation workflow that helps you get more work done while producing higher quality results. You can use dbt to modularize and centralize your analytics code, while also providing your data team with guardrails typically found in software engineering workflows. Collaborate on data models, version them, and test and document your queries before safely deploying them to production, with monitoring and visibility. -[dbt-databend-cloud](https://github.com/databendcloud/dbt-databend) is a plugin developed by {{{ .lake }}} with the primary goal of enabling smooth integration between dbt and {{{ .lake }}}. By utilizing this plugin, you can seamlessly perform data modeling, transformation, and cleansing tasks using dbt and conveniently load the output into {{{ .lake }}}. The table below illustrates the level of support that the dbt-databend-cloud plugin offers for commonly used features in dbt: +[tidbcloudlake-dbt](https://github.com/tidbcloud/lake-dbt) is a plugin developed by {{{ .lake }}} with the primary goal of enabling smooth integration between dbt and {{{ .lake }}}. By utilizing this plugin, you can seamlessly perform data modeling, transformation, and cleansing tasks using dbt and conveniently load the output into {{{ .lake }}}. The table below illustrates the level of support that the tidbcloudlake-dbt plugin offers for commonly used features in dbt: | Feature | Supported ? | |----------------------------- |----------- | @@ -19,36 +19,38 @@ summary: dbt is a transformation workflow that helps you get more work done whil | Sources | Yes | | Custom Data Tests | Yes | | Docs Generate | Yes | -| Snapshots | No | +| Snapshots | Yes | | Connection Retry | Yes | -## Installing dbt-databend-cloud +## Install tidbcloudlake-dbt -Installing the dbt-databend-cloud plugin has been streamlined for your convenience, as it now includes dbt as a required dependency. To effortlessly set up both dbt and the dbt-databend-cloud plugin, run the following command: +Installing the tidbcloudlake-dbt plugin has been streamlined for your convenience, as it now includes dbt as a required dependency. To effortlessly set up both dbt and the tidbcloudlake-dbt plugin, run the following command: ```shell -pip3 install dbt-databend-cloud +pip3 install tidbcloudlake-dbt ``` However, if you prefer to install dbt separately, you can refer to the official dbt installation guide for detailed instructions. ## Tutorial: Run dbt Project jaffle_shop -If you're new to dbt, {{{ .lake }}} recommends completing the official dbt tutorial available at . Before you start, follow [Installing dbt-databend-cloud](#installing-dbt-databend-cloud) to install dbt and dbt-databend-cloud. +If you're new to dbt, {{{ .lake }}} recommends completing the official dbt tutorial available at . Before you start, follow [Install tidbcloudlake-dbt](#install-tidbcloudlake-dbt) to install dbt and tidbcloudlake-dbt. This tutorial provides a sample dbt project called "jaffle_shop," offering hands-on experience with the dbt tool. By configuring the default global profile (~/.dbt/profiles.yml) with the necessary information to connect to your {{{ .lake }}} instance, the project will generate tables and views defined in the dbt models directly in your {{{ .lake }}} database. Here's an example of the file profiles.yml that connects to a {{{ .lake }}} instance: ```yml title="~/.dbt/profiles.yml" -jaffle_shop_databend: +jaffle_shop_lake: target: dev outputs: dev: - type: databend - host: 127.0.0.1 - port: 8000 + type: tidbcloudlake + host: tnxxxx.gw.aws-us-east-2.default.tidbcloud.com + port: 443 schema: sjh_dbt - user: databend + user: pass: ******** + warehouse: default + secure: true ``` -If you're using {{{ .lake }}}, you can refer to this [Wiki page](https://github.com/databendcloud/dbt-databend/wiki/How-to-use-dbt-with-Databend-Cloud) for step-by-step instructions on how to run the jaffle_shop dbt project. +For more information about configuring and using the adapter, see the [lake-dbt repository](https://github.com/tidbcloud/lake-dbt).