Remove default_from_api for google_bigquery_dataset.default_collation for new major version 8.0.0

[wj-chen]

Fixes hashicorp/terraform-provider-google#27384.

This is a copy of #18140 by @BBBmau , but opened against the 8.0.0 feature branch. Below is the original PR description:

---

Fixes a chronic nightly test failure in TestAccBigQueryDataset_collationUpdate (failing ~83% of recent runs on the Google Beta nightly-test branch — 49 failures / 10 passes over the last ~60 days, failing essentially every nightly for weeks).

Test history

TeamCity test history: https://hashicorp.teamcity.com/test/8347509247226202175?currentProjectId=TerraformProviders_GoogleCloud_GOOGLE_BETA_NIGHTLYTESTS&tab=testDetails

Validation build (TeamCity, Upstream MM Testing)

Triggered on refs/heads/auto-pr-18140 (scoped to TestAccBigQueryDataset_collationUpdate):
https://hashicorp.teamcity.com/buildConfiguration/TerraformProviders_GoogleCloud_GOOGLE_BETA_MMUPSTREAMTESTS_GOOGLEBETA_PACKAGE_BIGQUERY/696388

Root cause

The test creates a dataset with default_collation = "und:ci", then updates default_collation back to "" and asserts the value is cleared. It always failed with expected "", got "und:ci".

TeamCity debug logs show no update PUT is ever sent in the clearing step. default_collation was Optional + Computed (via default_from_api: true), and the legacy Terraform Plugin SDK reinterprets an empty-string value on a Computed attribute as "unset":

[DEBUG] A computed value with the empty string as the new value and a non-empty
        old value was found. Interpreting the empty string as "unset" ...
.default_collation: planned value cty.UnknownVal(cty.String) does not match
        config value cty.StringVal("") nor prior value cty.StringVal("und:ci")

So even though customCollationDiff called SetNew("default_collation", ""), the SDK turned it into an unknown/computed value, no diff was produced, no PUT was sent, and the post-apply read returned the unchanged und:ci.

Verified directly against the BigQuery API:

• Creating a dataset without defaultCollation returns it absent (no server-assigned default).
• PUT with "defaultCollation": "" clears the value and an immediate read-back returns "" (the API is consistent here — there was no eventual-consistency problem).

Fix

Remove default_from_api: true from default_collation so the field is Optional (not Computed). The SDK no longer reinterprets the empty string as "unset", a normal diff is produced, and (with send_empty_value: true) the clearing PUT is sent and reflected in state. Because the API returns no defaultCollation when a dataset is created without one, dropping default_from_api does not introduce a permadiff for users who never set the field.

Also remove the now-unnecessary customCollationDiff custom diff: it existed only to force the clearing diff, relies on SetNew (valid only on computed keys), and errors once the field is no longer computed.

Verified by running TestAccBigQueryDataset_collationUpdate against a real project (PASS).

---

Release Note Template for Downstream PRs (will be copied)

See Write release notes for guidance.

bigquery: changed `default_collation` on `google_bigquery_dataset` from computed to optional, allowing collation to be cleared by setting `default_collation = ""`

[github-actions]

Googlers: For automatic test runs see go/terraform-auto-test-runs.

@slevenick, a repository maintainer, has been assigned to review your changes. If you have not received review feedback within 2 business days, please leave a comment on this PR asking them to take a look.

You can help make sure that review is quick by doing a self-review and by running impacted tests locally.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes for commit 267471d:

Diff report

Your PR generated the following diffs in downstream repositories:

Repository	Diff Link	Changes
google provider	View Diff	2 files changed, 6 insertions(+), 22 deletions(-)
google-beta provider	View Diff	2 files changed, 6 insertions(+), 22 deletions(-)
terraform-google-conversion	View Diff	2 files changed, 41 deletions(-)

Breaking Change(s) Detected

The following breaking change(s) were detected within your pull request.

• Field default_collation transitioned from optional+computed to optional google_bigquery_dataset - reference

If you believe this detection to be incorrect please raise the concern with your reviewer.
If you intend to make this change you will need to wait for a major release window.
An override-breaking-change label can be added to allow merging.

Test report

Analytics

Total Tests	Passed	Skipped	Affected
165	155	10	0

Affected Service Packages

• bigquery

Learn how VCR tests work

---

Step 1: Replaying Mode

🟢 All tests passed in Replaying mode! No Recording was needed.

View the replaying VCR build log

@wj-chen, @slevenick, @sachinpro VCR tests complete for 267471d!

[c2thorn]

I think the TGC issue needs a fix in the template. Let's hold off until I can confirm

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes for commit 2bf3909:

Diff report

Your PR generated the following diffs in downstream repositories:

Repository	Diff Link	Changes
google provider	View Diff	2 files changed, 6 insertions(+), 22 deletions(-)
google-beta provider	View Diff	2 files changed, 6 insertions(+), 22 deletions(-)
terraform-google-conversion	View Diff	3 files changed, 47 deletions(-)

Breaking Change(s) Detected

The following breaking change(s) were detected within your pull request.

• Field default_collation transitioned from optional+computed to optional google_bigquery_dataset - reference

If you believe this detection to be incorrect please raise the concern with your reviewer.
If you intend to make this change you will need to wait for a major release window.
An override-breaking-change label can be added to allow merging.

Test report

Analytics

Total Tests	Passed	Skipped	Affected
165	155	10	0

Affected Service Packages

• bigquery

Learn how VCR tests work

---

Step 1: Replaying Mode

🟢 All tests passed in Replaying mode! No Recording was needed.

View the replaying VCR build log

@wj-chen, @slevenick, @sachinpro VCR tests complete for 2bf3909!

[c2thorn]

I think the TGC issue needs a fix in the template. Let's hold off until I can confirm

#18588 should address this

[c2thorn]

/gcbrun

[wj-chen]

I think the TGC issue needs a fix in the template. Let's hold off until I can confirm

#18588 should address this

Thanks, I also made some edits here earlier to try to get around that, let me revert that and test again.

[wj-chen]

I think the TGC issue needs a fix in the template. Let's hold off until I can confirm

#18588 should address this

Thanks, I also made some edits here earlier to try to get around that, let me revert that and test again.

@c2thorn I updated the PR with a new merge from the upstream feature branch but looks like the test is still failing for the same error. Should I bring back my custom_tgc_expand fix or is there more to be done in the template?

[c2thorn]

/gcbrun

[c2thorn]

@c2thorn I updated the PR with a new merge from the upstream feature branch but looks like the test is still failing for the same error. Should I bring back my custom_tgc_expand fix or is there more to be done in the template?

@wj-chen I ran locally just now to verify, and it passed for me with the prior fix. Maybe something happened with the previous run or my local somehow is different.

Either way, it is an error on the TGC side, not with your change. We care about the TPG behavior, which looks correct here. As soon as the build resolves with the git conflict resolution I just pushed, we should be good to merge. And if TGC is still failing, I'll address it separately.

[modular-magician]

Hi there, I'm the Modular magician. I've detected the following information about your changes for commit 0201add:

Diff report

Your PR generated the following diffs in downstream repositories:

Repository	Diff Link	Changes
google provider	View Diff	2 files changed, 6 insertions(+), 22 deletions(-)
google-beta provider	View Diff	2 files changed, 6 insertions(+), 22 deletions(-)
terraform-google-conversion	View Diff	2 files changed, 41 deletions(-)

Breaking Change(s) Detected

The following breaking change(s) were detected within your pull request.

• Field default_collation transitioned from optional+computed to optional google_bigquery_dataset - reference

If you believe this detection to be incorrect please raise the concern with your reviewer.
If you intend to make this change you will need to wait for a major release window.
An override-breaking-change label can be added to allow merging.

Test report

Analytics

Total Tests	Passed	Skipped	Affected
165	155	10	0

Affected Service Packages

• bigquery

Learn how VCR tests work

---

Step 1: Replaying Mode

🟢 All tests passed in Replaying mode! No Recording was needed.

View the replaying VCR build log

@wj-chen, @slevenick, @sachinpro VCR tests complete for 0201add!

[c2thorn]

docs-examples failing test seems to be a transient issue

Error: fatal: unable to access 'https://github.com/GoogleCloudPlatform/magic-modules/': server certificate verification failed. CAfile: none CRLfile: none