adding files related to NOA GFS poc by balit-raibot · Pull Request #1824 · datacommonsorg/data

balit-raibot · 2026-01-04T18:12:13Z

Setting up auto-refresh of NOAA GFS Data for Data Commons ingestion.

gemini-code-assist · 2026-01-04T18:12:50Z

Summary of Changes

Hello @balit-raibot, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request establishes the foundational data and configuration for a Proof of Concept (PoC) to integrate NOAA Global Forecast System (GFS) data into the system. It includes the raw GFS data, the necessary metadata and mapping files to process this data, and the resulting transformed output, along with the corresponding statistical variable definitions and schema.

Highlights

Raw GFS Data Addition: Added a CSV file containing raw NOAA Global Forecast System (GFS) meteorological data for December 24, 2025, covering various atmospheric and surface parameters.
Data Processing Configuration: Introduced metadata and parameter-value mapping files (noa_gfs_metadata.csv and noa_gfs_pvmap.csv) essential for processing and transforming the raw GFS data.
Transformed Output Data: Generated a CSV file (noa_gfs_output.csv) with the GFS data transformed and mapped to Data Commons statistical variables and properties.
Statistical Variable Definitions: Included MCF files (noa_gfs_output_stat_vars.mcf and noa_gfs_output_stat_vars_schema.mcf) that define the new statistical variables and their corresponding schema for the GFS dataset within the Data Commons framework.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request adds new files for importing NOAA GFS data as a proof-of-concept for Data Commons. While this is a valuable addition, I've identified several critical and high-severity issues in the generated files and configurations that should be addressed to ensure the data is imported correctly and the process is robust. Key issues include hardcoded values leading to data precision loss, incomplete and fragile TMCF definitions, malformed data in the output CSV, and stylistic issues in the MCF files.

statvar_imports/noa_gfs/noa_gfs_output.tmcf

statvar_imports/noa_gfs/noa_gfs_pvmap.csv

statvar_imports/noa_gfs/noa_gfs_output.csv

statvar_imports/noaa_gfs/test_data/noaa_gfs_output_stat_vars.mcf

statvar_imports/noaa_gfs/test_data/noaa_gfs_output_stat_vars_schema.mcf

statvar_imports/noa_gfs/noa_gfs_pvmap.csv

statvar_imports/noaa_gfs/test_data/noaa_gfs_output.csv

…to noa_gfs_poc

statvar_imports/noaa_gfs/custom_script_final.py

statvar_imports/noaa_gfs/custom_statvar_processor.py

statvar_imports/noaa_gfs/custom_script_final.py

…to noa_gfs_poc

ajaits · 2026-04-01T13:41:28Z

statvar_imports/noaa_gfs/run.sh

+
+# --- Updated 1. Runtime Installation ---
+# Added libaec-dev to the list
+$SUDO apt-get update && $SUDO apt-get install -y \


please verify if apt-get can run in auto-refresh setup on cloud batch jobs.

curl is already present. For other tools, consider adding it to the docker image:

data/import-automation/executor/Dockerfile

Line 24 in ae626cc

RUN apt-get update && apt-get install -y jq \

ajaits · 2026-04-01T13:43:39Z

statvar_imports/noaa_gfs/run.sh

+    build-essential gfortran cmake git libg2c-dev libaec-dev curl
+
+# Clone and build
+rm -rf wgrib2 


should we try grib2io developed by NOAA as a python module instead of building this package locally?

ajaits · 2026-04-01T13:45:10Z

statvar_imports/noaa_gfs/run.sh

+    --bucket_name="${BUCKET}" \
+    --input_local="./${FILE_NAME}.csv" \
+    --forecast_hour="${FHOUR}" \
+    --output_blob_name="noaa_gfs/${DATE_STAMP}/output/noaa_gfs_output.csv"


let's make this an incremental build.
pls add a date suffix to the GCS file name soa new file gets created per day rather than rewriting an existing file.

ajaits · 2026-04-01T13:47:43Z

statvar_imports/noaa_gfs/run.sh

+    --input_local="./${FILE_NAME}.csv" \
+    --forecast_hour="${FHOUR}" \
+    --output_blob_name="noaa_gfs/${DATE_STAMP}/output/noaa_gfs_output.csv"
+


if the python script exits, return a non-zero status code so the autorefresh monitoring can detect a failure.

(( $? > 0 )) && echo "Failed to run script for ${FILE_NAME}" && exit 1

ajaits · 2026-04-01T13:49:59Z

statvar_imports/noaa_gfs/custom_statvar_processor.py

+    if l == "mean sea level": 
+        return "0MetersAboveMeanSeaLevel"
+    if "m above mean sea level" in l:
+        val = l.split(" ")[0].replace("-", "To")
+        return f"{val}MetersAboveMeanSeaLevel"
+
+    if l == "surface": return "SurfaceLevel"
+    if "entire atmosphere" in l: return ""
+    if l == "planetary boundary layer": return "PlanetaryBoundaryLayer"
+    if "low cloud layer" in l: return "LowCloudLayer"
+    if "middle cloud layer" in l: return "MiddleCloudLayer"
+    if "high cloud layer" in l: return "HighCloudLayer"
+    if l == "0c isotherm": return "Isotherm0C"
+    if l == "highest tropospheric freezing level": return "HighestTroposphericFreezingLevel"


For the 5 conditions with '==' consider using a map.

ajaits · 2026-04-01T13:54:57Z

statvar_imports/noaa_gfs/run.sh

+
+# --- 6. Upload TMCF to GCS ---
+echo "Uploading TMCF to gs://${BUCKET}/noaa_gfs/${DATE_STAMP}/output/..."
+gsutil cp "gs://${BUCKET}/noaa_gfs/noaa_gfs_output.tmcf" "gs://${BUCKET}/noaa_gfs/${DATE_STAMP}/output/"


please keep a local copy of the TMCF checked into github and copy that to the GCS destination folder.

ajaits · 2026-04-01T14:08:13Z

statvar_imports/noaa_gfs/run.sh

+# --- 3. Convert GRIB2 to CSV ---
+echo "Converting to CSV..."
+# Call the binary using the confirmed install path
+$WGRIB2_BIN "./${FILE_NAME}" -csv "./${FILE_NAME}.csv"


wgrib2 has awarnign for converting entire file sinto csv:
https://screenshot.googleplex.com/8hWbbRqivKw7kht

Have you considered other options such as pygrib:
https://screenshot.googleplex.com/8CcyzTLTYn3WGuF

adding files related to NOA GFS poc

fee7a82

balit-raibot requested review from hareesh-ms and smarthg-gi January 4, 2026 18:13

balit-raibot self-assigned this Jan 4, 2026

gemini-code-assist bot reviewed Jan 4, 2026

View reviewed changes

balit-raibot added 11 commits January 6, 2026 11:11

resolving comments

adcf7ab

resolvingcomments

7e0150e

refined schema

069eadd

Merge branch 'master' into noa_gfs_poc

0e97f29

refined schema

5b46602

refined schema

94c3769

used existing stat var mcf file

644825a

resolving comments

6ab31c8

resolving comments

68b28b8

resolving comments

2af5ae9

resolving comments

e0bfb51

ajaits reviewed Jan 16, 2026

View reviewed changes

statvar_imports/noaa_gfs/test_data/noaa_gfs_output.csv Show resolved Hide resolved

balit-raibot added 8 commits January 19, 2026 07:54

adding custom stat var processor script

8532138

refined schema

b13690b

test_data folder creation

0a643dd

renamed parent directory

18461a0

Merge branch 'master' into noa_gfs_poc

d60df2f

modified schema file

4e3b9f6

Merge branch 'master' into noa_gfs_poc

3b7efda

Merge branch 'noa_gfs_poc' of https://github.com/balit-raibot/data in…

d50a1f3

…to noa_gfs_poc

balit-raibot requested a review from ajaits January 23, 2026 18:46

balit-raibot and others added 3 commits January 27, 2026 16:59

Merge branch 'master' into noa_gfs_poc

ab3db64

Merge branch 'master' into noa_gfs_poc

2c83c8f

adding files

d7940c5

balit-raibot added 3 commits March 2, 2026 08:30

Merge branch 'master' into noa_gfs_poc

6200509

Merge branch 'master' into noa_gfs_poc

0cbda78

Merge branch 'master' into noa_gfs_poc

e979a2d

hareesh-ms reviewed Mar 12, 2026

View reviewed changes

balit-raibot added 9 commits March 26, 2026 11:16

Merge branch 'master' into noa_gfs_poc

dbee392

Merge branch 'master' into noa_gfs_poc

ec24453

resolved comments

9d6b0a1

Merge branch 'noa_gfs_poc' of https://github.com/balit-raibot/data in…

553d0e9

…to noa_gfs_poc

added multiplier logic

d7987c2

adding run.sh

413a3ce

adding manifest

8e98ca5

modified README

ab52999

modified resources

d31da68

balit-raibot requested a review from hareesh-ms March 31, 2026 04:49

balit-raibot added 4 commits March 31, 2026 05:56

corrected custom_processor

5827272

Merge branch 'master' into noa_gfs_poc

f8ebe59

modifying measurementMethod

e258605

Merge branch 'noa_gfs_poc' of https://github.com/balit-raibot/data in…

f7a7bb4

…to noa_gfs_poc

balit-raibot force-pushed the noa_gfs_poc branch from d93478f to f7a7bb4 Compare April 1, 2026 07:09

balit-raibot added 2 commits April 1, 2026 07:41

modified test data as per new schema

e46638f

Merge branch 'master' into noa_gfs_poc

7973740

ajaits reviewed Apr 1, 2026

View reviewed changes

Conversation

balit-raibot commented Jan 4, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist bot commented Jan 4, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

ajaits Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

ajaits Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

ajaits Apr 1, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

ajaits Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

ajaits Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

ajaits Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

ajaits Apr 1, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

balit-raibot commented Jan 4, 2026 •

edited

Loading

ajaits Apr 1, 2026 •

edited

Loading