Description
Add flags to select, rename, and cast columns during loading, before data reaches SQLite.
# Select specific columns only
sql-pipe data.csv --select 'name,amount' 'SELECT * FROM t'
# Rename columns during load
sql-pipe data.csv --rename 'old_name:new_name,amount:revenue' 'SELECT new_name, revenue FROM t'
# Cast column types (override inference)
sql-pipe data.csv --cast 'zip:TEXT,amount:REAL' 'SELECT * FROM t'
# Combine all three
sql-pipe data.csv --select 'name,zip,amount' --rename 'amount:revenue' --cast 'zip:TEXT' 'SELECT * FROM t'
Motivation
Type inference is imperfect — ZIP codes like "01234" get detected as INTEGER (losing the leading zero). Column renaming is needed when source data has spaces, special characters, or non-SQL-friendly names. Currently users must use CAST() in every SQL query, which is verbose and must be repeated.
Acceptance Criteria
Implementation Notes
- Intervene between header parsing and table creation in the loader pipeline
--select: filter column names before type inference and INSERT binding
--rename: apply name mapping after header parsing
--cast: override type inference results before CREATE TABLE
- The
PRAGMA table_info approach (already in getTableColumns) provides column list for validation
- Multi-character delimiters make index-based column skipping in the CSV parser more complex — consider post-filtering the record array instead
Description
Add flags to select, rename, and cast columns during loading, before data reaches SQLite.
Motivation
Type inference is imperfect — ZIP codes like "01234" get detected as INTEGER (losing the leading zero). Column renaming is needed when source data has spaces, special characters, or non-SQL-friendly names. Currently users must use
CAST()in every SQL query, which is verbose and must be repeated.Acceptance Criteria
--select <cols>flag: load only specified columns (comma-separated)--rename <mapping>flag: rename columns during load (old:newpairs, comma-separated)--cast <mapping>flag: override inferred types (col:TYPEpairs, comma-separated)--selectpreserves column order as specified--castsupports: INTEGER, REAL, TEXT, DATE--renameand--castImplementation Notes
--select: filter column names before type inference and INSERT binding--rename: apply name mapping after header parsing--cast: override type inference results beforeCREATE TABLEPRAGMA table_infoapproach (already ingetTableColumns) provides column list for validation