Data Pipeline Cost Optimization: A Decision Guide for Engineering Teams
How to identify and reduce data pipeline costs on cloud infrastructure — warehouse compute, storage, egress, and scheduling decisions that compound at scale.
Before optimizing a data pipeline, separate warehouse compute, storage, transfer, and orchestration costs. Compare spend with data volume and freshness requirements to find changes that are worth the engineering time.
Where Data Pipeline Costs Come From
Before optimizing, identify the cost drivers. Cloud data pipeline costs typically fall into four categories:
Warehouse compute — the cost of running queries, transformations, and aggregations inside the data warehouse (Snowflake, BigQuery, Redshift). This is usually the largest cost component in mature ELT stacks.
Storage — the cost of raw data, staging tables, intermediate results, and historical snapshots stored in the warehouse or object storage (S3, GCS, Azure Blob).
Data transfer and egress — moving data between regions, between cloud providers, or out of the cloud entirely. Egress charges are often invisible until they appear on the bill.
Orchestration and compute infrastructure — the cost of running orchestrators (Airflow workers, managed workflow services) and any external compute used for transformation (Spark clusters, Lambda functions).
Pull your cloud spend by service and tag before optimizing. Without knowing which category is driving the cost, you will optimize the wrong thing.
Warehouse Compute Optimization
Use incremental models, not full refreshes
Full table refreshes scan every row in the source on every run. Incremental models scan only new or changed rows. In most warehouses, query costs scale with data scanned. Incremental processing can reduce repeated work on large tables where only a small share of rows changes.
In dbt, the incremental materialization with a unique_key handles this. Set the on_schema_change behavior explicitly so schema changes are handled rather than silently ignored.
Right-size warehouse clusters
Snowflake virtual warehouses, Redshift clusters, and BigQuery slots all have a "bigger is faster, but costs more" relationship. Teams commonly over-provision. Benchmark your workloads at different sizes. A transformation that takes 12 minutes on a Small warehouse and 6 minutes on a Medium costs the same in Snowflake credits (smaller warehouse for longer). Use auto-suspend aggressively — idle warehouses drain credits with no benefit.
Partition and cluster destination tables
Queries that filter on unpartitioned large tables scan the full table. Partition tables on the most common filter column (usually a date column) so queries scan only relevant partitions. Most warehouses also support clustering or sorting within partitions for further scan reduction.
Avoid SELECT * in production queries
Columnar warehouses (BigQuery, Snowflake) charge by data scanned. SELECT * scans every column. Select only the columns your query needs. This is especially impactful on wide tables (100+ columns) where most columns are irrelevant to a given query.
Schedule heavy transformations during off-peak warehouse windows
If you are on a shared warehouse, running heavy transforms during business hours competes with analyst queries and may require a larger cluster. Schedule heavy batch transforms for nights or weekends when analyst demand is low, and use a smaller cluster.
Storage Optimization
Do not store what you do not need to retain
Raw staging tables are often retained indefinitely "just in case." Define a retention policy. Raw data that has been successfully transformed and loaded into production tables is safe to expire after 30-90 days in most cases. Implement lifecycle rules in your warehouse or object storage to automatically expire old data.
Use compressed, columnar formats for object storage
If you are staging data in S3 or GCS before loading, use Parquet or ORC rather than CSV or JSON. Columnar formats compress 5-10x better than text formats for typical data. Smaller files cost less to store and less to transfer.
Avoid unbounded historical snapshots
Some teams snapshot entire tables daily "for historical analysis." This is expensive. If you need point-in-time history, use type-2 slowly changing dimensions or the source system's change history rather than storing full table snapshots.
Egress and Transfer Optimization
Keep compute and storage in the same region
Data moving between AWS regions, GCP regions, or across cloud providers incurs egress charges that add up quickly at scale. Run your warehouse, object storage, and pipeline compute in the same region and cloud. If your sources are in another cloud, consider whether a managed connector that stages data in your cloud first is cheaper than direct cross-cloud queries.
Cache API responses for repeated ingestion
If your pipeline calls external APIs (enrichment, exchange rates, geolocation), cache responses aggressively. API calls that repeat the same request across pipeline runs waste both API quota and network transfer.
Orchestration Cost Optimization
Scale down orchestration workers during off-hours
If you are running self-hosted Airflow on EC2 or Kubernetes, use scheduled scaling to reduce worker count during periods with no scheduled pipelines. A cluster running full capacity overnight to serve two 3 AM jobs is wasteful.
Consolidate frequent small pipelines into fewer larger runs
Many small pipelines with short schedules (every 5 minutes) generate significant orchestration overhead — scheduler polling, worker provisioning, state management. If multiple small pipelines feed the same downstream consumer, evaluate whether they can be consolidated into fewer, slightly less frequent runs without violating freshness requirements.
When Cost Optimization Is Worth the Engineering Investment
Cost optimization has diminishing returns. The right time to invest engineering effort:
- When storage or compute costs are growing faster than data volume — this suggests an architectural issue (unbounded storage, unnecessary full refreshes) rather than just growth.
- When a specific pipeline or query accounts for a disproportionate share of warehouse spend — optimize the expensive outlier first.
- Before scaling a system — it is cheaper to optimize the pipeline architecture at 1 TB than at 10 TB.
Prioritize changes by expected savings, implementation cost, and operational risk.