Payload Optimization and Response Shaping

The Symptom

The product listing page calls GET /api/products?category=electronics and receives a 42KB JSON response. The product listing component uses 6 fields per product: id, name, price, thumbnailUrl, category, and inStock. The API response includes 28 fields per product, including description (avg 800 bytes), specifications (avg 1,200 bytes), reviews (avg 2,400 bytes), relatedProducts, seoMetadata, and internalNotes.

22 fields, representing 78% of the payload, are fetched and immediately discarded by the frontend.

The Cause

APIs designed for reuse serve every field every consumer might need. The product detail page needs description, specifications, and reviews. The product listing page does not. But both call the same endpoint with the same response shape, because the API was designed as a generic resource endpoint, not a purpose-built data contract.

This is over-fetching. The network cost is paid by every user on every product listing page load. On 4G, 42KB takes 210ms to download. If the response only contained the 6 needed fields, it would be 9.2KB, taking 46ms. The difference is 164ms of unnecessary download time, multiplied by every page load.

The Baseline

Product listing API payload audit:

Field	Size (avg per product)	Used by Listing?	Used by Detail?
id	36 bytes	Yes	Yes
name	42 bytes	Yes	Yes
price	8 bytes	Yes	Yes
thumbnailUrl	64 bytes	Yes	Yes
category	18 bytes	Yes	Yes
inStock	5 bytes	Yes	Yes
description	800 bytes	No	Yes
specifications	1,200 bytes	No	Yes
reviews	2,400 bytes	No	Yes
relatedProducts	640 bytes	No	Yes
seoMetadata	320 bytes	No	No (server only)
internalNotes	180 bytes	No	No
… (16 more)	~1,800 bytes	No	Various

Per-product payload: 7,513 bytes total, 173 bytes needed for listing. For 20 products: 150KB total, 3.5KB needed. Over-fetch ratio: 43x.

The Fix

Sparse Fieldsets in REST

Add field selection support to the API:

// Server: Express route with field selection
import type { Request, Response } from "express";

const ALLOWED_FIELDS = new Set([
  "id",
  "name",
  "price",
  "thumbnailUrl",
  "category",
  "inStock",
  "description",
  "specifications",
  "reviews",
  "relatedProducts",
]);

function parseFields(fieldsParam: string | undefined): Set<string> | null {
  if (!fieldsParam) return null; // No filtering, return all

  const requested = fieldsParam.split(",").map((f) => f.trim());
  const validated = requested.filter((f) => ALLOWED_FIELDS.has(f));

  if (validated.length === 0) return null;
  return new Set(validated);
}

function filterObject<T extends Record<string, unknown>>(
  obj: T,
  fields: Set<string> | null,
): Partial<T> {
  if (!fields) return obj;

  const result: Record<string, unknown> = {};
  for (const field of fields) {
    if (field in obj) {
      result[field] = obj[field as keyof T];
    }
  }
  return result as Partial<T>;
}

async function getProducts(req: Request, res: Response): Promise<void> {
  const fields = parseFields(req.query.fields as string | undefined);
  const products = await productService.getAll();

  const filtered = products.map((p) => filterObject(p, fields));
  res.json(filtered);
}

// Client:
// GET /api/products?fields=id,name,price,thumbnailUrl,category,inStock
// Response: 9.2KB instead of 42KB

The ALLOWED_FIELDS set prevents arbitrary field access. Without this validation, a client could request internal fields like internalNotes or trigger expensive computed fields. The server controls which fields are selectable.

Client-side typed fetch

// SLOW: Fetch full payload, use 6 fields
interface ProductFull {
  id: string;
  name: string;
  price: number;
  thumbnailUrl: string;
  category: string;
  inStock: boolean;
  description: string;
  specifications: Record<string, string>;
  reviews: Review[];
  // ... 18 more fields
}

const products: ProductFull[] = await fetch("/api/products").then((r) =>
  r.json(),
);

// FAST: Fetch only needed fields
interface ProductListing {
  id: string;
  name: string;
  price: number;
  thumbnailUrl: string;
  category: string;
  inStock: boolean;
}

const LISTING_FIELDS = "id,name,price,thumbnailUrl,category,inStock";

const products: ProductListing[] = await fetch(
  `/api/products?fields=${LISTING_FIELDS}`,
).then((r) => r.json());

The ProductListing type is a strict subset of the full product type. The TypeScript compiler ensures the client code does not access fields that were not requested. If a developer adds product.description to the listing component, the type error is caught at compile time.

JSON Serialization Optimization

For large arrays of objects with repeated keys, the JSON envelope overhead is significant. Each product object repeats the key names ("id":, "name":, "price":, etc.). For 20 products with 6 fields, the keys are repeated 120 times.

A columnar response format eliminates key repetition:

// Standard JSON: keys repeated per object (9.2KB for 20 products)
[
  { "id": "SKU-001", "name": "Widget A", "price": 29.99, ... },
  { "id": "SKU-002", "name": "Widget B", "price": 34.99, ... },
  ...
]

// Columnar JSON: keys listed once (6.8KB for 20 products)
{
  "columns": ["id", "name", "price", "thumbnailUrl", "category", "inStock"],
  "rows": [
    ["SKU-001", "Widget A", 29.99, "/img/sku001.avif", "electronics", true],
    ["SKU-002", "Widget B", 34.99, "/img/sku002.avif", "electronics", true],
    ...
  ]
}

The columnar format saves 26% on this payload (6.8KB vs 9.2KB). The saving increases with more rows. The client reconstructs objects:

interface ColumnarResponse<T> {
  columns: (keyof T)[];
  rows: unknown[][];
}

function fromColumnar<T>(response: ColumnarResponse<T>): T[] {
  return response.rows.map((row) => {
    const obj = {} as Record<string, unknown>;
    for (let i = 0; i < response.columns.length; i++) {
      obj[response.columns[i] as string] = row[i];
    }
    return obj as T;
  });
}

The Proof

Metric	Full Payload	Sparse Fields	Sparse + Columnar
Response size	42 kB	9.2 kB	6.8 kB
Download time (4G)	210ms	46ms	34ms
JSON parse time	12ms	3ms	2ms + 1ms reconstruct
LCP contribution	+210ms	+46ms	+34ms

The LCP impact comes from the API response being on the critical rendering path. The product listing cannot render until the API response is received and parsed. Reducing the response from 42KB to 6.8KB saves 176ms of download time on 4G.

Over 200,000 daily product listing page loads, the bandwidth saving: (42KB - 6.8KB) * 200,000 = 7.04GB per day. At CDN bandwidth pricing of $0.08/GB, that is $0.56/day in bandwidth cost savings, not including the user experience improvement.

The CI Lighthouse gate’s resource size assertion catches payload growth. If a developer adds a field to the sparse fieldset without updating the budget, the total transfer size increases and triggers a warning.

The Trade-off

Sparse fieldsets add server-side complexity. The API must parse, validate, and apply field filters on every request. For the e-commerce platform’s product endpoint handling 500 requests/second, the field filtering adds <1ms of processing time, which is negligible.

The columnar format sacrifices readability. Debugging API responses in Chrome DevTools becomes harder because the data is not in a natural object shape. The standard JSON format should be the default, with columnar format available as an opt-in for high-volume endpoints where the size saving is material.

Cache key impact: ?fields=id,name,price and ?fields=id,name,price,category are different cache keys. If the CDN caches responses with field parameters, each unique combination creates a separate cache entry. For the e-commerce platform with two standard field sets (listing and detail), this means two cache entries per product endpoint, which is manageable. For APIs with arbitrary field combinations, the cache fragmentation defeats CDN caching.