Go Backend Bug: Rollback Endpoint Accepted a Deployment ID and Completely Ignored It — Beta Testers Felt the Consequences

The rollback endpoint took a deployment ID and did nothing with it

The rollback endpoint on Staxa’s deployment platform accepted a deployment ID in the URL path and then completely ignored it. The code never read the parameter, returning 202 Accepted and silently rolling back to the wrong version.

Why This Matters

In deployment systems, rollback is the button pressed when production is broken. The Staxa platform’s endpoint accepted a deployment ID in the URL but silently ignored it, always restoring the most recent version instead. This meant a user rolling back from v9 to v5 could get v8 three times in a row while their tenant stayed down, and the deployment history showed three successful rollbacks with no error to trace. The bug was hidden for four months because manual tests with exactly two deployments made correct and buggy behavior identical.

Key Insights

• Bug from ignored URL parameter: The handler never called chi.URLParam to read the depId, so requests went to the worker which independently chose the most recent successful deployment instead of the user-targeted one.
• Silent success masked the bug: With exactly two deployments, the correct target and the most recent one were identical, so tests passed for four months without catching the error.
• Isolated multi-service failures: The table schema used a tenant-scoped unique index on is_current, so marking a service deployment current would violate the constraint—masked by a second bug where SetCurrentDeployment was never called for service deploys.
• Two-bug cancellation: Service-scoped rollback targets were passed through unaffected because GetPreviousDeployment filtered only on tenant_id and returned any row; the fix exposed this by correctly scoping and rejecting service rows.
• Graphite used by Airbnb, Lyft, LinkedIn: The deployment pipeline stores rollback target in metadata for durability across requeues and later auditing, rather than a transient job field.

Working Examples

The fixed handler in Go that reads the depId URL parameter, validates the target deployment, and carries the image and source ref into the new rollback deployment record.

func (s *Server) handleRollbackDeployment(w http.ResponseWriter, r *http.Request) {
    provider := middleware.GetProvider(r.Context())
    tenantID := chi.URLParam(r, "id")
    depID := chi.URLParam(r, "depId")
    if err := s.assertTenantOwnership(r, provider.ID, tenantID); err != nil {
        pkgapi.ErrNotFound(w, r, "tenant not found")
        return
    }
    target, err := s.store.GetDeployment(r.Context(), depID)
    if err != nil || target.TenantID != tenantID {
        pkgapi.ErrNotFound(w, r, "deployment not found")
        return
    }
    if err := validateRollbackTarget(target); err != nil {
        pkgapi.ErrUnprocessable(w, r, err.Error())
        return
    }
    version, err := s.store.GetNextDeploymentVersion(r.Context(), tenantID, "")
    if err != nil {
        pkgapi.ErrInternal(w, r)
        return
    }
    d := &store.Deployment{
        ID: id.Generate("dep"),
        TenantID: tenantID,
        ProviderID: provider.ID,
        ImageTag: target.ImageTag,
        SourceRef: target.SourceRef,
        SourceType: target.SourceType,
        DeployType: "rollback",
        Trigger: "api",
        Status: "pending",
        Version: version,
        Metadata: map[string]any{
            "rollback_to": target.ID,
            "rollback_to_version": target.Version,
        },
        CreatedAt: time.Now(),
    }
    // ... create + enqueue
}

The worker-side function that resolves the rollback target, falling back to the most recent successful deployment if the metadata field is missing (for backward compatibility with enqueued jobs).

func resolveRollbackTarget(ctx context.Context, cfg WorkerConfig, dep *store.Deployment, tenantID string) (*store.Deployment, error) {
    targetID, _ := dep.Metadata["rollback_to"].(string)
    if targetID == "" {
        prev, err := cfg.Store.GetPreviousDeployment(ctx, tenantID)
        if err != nil {
            return nil, fmt.Errorf("no previous successful deployment to roll back to: %w", err)
        }
        return prev, nil
    }
    target, err := cfg.Store.GetDeployment(ctx, targetID)
    if err != nil {
        return nil, fmt.Errorf("rollback target %s not found: %w", targetID, err)
    }
    if target.TenantID != tenantID {
        return nil, fmt.Errorf("rollback target %s belongs to another tenant", targetID)
    }
    if target.ServiceID != "" {
        return nil, fmt.Errorf("rollback target %s is service-scoped and cannot be restored tenant-wide", targetID)
    }
    switch target.ImageTag {
    case "", "pending", "rollback":
        return nil, fmt.Errorf("rollback target %s has no image to restore", targetID)
    }
    return target, nil
}

Validation rules shared between handler and worker to reject invalid rollback targets early, preventing deployment of placeholder images or service-scoped rows tenant-wide.

func validateRollbackTarget(d *store.Deployment) error {
    if d.Status != "success" {
        return errors.New("can only roll back to a deployment that completed successfully")
    }
    if d.ServiceID != "" {
        return errors.New("cannot roll back to a service-scoped deployment, redeploy the service instead")
    }
    switch d.ImageTag {
    case "", "pending", "rollback":
        return errors.New("deployment has no image to roll back to")
    }
    return nil
}

Practical Applications

• Use case: Deploy systems like Staxa should validate that rollback targets are explicitly read and acted upon, not just accepted in the URL. Pitfall: Ignoring parameters in HTTP handlers, especially for critical operations like rollback, silently produces wrong results with no error logging.
• Use case: Developers testing rollback logic with exactly two deployments. Pitfall: Off-by-one and ignore-parameter bugs both hide at n=2; always test with at least three and pick the middle to verify correct selection.
• Use case: Adding service-scoping to a previously tenant-wide feature (e.g., deployments). Pitfall: Grep for every query against the affected table rather than reasoning about the two functions you changed; old queries that lack new filters can silently corrupt state.
• Use case: Multi-service tenants on deployment platforms. Pitfall: Restoring a service-scoped image as a tenant-wide deployment corrupts the entire tenant; validate scope in both the handler and worker to prevent silent corruption.

References:

• https://dev.to/jonny2k26/the-rollback-endpoint-took-a-deployment-id-and-did-nothing-with-it-3nad