Fix API Pagination Duplicates and Missing Items
Pagination gaps and repeats usually come from unstable ordering, data changes between requests, or incorrect cursor handling. Reproduce the full sequence with request and item identifiers.
Record the page sequence
Capture the endpoint, filters, sort order, page size, response timestamp, request ID, next token or URL, and returned item IDs for every page. Redact credentials and private fields. Check whether duplicates appear within one page or across a boundary, and whether missing items satisfy the same filter at the time they are investigated.
Understand offset drift
With offset pagination, inserting or deleting records before the current offset shifts later rows. A client can then see an item twice or skip it entirely. Offset works well for small, relatively static administrative lists, but it does not provide a stable scan of a frequently changing dataset unless the API also defines snapshot semantics.
Use deterministic ordering
Every page must use the same total order. Sorting only by a non-unique timestamp, name, or score leaves tied rows free to change position. Add a stable unique tie-breaker, for example created_at, id, and encode both values in keyset or cursor state. Apply filters and sort direction identically on every page.
Treat cursors as opaque
Clients should pass the provider's next cursor or next-page URL unchanged. Do not decode, increment, trim, combine, or reuse it with different filters or page sizes unless the contract permits that. Stop only when the documented terminal value appears. Prevent concurrent workers from fetching the same next token and persist progress only after the current page is safely processed.
Define consistency expectations
A live collection can change during traversal. If consumers require an exact export, the API can provide a snapshot token, fixed upper bound, version, or asynchronous export. Otherwise document whether new records may appear and whether updated records can move between pages. Client-side deduplication by immutable ID prevents duplicate effects but cannot recover an item the server never returned.
Check joins and authorization filters
Database joins can duplicate one resource across matching child rows unless the query selects distinct resource keys before pagination. Authorization or tenant filters applied after pagination can create short pages and gaps. Apply visibility, filters, ordering, and pagination at a consistent query layer, then fetch related data without changing the resource set.
Test mutation boundaries
Create tied sort values and insert, update, or delete items between page requests. Verify no duplicate effects, no infinite cursor loop, deterministic replay under snapshot mode, and a clear live-mode contract. Compare redacted ID lists with the JSON Diff tool.
Frequently Asked Questions
Why can offsets skip items?
Mutations before the offset shift later rows.
Is timestamp ordering enough?
No when values tie; add a stable unique key.
Should cursors be modified?
No. Treat them as opaque provider values.