The El Niño/Southern Oscillation (ENSO) is the leading mode of interannual climate variability. However, it is unclear how ENSO has responded to external forcing, particularly orbitally induced changes in the amplitude of the seasonal cycle during the Holocene. Here we present a reconstruction of seasonal and interannual surface conditions in the tropical Pacific Ocean from a network of high-resolution coral and mollusc records that span discrete intervals of the Holocene. We identify several intervals of reduced variance in the 2 to 7 yr ENSO band that are not in phase with orbital changes in equatorial insolation, with a notable 64% reduction between 5,000 and 3,000 years ago. We compare the reconstructed ENSO variance and seasonal cycle with that simulated by nine climate models that include orbital forcing, and find that the models do not capture the timing or amplitude of ENSO variability, nor the mid-Holocene increase in seasonality seen in the observations; moreover, a simulated inverse relationship between the amplitude of the seasonal cycle and ENSO-related variance in sea surface temperatures is not found in our reconstructions. We conclude that the tropical Pacific climate is highly variable and subject to millennial scale quiescent periods. These periods harbour no simple link to orbital forcing, and are not adequately simulated by the current generation of models.