Because everything that begins must end. What causes us to suffer is not in the past or the future: it is here, now, in our memory, in our expectations. We long for timelessness, we endure the passing of time: we suffer time. Time is suffering.
Before Newton, time for humanity was the way of counting how things changed. Before him, no one had thought it possible that a time independent of things could exist. Don't take your intuitions and ideas to be 'natural': they are often the products of the ideas of audacious thinkers who came before us.