An inexpressible pity hides within love. The people who are engaged in trade, the cold wind, and the hazel grove all threaten the poet's beloved.
The object of Yeats's affection in this poem, as in all poems of this period, is Maud Gonne. Yeats takes a self-pitying note, arguing that he is wretched due to his experience of unrequited love. He cannot even experience the innocuous pleasures of nature - "the shadowy hazel grove" - without feeling his love for Gonne to be threatened.
Yeats is quite right in worrying that any number of natural or social phenonomena might take his love away from him. Maud Gonne was deeply involved in Irish nationalism, a movement for which Yeats had complicated feelings. The "folk who are buying and selling" could represent those who were negotiating Ireland's future - in Ireland of the 1890s, that was the Home Rule party, led by John Redmond. The result of these negotiations could and did affect Yeats's relationship with Gonne. Because they were unsuccessful, Gonne radicalized further, moving away from Yeats' politics.