Mark, I like your idea that this is ultimately a love poem --

What can be said between two, cannot always be said among three, and so the first two-thirds of the poem carry a tone of restraint and delicacy -- the socially acceptable tone, however intimate, of three people “making conversation.” When this is finally violated and the lover addresses the beloved directly --

but are you suggesting the speaker actually speaks to the woman at the end of the poem? For me, he's only thinking it.

even though there is nothing remarkable about what he says,

For me, what he says is pretty remarkable. Earlier, the speaker says,

It's certain there is no fine thing
Since Adam's fall but needs much labouring.

Yeats' finely crafted poem and the second woman's comment suggest that if you labor enough, you will be able to achieve that "fine thing," whether it is a poem or feminine beauty (wrought with powders, etc.) or love.

And the speaker implies he labored long "in the old high way of love," yet was unsuccessful. And that, ultimately, might be Adam's Curse -- that love does not come as easily as Milton says it did in Eden.

And then there's that last phrase -- we'd grown / As weary-hearted as that hollow moon. As Tonto once said, "who's this 'we,' kimosabe?" It seems to me the speaker in the poem is putting his own spin on their failed relationship. He's suggesting she tried as hard as he did to make the relationship work, and now they're both weary of trying. But perhaps she never really cared for him, and it's just the speaker who is weary and

worn as if it had been a shell
Washed by time's waters as they rose and fell
About the stars and broke in days and years.