Bottomfish and Terese Svoboda, Yeats pretty cleary addresses the woman he loves, you, in the second person, from the very first sentence: "your" friend in line 2 and "you" in line 3. He also addresses her in the thought for only her ears at the end: that "you" are beautiful. The entire poem is addressed to her (though it is courtly to her friend).

Both women are "beautiful," says, but he says it about "you" in an intimate way, maybe from deeper inside? Though with a past-tense "were" that may be ambiguous? (You were beautiful as I thought it? Or, you were beautiful when I strove? My conviction is that he absolutely and completely continues to find her beautiful-- the weariness the more mysterious and powerful therefore.)

(On the other hand, he uses the same, arguably worn or perfunctory adjective, about old books! Maybe the skyscape description, the silence, lets him go into himself enough to use the adjective in a more heartfelt or urgent way?)

Mainly, for me "you" is "you" and there's only one "you," each time I say this poem to myself. The beautiful mild woman is suitable for someone else to love, but she's a bystander.