A pleasure to speak about words and their sounds with someone from New Jersey, the Best State. And thank you, OneArt, for your high-grade attention here.
Part of the pleasure, part of the interest and play in poetry, is the interplay of accidents-- the sounds that have become attached to meanings in a given language at a given time-- and intentions or meanings. A poem like Landor's arranges the somewhat accidental or arbitrary grunts of meaning, so that they become part of the meaning, part of the feeling.
Which each reader will perceive a little differently. To my ear, in a poem so rich in like sounds of vowel and consonant that it could get too rich, it is pleasing--and expressive-- that "Time" contains vowels and consonants unlike the other words.
At a certain point, there's more to a work of art than anyone can talk about. I hope I am not going beyond that point! But "Time"-- primary concept in the poem, and first word in its clinching line-- stands out in a way that is appropriate.
In writing (and probably dance, music, film), the texture can get too rich, and clog or cloy a bit. In this case, the word that is "harsh" (as you say) is appropriately so.