Randians and other libertarian types have come up with a wide range of creative solutions to what appear to be intractable problems. Yours is a typical one that has been explored many times. Here's my shot at the Randian approach.

The rest of the group would exercise their own property rights and try to come up with a way to persuade Bob to stop. Some options:

The owner of the road to Bob's property (let's assume that Bob does not own his own road) could see that the value of his road will drop precipitously if Bob becomes his only customer. In the interest of maintaining a wide customer base, he could refuse to transport Bob's supplies.

Other vendors, such as the local grocer, could do the same. Residents could threaten a boycott if the vendors do not see the wisdom of cutting Bob off on their own.

Similarly, they could hire Bob's workers away from him, making his business untenable.

The other residents could pay Bob to cease and desist. Bob may not be reasonable, of course, and may not take a fair price. They could also pay him to install noise barriers, exhaust treatment systems, or other measures that would benefit them.

The other residents could find other ways to make Bob's life unpleasant without violating his rights. They could refuse to interact with him at all, pay his customers to switch to other suppliers, or go into competition with him.

If Bob can still make money even under all of this pressure, then it is right that a tannery should be there, and not a bunch of houses. The rightness is established by the capacity to make money. Your assumption that the existing houses are the right state of affairs is faulty. There is no guarantee that any given asset will always retain value; heroic workers must use their innate abilities to achieve what they desire, and never take for granted that others will not do the same.