I always offer the leftovers to anyone who is "packing up" about the same time I am. Unless there was a huge amount of food brought that didn't get eaten, half left might be a sign that your offering wasn't well received. Feel free to take your leftovers after you offer them to anyone who would like.
As a frequent pot luck hostess, please do take your dishes when you go. Take or leave your leftovers at your preference, but there is enough to do without having to wash everybody's dishes and then make sure they get them back.