is there a way the point system can be set up to only allow people who receive votes to get the votes?
If you mean to reward members with points only when they vote for something? If so then yes, you just need to set points only for voting activities.
I just want the voted to get the point
Sorry I don't understand.
For example: if you are member A and some member B voted for your post, then member A will get the points. Do make it to work using some another way?
If member A votes for member B, i want it in such a way that member B gets the point only
Yes, we have two-sided actions. It means that you may set points for person who preform the vote and points for content's owner. In this case both members A and B get points or you may set points only for one of both.
See please my image below: