Genesis 42

"Joseph then gave orders to fill their containers with grain, return each man's money to his sack, and give them provisions for their journey.  This order was carried out."

Genesis 42:25

Joseph, despite the troubles he's had with his brothers in the past, has compassion on them.  Not only does he fill their sacks with grain, but he also gives back their money.  He does all of this without them even knowing who he is!  He gives them instructions to go home and bring back their youngest brother who wasn't with them.

Since the brothers do not recognize Joseph, they are afraid when they see their money still in their bags.  Their specific fears are not specified, but my guess would be that they fear being framed for stealing the grain.  Their father Jacob refuses to let them take his youngest living son back with them, so they stay and do not return - leaving their brother Simeon in Egypt and assuming he has been killed.

As a father, I can't blame Jacob for grieving and not wanting to send any of his sons back to Egypt.  He believes that he's lost a second son now, and he can't stand the idea of losing another.  I can hardly imagine what it would be like to lose my own son... and I don't even want to think about it!