Overselling is basically the selling of resources beyond the limits you are given. Let's take bandwidth for example, you are given 10GB of bandwidth to sell. You then sell 5GB of bandwith to one customer and 10GB of bandwidth to another customer. That would be overselling.
Overselling has a bad name in the hosting industry cos it may often encourage bad selling practices where hosts promise ridiculously huge amounts of resources that they can't afford to sell. But used wisely, overselling can be beneficial to sellers and customers alike since resources are actually used more efficiently than if there was no overselling. The problem is, "overselling wisely" is a term that many people view very cynically.
That's not a very good example. Reason is, additional transfer in that case can be paid for.A better example would be that the server is connected to a 10mbits port. The maximum theoritical transfer is 3TB for a month. If you sell more than 3TB to your customers in total (in the same server), its overselling. In short, overselling would be selling more than what is actually possible.
The host/reseller promises you more space/bandwidth than what they are allocated and thus, allowing them to put more accounts into one server as they believe that the users would not be able to use that much amount of what you are given. This might allow them to save some money from not having to invest in more space/bandwidth for the server or even maybe getting a new server and also open up the possibility of overages.