Most self-defense specialists recommend the use of open-hand strikes (chops, palms, gouging, etc.) over the use of closed fists. Their logic is that such hand positions are less prone to injury from striking an unintended target (e.g. a skull rather than a nose), and that in many cases, they can cause greater damage (e.g. a chop rather than a punch to the throat).
They are correct in their logic.
However, rigorously limiting one's self-defense "hand tools" to only open-hand methods is myopic, especially once the practitioner has some experience and conditioning under his or her belt.
Limiting one's use of the fists in self-defense to those methods taught in the sport of boxing or other martial sports is equally myopic, and in fact rather dangerous.
We should approach the use of the fists in self-defense free from preconceived notions of sport and structure. The methods used in any well developed sport are optimized precisely for the rules, equipment and goals of the sport, applied to the specific athletes involved.
In self-defense, we should accept no such limitations and should keep our own goals, resources and natural abilities in mind. Let's begin with a clean slate.
Beyond the Punch: Increasing the Arsenal
The un-gloved fist is better at some things than others, and can be used in a variety of ways.
The classic bare-fisted punch, using the front of the four base knuckles of the fingers (or two or three of those knuckles, depending on whom you ask), is actually not very versatile due to its relative fragility when striking hard targets (like the skull), unless tempered by high levels of conditioning in terms of hand strength, bone density and pure dynamic hitting experience. However, it can excel at hitting softer targets, especially those requiring significant penetration, which can be difficult to achieve with open-hand strikes to most targets below the neck at various angles. Floating ribs, diaphragm, liver, kidneys and bladder are good examples of targets a punch is ideally suited to attack, from a variety of angles. The effectiveness and versatility of the punch can be greatly enhanced with the addition of a push dagger, fist load (e.g. a roll of quarters), brass knuckles or similar object (e.g. a correctly sized carabiner) and other purpose-built and improvised weapons. Just be sure to practice with such weapons full-power against dummies or targets, so that you know in advance how ergonomic and reliable the weapon is, and its limitations.
Hitting with the pinky-side edge of the fist, often called the "hammerfist," is one of the most versatile uses of the closed fist, due to its durability. It can hit nearly any target from nearly any angle with little chance of injury to the fist. This allows its use for "melee" strikes without precisely defined targets, as are often needed amidst the chaos of fighting off multiple assailants, when surprised, and/or in the dark. The classic punch is by necessity more of a precision tool (as are, to a certain degree, chops and palms as well). The hammerfist, delivered with the right combination of balance, looseness (with a quick "squeeze" at the right moment during impact) and full-body unity, can cause severe damage to harder targets such as the head, collarbones, arms, etc. A knife or any other pointed or hard blunt object held projecting from the pinky side of the fist can of course add to the utility of the basic hammerfist.
The backfist, hitting with the points of the base knuckles (as opposed to the back of the hand) is ideal for hitting point targets (e.g. the temple, mastoid process, top of cervical vertebrae) and for getting sharp penetration into shallow targets, for example the muscles of the arms. When applied with a loose and mobile body, it can come from some surprising angles that an opponent might have thought were blocked.
With adequate conditioning and experience, even the thumb side of the fist, the second knuckles (like knocking on a door) and the flat of the fist (like a slap but with the fist closed) can be applied effectively.
Beyond Sport: Developing a Combat Delivery System
Even more important than the utilized striking surfaces of the fist is the "delivery system," the method of movement and training you use to effectively deliver the tools to the targets while avoiding damage to yourself. Here again the boxing/sportive paradigm does not transfer very well to self-defense. The sportive delivery system is predicated on the equipment and rules in place to assure a relatively safe, fair and exciting contest. You don't want your self-defense to be safe (for the attackers), fair (you want every advantage) nor exciting to onlookers (you want to end things quickly with a minimum of fuss)! Your delivery system for self-defense has to be far more versatile than that of sport, as real life attacks can happen from any angle, at any distance, in any circumstances and at any time, perpetrated by any sort and size of person or group thereof. Therefore, we need:
1) Development of attributes that can give us an edge over conventionally physically superior attackers;
2) Power generation methods that are effective at all angles and directions, which do not leave us especially vulnerable in case of a miss or ineffective hit;
3) A delivery system that can support application of not only the fists but all anatomical and external weapons;
4) Hand conditioning methods that can prepare us to attack all possible targets, not just those allowed in sport with the aid of protective equipment.
Regarding #1, we will focus on developing our Balance, Looseness, Body Unity and Sensitivity.
Balance is the foundation that allows us to move and generate power effectively, where and when we need it, from any position. The ability to disrupt an attacker's balance can render even physically imposing attackers far less effective, giving us the momentary edge we need to prevail.
Looseness keeps us safer (i.e. harder to break) and allows the effective use of practically infinite angles of attack, far in excess of those limited angles typically trained in combative sport.
Body Unity training develops the habit of moving your whole skeleton to most efficiently support every movement, no matter how subtle or unconventional. It helps us maximize the potential power we can deliver and also keeps us safer and subtler in evasive movement.
Sensitivity, both tactile and subcortical (i.e. subconscious, non-thinking) visual, is necessary for self-defense because of the fast, typically close-range, unpredictable nature of real assaults. You simply cannot react and flow effectively in close-quarters, unregulated, unlimited, chaotic violence using only the conventional visual awareness skills developed by most fistic sportsmen. (Those at the very top of the combative sports have often developed some degree of sensitivity on their own or by their nature. We need to specifically train it from the very beginning.)
Regarding #2 and #3, development of external and eventually internal "dropping" or "drop-stepping" (a plyometric principle using gravity to add whip-like power and speed with no windup or chambering), combined with the attributes discussed above, allows you to hit with severe impact in any direction, at any angle, in very tight space, without ever losing your balance or overextending (even if you miss). Although in many cases it may not produce the "long" delivery of power necessary to transfer impact through the padding of a boxing glove, it is far better suited to the unpredictable chaos of real violence, either bare-handed or with hand-held weapons.
Finally, there is a world of difference between fists accustomed to boxing with gloves on versus fists that have been safely conditioned for bare-handed self-defense. We should avoid conditioning methods that over time damage the skin, bones and nerves of the hands, such as repeatedly hitting relatively unyielding targets like makiwaras and heavy bags. However, hitting and catching airborne "slambags" (specially built bean-filled leather bags), as well as dynamic contraction exercises (with and without equipment such as horse shoes and bars) that work both the contractor and extensor tendons of the hands and fingers, can over time increase the strength, toughness and bone density of the hands without causing any damage. This conditioning, combined with well honed sensitivity, looseness, dropping power and striking accuracy, allows the practitioner to use the fists in an incredibly dynamic fashion. The fists, in close cooperation with the forearms, elbows and all other anatomical weapons, can be used to strike the attacker's arms to break down his guard and disrupt his balance while the practitioner closes the distance to advantageous positions to unload a chaotic maelstrom of strikes (including all kinds of fist strikes) to all targets legal and illegal, seemingly instantaneously due to the short power and rapid reload and return allowed by drop hitting. This is aided by a key principle called "ricocheting" (not found in conventional boxing) whereby strikes bounce off one target into further targets within the same motion (enhanced by the "dropping" principle mentioned previously).
Hitting is a Part of Sensitivity
By simultaneously developing tactile sensitivity (sometimes found in advanced practitioners of internal arts such as ta chi) to dictate defensive and offensive moment-by-moment adaptation, you can facilitate truly penetrating, anti-grappling, close-in fighting. Without gloves on, your hitting in essence becomes a part of sensitivity because you feel where openings are (both yours and his) with each impact.
By this point it is of course obvious that "clinching" in conventional boxing can be abandoned totally since it serves merely as a sportive safety measure to prevent both dirty maneuvers and grappling. Grappling, by the way, has its own limitations for saving your life since it locks you to a single dangerous attacker who may have accomplices quite willing to kick your brains in. To survive, the goal is to cause maximum damage in the shortest time--and escape.