The Genetics of Mating Systems: How Who Chooses Whom Shapes Evolution

What Are Mating Systems? A Genetic Perspective

At its most basic level, a mating system describes the pattern of mating between males and females in a population. But from a genetic standpoint, mating systems determine how genes are distributed across generations, how genetic diversity is maintained, and how selection operates on heritable traits.

Population geneticists classify mating systems along several dimensions:

- Monogamy: One male mates with one female

- Polygyny: One male mates with multiple females

- Polyandry: One female mates with multiple males

- Polygynandry: Multiple males mate with multiple females

- Promiscuity: No pair bonds; multiple matings by both sexes

- Selfing: Self-fertilization (in hermaphroditic species)

But these simple categories mask tremendous complexity. In reality, most species exhibit variation in their mating systems, and individuals within populations often adopt different strategies depending on ecological conditions, their own genetic quality, and the behavior of potential mates.

The genetic consequences of these different systems are profound and far-reaching.

The Genetic Architecture of Mating Behavior

Before we can understand how mating systems affect populations, we need to understand what determines mating behavior itself. The decision of who mates with whom is shaped by a complex interplay of genes, environment, and developmental history.

Genes That Influence Mate Choice

Mate choice genes are specific genetic loci that influence preferences for particular traits in potential partners. These genes have been identified across diverse taxa, from fruit flies to humans.

Perhaps the most famous example involves the major histocompatibility complex (MHC)—a cluster of genes critical for immune function. In many vertebrates, including mice and humans, individuals prefer mates with dissimilar MHC genotypes. This preference, which can be detected through scent, serves an important genetic function: it produces offspring with more diverse immune capabilities.

In house mice, a specific region on chromosome 17 called the t-complex contains genes that dramatically influence mating behavior. Males carrying certain t-haplotypes manipulate sperm to favor their own fertilization success—a remarkable example of genetic elements "cheating" the system of Mendelian inheritance.

The Genetic Basis of Mating Strategies

Some species exhibit discrete alternative mating strategies, often genetically determined. In the ruff (Philomachus pugnax), a shorebird, males display three distinct morphs:

- Independent males with dark neck ruffs who defend territories

- Satellite males with white neck ruffs who cooperate with independents

- Faeder males that mimic females and sneak matings

These strategies are controlled by a supergene—a cluster of genes on a single chromosome that are inherited together. In ruffs, a 4.5-megabase inversion on chromosome 11 contains over 100 genes that determine these alternative phenotypes. The inversion acts as a genetic switch, creating distinct morphs that persist in populations because each strategy succeeds under different conditions.

This genetic architecture—where multiple traits are packaged together on a single chromosome—has profound implications for how mating systems evolve and diversify.

Monogamy: Genetic Consequences of Pair Bonding

Monogamy is rarer in the animal kingdom than many people assume, but when it occurs, it has distinctive genetic signatures.

True Monogamy vs. Social Monogamy

It's crucial to distinguish between social monogamy (pair bonds for raising young) and genetic monogamy (exclusive mating). Long-term studies using DNA fingerprinting have revealed a surprising truth: many socially monogamous species engage in extra-pair copulations.

Take the humble house sparrow (Passer domesticus). While these birds form socially monogamous pairs, genetic analysis reveals that 10-30% of offspring are fathered by males outside the pair bond. This pattern, known as extra-pair paternity, is widespread across birds and has significant genetic consequences.

Genetic Consequences of Monogamy

When mating is truly monogamous, several genetic effects follow:

Reduced effective population size: Monogamy reduces the variance in reproductive success between individuals, which increases the effective population size (Ne) compared to polygynous systems. This has implications for how quickly genetic diversity is lost through drift.

Slower response to selection: With more equal reproductive success among males, beneficial alleles spread more slowly through the population. This can limit the rate of adaptive evolution.

Maintenance of genetic variation: Monogamous systems, by reducing sexual selection, may help maintain greater genetic diversity at some loci compared to highly polygynous systems, where a few males dominate reproduction.

Interestingly, even socially monogamous species with extra-pair paternity show intermediate patterns. The degree of extra-pair mating essentially represents a trade-off between the benefits of genetic diversity for offspring and the costs of seeking additional mates.

Polygyny: When Few Males Father Many

Polygyny—one male mating with multiple females—represents the most common mating system across mammals and has some of the most dramatic genetic consequences.

The Genetic Signature of Polygyny

When a few males monopolize reproduction, the genetic landscape changes in predictable ways:

Skewed reproductive success: In northern elephant seals (Mirounga angustirostris), dominant males may father over 100 offspring in a single season, while many males never reproduce at all. This extreme variance in male reproductive success creates what geneticists call a high variance in reproductive success.

Reduced effective population size: The effective population size (Ne) becomes much smaller than the census size. For every dominant male that sires many offspring, the population loses genetic diversity as if it were much smaller than it actually is.

Rapid fixation of beneficial alleles: In polygynous systems, a male carrying a beneficial mutation can spread that allele through the population much faster than in monogamous systems—but harmful alleles can also spread rapidly if they're linked to traits that help males win mates.

Strong sexual selection: Polygyny intensifies sexual selection on male traits, leading to the evolution of elaborate ornaments, weaponry, and behaviors. The genes underlying these traits experience powerful directional selection.

Case Study: The Red Deer

Studies of red deer (Cervus elaphus) on the Scottish island of Rum have provided detailed insights into polygyny's genetic consequences. Dominant stags control harems of females and sire most of the calves. Genetic analysis revealed that:

- 90% of calves in some years were sired by just 2-3 males

- Male reproductive success is highly correlated with antler size and body condition

- Despite strong selection, genetic variation in fitness-related traits persists

- Environmental conditions (food availability, weather) interact with genetic potential

This persistence of genetic variation despite strong selection—a phenomenon known as the paradox of variation—remains one of the central puzzles in evolutionary genetics.

Polyandry and Promiscuity: The Female Perspective

For decades, biologists assumed that females were passive participants in mating systems, while males competed for access to them. Modern research has shattered this view, revealing the complex genetic benefits females can gain from mating with multiple partners.

Why Females Mate Multiply

The benefits of multiple mating for females are numerous and have important genetic consequences:

Genetic bet-hedging: By mating with multiple males, females increase the genetic diversity of their offspring, reducing the risk that all offspring will be poorly adapted to changing conditions.

Sperm competition and cryptic female choice: When females mate with multiple males, sperm from different males compete to fertilize eggs. Females can also influence which sperm succeed through cryptic female choice—post-copulatory mechanisms that bias paternity.

Reducing inbreeding: Multiple mating allows females to avoid mating exclusively with relatives, reducing inbreeding depression.

Material benefits: In some species, females gain direct benefits like food, protection, or access to resources by mating with multiple males.

Genetic Consequences of Polyandry

When females mate with multiple males, several genetic effects emerge:

Increased effective population size: Multiple mating increases the number of males contributing genes to the next generation, boosting Ne and maintaining more genetic diversity.

Reduced genetic load: By allowing more males to reproduce, polyandry can reduce the frequency of harmful recessive alleles in the population.

Intragenomic conflict: Multiple mating can intensify conflict between genes inherited from different parents, leading to evolutionary arms races within the genome.

Sexual selection reversal: In species where females control mating, sexual selection can operate on female traits as well as male traits, leading to more balanced evolutionary dynamics.

Inbreeding and Outbreeding: The Genetic Tightrope

One of the most critical genetic consequences of any mating system is the degree of relatedness between mates. Mating systems determine the probability of inbreeding, with profound implications for population health and evolutionary potential.

Inbreeding Depression

Inbreeding depression refers to the reduced fitness of offspring produced by related parents. This occurs because:

- Recessive deleterious alleles become homozygous (expressed) more often

- Overdominant loci lose their heterozygote advantage

- Epistatic interactions between genes are disrupted

The magnitude of inbreeding depression varies widely among species. In humans, offspring of first cousins have approximately a 3-5% increase in mortality and birth defects—a modest but real cost. In many wild populations, inbreeding depression can be severe, with inbred offspring showing dramatically reduced survival and reproduction.

Mating Systems and Inbreeding Avoidance

Many species have evolved mechanisms to avoid inbreeding, often mediated by genetic recognition systems:

MHC-based mate choice: As mentioned earlier, many vertebrates use MHC genes to identify and avoid mating with close relatives.

Kin recognition: Some species can recognize relatives through various cues, including scent, and avoid mating with them.

Sex-biased dispersal: In many mammals, one sex (often males) disperses from their natal group, reducing opportunities for mating with relatives.

Extra-pair copulations: In socially monogamous species, extra-pair mating often involves fewer related individuals than within-pair mates, serving as an inbreeding avoidance mechanism.

Conversely, some species have evolved inbreeding tolerance or even inbreeding preference under certain conditions. This occurs most often in: iii

- Species that regularly experience population bottlenecks

- Species where inbreeding allows for more efficient local adaptation

- Species where the costs of outbreeding (outbreeding depression) outweigh the benefits

Outbreeding Depression

Just as too much inbreeding can be harmful, excessive outbreeding can also reduce fitness. Outbreeding depression occurs when individuals from genetically distinct populations mate and produce offspring that are poorly adapted to local conditions or suffer from disrupted coadapted gene complexes.

This creates a genetic tightrope: populations must balance the benefits of genetic diversity against the costs of breaking up locally adapted gene combinations. Mating systems that maintain intermediate levels of gene flow often produce the highest fitness.

 Selfing vs. Outcrossing in Plants

Plants exhibit an extraordinary diversity of mating systems, from strict self-fertilization to obligate outcrossing. This diversity provides powerful insights into the genetic consequences of different mating strategies.

The Selfing Syndrome

When plants transition from outcrossing to selfing, a suite of correlated changes occurs—a phenomenon known as the selfing syndrome:

- Reduced flower size

- Loss of scent and nectar production

- Reduced pollen-to-ovule ratios

- Changes in flowering time

- Reduced genetic diversity

- Altered patterns of gene expression

These changes represent a fundamental shift in how genes are transmitted across generations. Selfing populations show:

Reduced effective population size: Selfing reduces Ne by approximately half compared to outcrossing populations.

Increased homozygosity: Selfing populations become highly homozygous, exposing recessive alleles to selection.

Purging of deleterious mutations: Some studies suggest that selfing populations can purge strongly deleterious recessive alleles, reducing their genetic load.

Reduced recombination: Selfing reduces opportunities for recombination, which can lead to the accumulation of linked deleterious mutations (Muller's ratchet).

The Maintenance of Outcrossing

Given the apparent advantages of selfing (reproductive assurance, transmission of favorable genotypes), why does outcrossing persist in so many species? Several factors maintain outcrossing:

Inbreeding depression: If inbreeding depression is severe, the fitness benefits of selfing are outweighed by the costs.

Pathogen pressure: Outcrossing produces more genetically variable offspring, which may be better able to resist rapidly evolving pathogens.

Sperm/ovule limitation: In some environments, outcrossing is limited by pollinator availability, but in others, it's highly successful.

Genetic conflict: Genes that promote outcrossing may spread because they gain transmission advantages.

Sexual Conflict and Genomic Imprinting

When the evolutionary interests of males and females diverge, sexual conflict can arise—a powerful force shaping mating systems and their genetic consequences.

Intragenomic Conflict

Some of the most fascinating genetic conflicts occur within the genome itself, where genes inherited from different parents have different evolutionary interests. Genomic imprinting—the silencing of genes based on parental origin—represents an evolutionary solution (or compromise) to this conflict.

The kinship theory of genomic imprinting proposes that imprinted genes evolve because:

- Paternally inherited genes favor increased resource extraction from mothers (benefiting offspring at some cost to mothers)

- Maternally inherited genes favor resource allocation across multiple offspring

This conflict is most intense in species with polyandry, where a female's offspring may have different fathers. In such systems, paternally expressed genes are selected to extract more resources, while maternally expressed genes are selected to restrain this extraction.

Imprinted genes are disproportionately involved in placental development, fetal growth, and maternal behavior—precisely the domains where parental conflict is expected to be most intense.

Sexual Conflict Over Mating Rate

At the behavioral level, sexual conflict often revolves around mating rate. Males typically benefit from mating with many females, while females often suffer costs from excessive mating. This conflict has driven the evolution of:

- Antiaphrodisiacs: Some male insects transfer chemicals that reduce female attractiveness to other males

- Mating plugs: Substances that physically prevent further matings

- Sperm manipulation: Strategies to displace rival sperm

- Female resistance: Morphological and behavioral adaptations to control mating

These conflicts leave genetic signatures, with genes involved in reproduction showing elevated rates of evolution (so-called "sexually antagonistic" genes).

Mating Systems and Conservation Genetics

Understanding the genetic consequences of mating systems has profound implications for conservation biology. When populations become small, changes in mating systems can accelerate extinction risk.

Allee Effects and Mate Limitation

In small populations, finding mates becomes difficult—a phenomenon known as the Allee effect. This can trigger a vicious cycle:

- Small population → difficulty finding mates → reduced reproduction

- Reduced reproduction → even smaller population → even more difficulty finding mates

This process can lead to rapid extinction, particularly in species with complex mating systems or those requiring specific conditions for breeding.

Genetic Rescue

One of the most powerful conservation tools is genetic rescue—introducing individuals from other populations to restore genetic diversity and reduce inbreeding depression. The success of genetic rescue depends critically on the mating system of the target species:

- In polygynous species, a single introduced male can have an enormous genetic impact

- In monogamous species, multiple introductions may be needed

- In selfing species, the benefits of outcrossing may be transient if selfing resumes

The Florida panther provides a famous example: after decades of inbreeding, Texas cougars were introduced, dramatically improving genetic diversity and reversing inbreeding depression. The subsequent recovery demonstrated the power of managing mating systems for conservation outcomes.

Mating Systems in Humans: A Unique Perspective

Humans present a fascinating case study in mating system genetics. While cultural practices vary enormously, human mating systems share some universal features with implications for our genetic makeup.

Human Mating System Diversity

Human populations show tremendous variation in mating practices:

- Monogamy is common across many societies

- Polygyny occurs in approximately 80% of traditional societies

- Polyandry exists in some Tibetan and Himalayan populations

- Serial monogamy characterizes many modern societies

This diversity reflects the flexibility of human behavior and the interaction between cultural and genetic evolution.

Genetic Signatures of Human Mating

Human mating systems have left detectable signatures in our genomes:

Reduced Y-chromosome diversity: The fact that Y chromosomes show less diversity than other parts of the genome suggests that, throughout human history, fewer males have reproduced than females—a signature of polygyny.

Sex-biased migration: Patterns of mitochondrial DNA (maternal lineage) and Y-chromosome (paternal lineage) variation reveal different migration patterns for males and females, reflecting cultural practices around marriage and residence.

Inbreeding avoidance: Humans show strong cultural and biological mechanisms to avoid inbreeding, including incest taboos that appear universal across cultures.

Selection on mating-related genes: Genes involved in mate choice, pair bonding, and reproduction show evidence of recent positive selection in humans.

The Evolution of Pair Bonding

The evolution of human pair bonding remains a topic of active research. Several genetic systems have been implicated:

- Oxytocin and vasopressin pathways influence pair bonding in voles and may play similar roles in humans

- Dopamine-related genes affect reward processing in social bonds

- MHC genes influence mate choice preferences

Understanding these genetic systems helps explain both our capacity for enduring relationships and the variation in human mating behavior.

The Future of Mating Systems Research

As genetic technologies advance, our understanding of mating systems continues to deepen. Several frontiers promise exciting discoveries:

Genomics of Mating Systems

Whole-genome sequencing is revealing the genetic architecture of mating system variation in unprecedented detail. Researchers can now:

- Identify the specific genes controlling mating behavior

- Track how mating systems evolve at the molecular level

- Study the genomic consequences of different mating systems across the tree of life

### Experimental Evolution

Laboratory experiments with model organisms allow researchers to directly observe how mating systems evolve under controlled conditions. By manipulating mating regimes in populations of fruit flies, beetles, or nematodes, scientists can:

- Measure the rate of adaptation under different mating systems

- Track changes in genetic diversity over generations

- Test predictions about sexual selection and genetic load

### Integrative Approaches

The future of mating systems research lies in integration: combining genomics, behavior, ecology, and evolutionary theory to build comprehensive models of how mating systems evolve and how they shape populations.

Advances in machine learning and artificial intelligence are enabling researchers to:

- Predict mating patterns from genomic data

- Model complex social interactions

- Analyze massive datasets of mating records

Conclusion: The Genetic Legacy of Who Mates with Whom

The study of mating systems reveals something profound about life on Earth: the seemingly simple question of who mates with whom has consequences that echo through entire genomes, shaping the trajectory of evolution itself.

From the selfing plants that sacrifice genetic diversity for reproductive assurance to the polygynous mammals where a few dominant males father entire generations, each mating system represents a unique solution to the fundamental challenge of reproduction. These solutions leave distinctive signatures in DNA—patterns of diversity, rates of adaptation, and the distribution of genetic variation that we can read like pages in a history book.

As we continue to unravel the genetic basis of mating behavior and its consequences, we gain not only a deeper understanding of the natural world but also practical insights for conservation, medicine, and our own species' remarkable evolutionary journey.