Apolipoprotein ε4 (apoE4) has been strongly linked with Alzheimer's disease (AD) and contributes to several other neurological disorders. We investigated the influence of ε4 allele carrier status on the pattern of gray matter atrophy and disease severity in 51 patients with probable AD and 31 patients with behavioral variant frontotemporal dementia (bvFTD), compared with 56 healthy controls. Voxel-based morphometry was performed by using statistical parametric mapping. The ε4 allele frequency was higher in the AD group (P < 0.001) than the controls but not in the bvFTD group. No differences in demographic or cognitive profiles were observed between ε4 allele carriers and noncarriers within any of the diagnostic groups. However, ε4 carrier status was associated with more severe brain atrophy in disease-specific regions compared with noncarriers in both AD and bvFTD. AD ε4 carriers showed greater atrophy in the bilateral parietal cortex and right hippocampus, and bvFTD ε4 carriers demonstrated greater atrophy in the bilateral medial, dorsolateral, and orbital frontal cortex, anterior insula, and cingulate cortex with right predominance. This regional ε4 effect is consistent with the hypothesis that apoE may affect the morphologic expression uniquely in different neurodegenerative diseases. The atrophy patterns in ε4 carriers may indicate that they are at greater risk for clinical progression.