Master Layer Masking: The Foundation of Professional Compositing

If you want to composite images convincingly, you need to understand layer masks. I’m going to walk you through the techniques I use on every composite project—from simple background replacements to complex multi-layer blends.

Why Layer Masks Beat Erasing

When I started compositing, I made a common mistake: I’d use the eraser tool to remove unwanted parts of a layer. The problem? That’s destructive. Once you erase pixels, they’re gone.

Layer masks work differently. A mask is a non-destructive grayscale image that controls transparency. Black hides, white reveals, and gray creates partial transparency. This means you can refine your compositing endlessly without losing original data.

To add a layer mask, select your layer and click the Add Layer Mask button at the bottom of the Layers panel. Choose Reveal All to start with a fully visible layer. This white mask means everything shows—now you’ll paint black to hide what you don’t need.

Painting Your First Mask

Here’s where precision matters. Select the Paintbrush Tool and make sure your foreground color is black. Lower your opacity to 30-50%—this prevents harsh edges and gives you control.

The brush hardness is critical. I use a soft brush (0% hardness) for organic edges like hair or foliage. For architectural elements with defined edges, I increase hardness to 60-80%. Zoom in close—you can’t blend what you can’t see clearly.

As you paint black on your mask, you’re revealing the layer beneath. Work slowly along edges. If you over-mask, switch to white and paint to recover areas. This back-and-forth refinement is where professional composites happen.

The Refine Edge Technique

After basic masking, I always refine edges using Select and Mask. Go to Select > Select and Mask (or press Alt+Ctrl+R on Windows, Option+Cmd+R on Mac).

This workspace shows you your mask’s edge quality. Increase the Radius slider to 2-5 pixels—this softens harsh transitions. Check Decontaminate Colors if you’re working with images shot against different backgrounds; it removes color fringing from the original layer.

The Output dropdown is important: choose Layer Mask to apply refinements directly to your mask. This saves steps and keeps your workflow clean.

Blending Multiple Layers

Real compositing usually means blending three, four, or more layers. Each gets its own mask, and they work together.

Here’s my process:

  1. Stack logically—background first, then mid-ground elements, foreground last
  2. Mask from top down—start with your topmost layer and mask what overlaps
  3. Check layer blend modes—sometimes Screen, Multiply, or Overlay modes create better blends than just masking

If a masked edge looks too sharp where two elements meet, double-click your layer mask in the Layers panel, then apply a slight Gaussian Blur (Filter > Blur > Gaussian Blur, 1-3 pixels). This softens the transition without affecting the actual image.

Practical Tip: Mask Visibility

While working, toggle your mask on and off to see your progress. Alt+click (Option+click on Mac) your layer mask to view it as grayscale—this shows you exactly what’s hidden and revealed. Click again to return to normal view.

Final Thoughts

Layer masking is a skill that improves with practice. Start simple: take two photos and blend them. Once you’re comfortable with basic masks, you’ll be ready for complex multi-layer composites.

The key is patience. Zoom in, use appropriate brush settings, and refine gradually. That’s how convincing composites are built—not with big strokes, but with careful, controlled work.