Modify Mask is a meta-operation working with raster images. It uses selected operation internally, but instead of applying it to the image, it applies the operation to the current image mask (selection).
The Modify mask operation is used for example with the Blur effect in the Blur selection command.