Custom brushes are 100x100 grey scale .png images, black applies no strength and white is full strength. You place these images in your "RustEdit install dir/Brushes" folder and they apply to both paint and terrain tools. ex.