Aside from dirt build up, your siding and roof tend to accumulate pollen and can even grow mold, mildew, and moss. These cause visible stains that your HOA will not appreciate.
On top of that, if you let these issues go on for too long, you can end up with damage to the structure of your home.
Mold can grow under siding or shingles, and wood can warp or rot. You can avoid these issues by washing your home on a regular basis. Many HOA members hire a professional to come on a set schedule once or twice a year.