I just ran into that on my Parents Lake Home in Wisconsin.
It is algae blooms growing on the asphalt.
When you try to gently dust off the algae, the already loosened granules come right off with it.
There will also be multitudes of other spots that are already barren and void of any sign of the algae and any granules, because they have been washed away from the storms in the past after they have grown large enough and dislodged the granules significantly enough.
Actually, at that point, it would potentially be an insurable event, if the storms rain eroded away the protective granules surfacing.