Do you want your fish swimming in poison, of course not, keeping gravel clean is just as essential in the maintenance of an aquarium as are the partial water changes, cleaning the filters and water testing.
So how do you avoid it?

Many years ago I was given a small aquarium with about 10 fish. It had an external siphon filter that hung on the side by today's standards it was very basic. I eventually gave the aquarium away.