I think that many times it is not only a matter of what you ingest, or are exposed to, but how much.

For instance almonds contain cyanide which is a lethal poison, but you can safely eat almonds. This is because it is a very small amount.

water is not considered a dangerous substance, but you can die from drinking abnormal amounts of it, because it will lead to significant alteration in your electrolyte balance.

It also depends on what the food does-if it affects a metabolic pathway, or activates the immune system and how.

For instance, patients with celiac can not be exposed to even a very small amount of gluten, whereas patients with milk intolerance can have a small amount of milk containing products with no ill effects.