When you don't know, then yes, the best way to find out is stop everything and add things back one thing at a time.

I can not image it is a supplement causing your symptoms to worsen but I guess if you dump them all and start over you will know for sure. My guess is that it will not be the supplements.