Caffeine is not the only thing that can affect you adversley in coffee. Decaf coffee still has caffeine in it so that's not really a good test.
Many sugar sub's can wreak havoc with the stomach and liver.
You say that you never had to give up dairy but maybe it is affecting you now. I know that my dd was on goat dairy for 2 years before we started seeing 'suspicious signs'.
I can understand you not wanting to give up coffee but maybe it's time you started thinking about it. I know it's hard to think of giving up something else. I sometimes feel the same way and have to work up to it.
Maybe just cut it out one day of the week at first and replace it with something healthy.
I ended up cutting out coffee for a year ... even though "I didn't react to it." Then when I added it back in (a couple of years ago), I really noticed a difference in how it drained my energy after about 3 days. This is why I now limit it to weekends because even though I didn't think I was reacting to it, it was still obviously not doing me any good to drink it every day. Note: I really love coffee and have found that it has become even more special since I've limited it to weekends. It is a real 'treat' now that I enjoy looking forward to all week long.
I guess it depends how badly you want to find out what's causing you distress. I know that for me, it depends on how uncomfortable I am as to whether I am willing to give stuff up. Sometimes, though there isn't much discomfort, I can work myself toward getting rid of something for a set period of time... like 6 weeks... or something like that.
You have three big offenders that you are ingesting every day. That's lots to figure out... if you only remove one, then you risk not seeing your results if you are also reacting to one of the other two.
Hope you find your answer.