I'm all about a whole foods paleo lifestyle, eat more healthy fat etc (and that is basically what I practice). However, I keep hearing this community say that cholesterol doesn't matter and that if you have light fluffy particles, then you are okay. This is not really accurate.

There is a study (maybe more than one for all I know) that shows that LDL size loses significance when you adjust for LDL particle number. Therefore, the particle number will be the most important risk factor in your profile, more important than your particle type. Particle type matters also, but it's secondary to the number of particles. It's the whole cars analogy- the more cars (particles) that you have on the road (arteries) the higher the incidence of accidents (atherosclerosis).

Let us know what your doctor tells you.