Over stimulation is one of the modern worlds biggest problems.
The more we have of something, the more we need to get our kicks.
Sex is no different. Just as too much porn can dull the libido to such an extent that regular sex is impossible. Too much swinging can do the same.
It is possible that he simply does not get the urge for normal sex outside of a swinging scenario.

The other problem can be testosterone. As a man ages it drops, making sex less and less exciting. In cases like this, heightened stimulation from swinging may counter the effects and provide the boost he needs to feel sensual again.

However neither of these are necessarily a reflection on what he feels for you. I think you need to look at his overall interaction with you to find out what is behind his actions, or lack thereof.

At the end of the day he may be just treading water in the relationship, in which case it might be time to move on.