Your man's experience is certainly not unusual. Many guys who can get rock-hard erections with their lady at a drop of a hat find it supremely impossible to get it up during group scenarios. To be honest, it will either take some time and getting used to the swingers scene before erections happen naturally, or he may be the type who won't ever relax mentally enough with more than one other person to worry about in the bedroom. If over time it appears his penis is not going to cooperate you may just need to review how you guys play in the swinging scene. If he's happy for you to have fun with others while he saves his orgasm for you alone later, then so be it. It's still fun!
If it bothers him that he can't get an erection, then you may need to take a step backwards. It sounds like you may have gone into swinging head-on and swapped partners immediately. Why not slow things down and start with a bit of flirting with clothes on. Go out to dinner with just one other couple and get to know them. I actually don't know too many guys who can front up at a house and shag a stranger without getting to know them even just a little bit. If you decide to move things into the bedroom that night, then just concentrate on having same room, same partner sex. Take the pressure off your man to perform for others. Get him used to the idea of other people being naked in the same room but holding off from swapping contact. If his penis seems to like that idea, then you can introduce some level of swapping (kissing/touching) but again, no pressure to have intercourse. You may need to target couples who are only seeking this sort of interaction until your man gets used to playing in groups.
Other things to consider is ensuring that your partner does not drink too much, if at all, during swinging encounters. He may need some Dutch courage but ultimately alcohol will be his undoing. As will leaving playtime too late at night. Condoms are a must so maintaining an erection with a raincoat on will be his next challenge so all the more reason to start with same partner fun first with others watching.
If you have tried all that then this is the point where you may wish to consider assistance. See a GP to rule out any health issues, then get him/her to write up a box of Viagra or Cialis. No need for embarrassment as a lot of men in the scene use these magic pills from time to time. Think of them as enhancers and nothing more.