The main thing you shouldn't do is try to squeeze everything in while keeping it cozy. I once invited 8 people to my husband's birthday and left half the furniture in place, thinking it would feel more "homey." As a result, everyone was standing in the hallway, shifting from foot to foot, and the vibe was tense. Now I know: it's better to move the armchairs, chairs, and side tables to the bedroom or even the balcony than to struggle trying to fit everyone in. Space is worth more than keeping your furniture comfortable for one evening.
The second thing everyone forgets to complain about is ventilation. With 10-12 people in a one-bedroom apartment, the air gets stuffy within an hour or so, even if you aired the place out before guests arrive. Open the window as much as the noise and weather allow. Fresh air works wonders for both the mood and the guests' voices (they won't be shouting so loud).
As for the format itself - I'd put the emphasis on food. I have a few signature appetizers that I prep in advance, cut everything up, and stick in the fridge. Then I take them out, arrange them on nice boards and platters right in the room. People are constantly eating, moving around for more, standing by the table - it naturally breaks up the crowd. Plus, if the focus is on food and drinks rather than sitting and talking face to face, nobody feels the need for deep meaningful conversations. I'd put some background music on low, maybe a board game on the table. It all ends up feeling more relaxed and natural.