You shouldn't be afraid - the Welsh rugby crowd I have mixed with are a great mob to have a beer with. Rugby in Wales, however, is a working man's game traditionally played by 8ft coal miners with no teeth. Not some poncey game for public school boys.
If you are stupid enough to start a fight, however, there will be only one winner.