I think that's an unfair characterisation of us.
Millwall was always rough and ready - the club is based in the London docklands and right up to the late 70s a significant number of the crowd were dockers. It was the sort of place where if you stepped out of line you got a right-hander. Before football violence became a culture, it was never a nasty / poisonous type of violence (bare with me here), it was more like some cocky away fans mouthing off on the Halfway and getting a slap and then slung out. The dockers never went looking for trouble or tried picking on people (they were almost a cliche of that old working class image, ie hated bullies, loved their mms, but liked a scrap).
And as for the hooligan gangs that started out in the late 1960s, in all my time of following Millwall I have only ever seen them attack & fight other like minded individuals.
Yes, we do have some scumbag hangers-on that have come out for big games & tried living up to the Millwall 'rep' by being as obnoxious as possible - but I certainly do not consider them to be part of our bread and butter support.