32 GB RAM, I think changing lightmap scale to 128 works, only issue now is it's unplayable in fullbright because it's blinding and is horrible to look at. I assume this wouldn't be an issue though when I finally add lighting to the map and it's no longer fullbright.
I had a similar issue, but I compiled with lighting.
Seems strange to me; if you don't have a light_environment entity I assumed that the game just didn't calculate lighting at all?
Regardless from what I gathered from talking to others/googling the problem, the issue arises when your map's lightmap data is too large. This can be caused by huge displacements and just generally a very large/open map. There are console commands to get around this(type r_hunkalloclightmaps 0 in the console; it's set to 1 by default), but the main solution is to lower the information required for rendering the lighting.
This is a common problem with Garry's Mod maps, as a lot of them are very big. Garry's mod has r_hunkalloclightmaps automatically set to 1 to counteract this problem. I don't really know why this isn't the default normally if it fixes this particular problem, but maybe it has something to do with performance or something.