Ok, in all humble honesty.. I don't know what's wrong for the time being, trying to look into it
But I very much doubt it's the traffic or the number of accounts. Since we have a sql database not all account are loaded into memory at once, only those online so it doesn't really matter how many in total.
I was asleep when the map server crashed again, I rebooted the computer and restarted it this morning, hoping it is better now.