To our knowledge the largest spatial run on ST-Sim was for 21 million cells by researchers at NCSU so I believe that your simulations with 17 million cells should be possible. What may be happening is that you are running out of memory. I know that the NCSU folks had access to powerful computers with more than 21GB of RAM. Can you be more specific about the error you are getting when you attempt to load your sclass raster? You will also need to use the 64bit version of the application for this size of landscape.