Memory exhausted error

[kzollove]

There is a recurring but intermittent error that appears when ingesting a shapefile into R from the database. This error seems to be somehow related to connect/disconnect pattern, as it occurs more frequently if connections are not well managed. However, it does appear after multiple consecutive spatial table imports.

More context:

• spatial table imports are done in batches of 1000 rows at a time. This was initially put in place to avoid memory problems from spatial imports of ~3000 rows.
• even with the batch import, mutliple consecutive imports (pulling the same 3k row table in 3-4 times) will cause the error.
• Since the problem is still occurring after a total of ~10000 rows are imported during a single session, it is worrisome to imagine a larger table failing to load even once.
• The problem always goes away/ memory resets when the R session is restarted. Obviously, any imported data is cleared and all packages are unloaded.

[kzollove]

Ha! This may just be on my personal computer. Steps for alleviating:

• change the java heap size that is allowed
• change the R system variable for memory allocation
  After those changes are made, does this still happen? How often and at what load size?

[kzollove]

This may or may not affect many new users.

Document the fix and close the issue.