How to handle huge data in java

2019-08-13 02:34发布

right now, i need to load huge data from database into a vector, but when i loaded 38000 rows of data, the program throw out OutOfMemoryError exception. What can i do to handle this ?

I think there may be some memory leak in my program, good methods to detect it ?thanks

标签: java database
2楼-- · 2019-08-13 02:49

Provide more memory to your JVM (usually using -Xmx/-Xms) or don't load all the data into memory.

For many operations on huge amounts of data there are algorithms which don't need access to all of it at once. One class of such algorithms are divide and conquer algorithms.

3楼-- · 2019-08-13 02:50

If you must have all the data in memory, try caching commonly appearing objects. For example, if you are looking at employee records and they all have a job title, use a HashMap when loading the data and reuse the job titles already found. This can dramatically lower the amount of memory you're using.

Also, before you do anything, use a profiler to see where memory is being wasted, and to check if things that can be garbage collected have no references floating around. Again, String is a common example, since if for example you're using the first 10 chars of a 2000 char string, and you have used substring instead of allocating a new String, what you actually have is a reference to a char[2000] array, with two indices pointing at 0 and 10. Again, a huge memory waster.

4楼-- · 2019-08-13 02:53

You can try increasing the heap size:

 java -Xms<initial heap size> -Xmx<maximum heap size>

Default is

java -Xms32m -Xmx128m
5楼-- · 2019-08-13 02:54

You could run your code using a profiler to understand how and why the memory is being eaten up. Debug your way through the loop and watch what is being instantiated. There are any number of them; JProfiler, Java Memory Profiler, see the list of profilers here, and so forth.

6楼-- · 2019-08-13 02:54

Use a memory mapped file. Memory mapped files can basically grow as big as you want, without hitting the heap. It does require that you encode your data in a decoding-friendly way. (Like, it would make sense to reserve a fixed size for every row in your data, in order to quickly skip a number of rows.)

Preon allows you deal with that easily. It's a framework that aims to do to binary encoded data what Hibernate has done for relational databases, and JAXB/XStream/XmlBeans to XML.

7楼-- · 2019-08-13 03:01

Let your program use more memory or much better rethink the strategy. Do you really need so much data in the memory?

登录 后发表回答