OpenDX Forum - Threaded OpenDX w/ better memory management

gda

Forum Newbie

Got Viz?

Posts: 4

Threaded OpenDX w/ better memory management
10/22/09 at 13:02:17

Anybody interested in a threaded version of OpenDX? By replacing the multi-process model with threads, I've been able to get rid of the complex shared-memory management and removed the limitation to 2GB max memory. Although any individual data object still has to be <2GB, the cache can grow to any arbitrary cap. Also alleviates the memory-fragmentation issue.

In the process of doing the above, I've fixed what seems to be an egregious bug in cache management, too.

Back to top

IP Logged

dmoroian Full Member If you like CFD, then you probably like OpenDX! Posts: 150	Re: Threaded OpenDX w/ better memory management Reply #1 - 10/28/09 at 01:48:51 Wow, somebody still intrested in developing OpenDX! Would you mind making the sources available (full or just patches)? Dragos
Back to top	IP Logged

gda

Forum Newbie

Got Viz?

Posts: 4

Re: Threaded OpenDX w/ better memory management
Reply #2 - 11/24/09 at 11:45:41

Sorry about the delay. I've created a SourceForge repository (sourceforge.net/projects/opendx) containing my updated source files. There are no binaries yet; right now, please only use svn access:

svn co https://opendx.svn.sourceforge.net/svnroot/opendx opendx

There are quite a lot of changes here. The most significant are:

- Changed multi-process model for multi-threading. This enabled me to remove shared-memory management. Defaults to 1 thread; add '-processors n' to get more.

- 64-bit addressing. If built on a 64-bit machine, you can tell it to use a lot more memory. You still specify a memory limit (default 2GB) so that caching happens without filling all available swap space - I've been using 16GB (-memory 16000). This entailed a lot of changes, particularly in the EDF code (thanks, Marylin). Though I haven't seen any, I think there'll be problems if any individual allocation exceeds 2GB. Probably ought to throw an error.

There are a bunch of bug fixes, too. The biggest was that it was never automatically flushing the cache. When it hit the memory limit, it would recover a small bit of memory by flushing the dictionary (don't ask) and would then limp along a bit longer. Now it will flush the cache and continue. There's a lot of room to be more intelligent about cache flushing.

These are substantial changes, and I've not been able to really test them thoroughly. I've been running on a 4-core Fedora R11 system, and have also tested somewhat on cygwin. I'd appreciate whatever feedback I can get.

One outstanding problem has arisen since I've been able to run it a whole lot longer. With many more objects in the cache, there have been cache collisions - two objects, created by different modules, can get the same cache tag. This causes downstream modules to raise errors when they receive the wrong object as input, and it can look pretty odd. I've only seen it when running a somewhat complicated network in a 30K sequencer loop. I think I can fix it by implementing open hashing in the cache, but that'll take some work.

Please try it out. To build from SVN, you need to enter the root directory (trunk) and run:

aclocal
autoheader
libtoolize
automake -a
autoconf

And then in a separate object directory, run {path to trunk}/configure. I'll look in here (the opendx.org mailing lists and user forum) from time to time, but for the time being, I've creates a mailing list at SourceForge for discussion of the SF version.

Thanks,
Greg

Back to top

IP Logged

joelr OpenDX Forum Administrator Got Viz? Posts: 388	Re: Threaded OpenDX w/ better memory management Reply #3 - 11/25/09 at 04:32:33 Hey Greg, This is great. I'm sure this is going to be a huge hit (potential bugs, notwithstanding). On behalf of everyone, thanks. Joel
Back to top	IP Logged

dmoroian Full Member If you like CFD, then you probably like OpenDX! Posts: 150	Re: Threaded OpenDX w/ better memory management Reply #4 - 11/25/09 at 07:37:55 ...that everyone includes me, too! Dragos
Back to top	IP Logged

dmoroian Full Member If you like CFD, then you probably like OpenDX! Posts: 150	Re: Threaded OpenDX w/ better memory management Reply #5 - 11/30/09 at 02:39:16 Hello Greg, Unfortunately I could not complete the "configure" part . I get the following error: Quote: configure: error: cannot find macro directory `m4' Any clues? Dragos
Back to top	IP Logged

gda

Forum Newbie

Got Viz?

Posts: 4

Re: Threaded OpenDX w/ better memory management
Reply #6 - 11/30/09 at 08:39:37

Dragos, I bet this is an automake/autoconf version issue. Googling around, it looks like some versions expect there to be a 'm4' directory in the project root directory - or maybe the object root directory. Could you make try making those and seeing if it fixes it? BTW - I'm using automake 1.11 and autoconf 2.63.

Greg

Back to top

IP Logged

dmoroian Full Member If you like CFD, then you probably like OpenDX! Posts: 150	Re: Threaded OpenDX w/ better memory management Reply #7 - 12/21/09 at 11:06:33 Hi Greg, Yes it worked. I had to create an empty "m4" directory in the source directory. Sorry for the late response. Now I have to find some time to test it. Regards, Dragos
Back to top	IP Logged

cloudvis

Forum Newbie

Got Viz?

Posts: 17

Re: Threaded OpenDX w/ better memory management
Reply #8 - 02/18/10 at 11:28:51

I am definitely interested in using this, although I'll have to try to find someone to install it for me as I am a complete dummy on that end. Any news about further testing ?

I found interesting the comments about the cache collision problem that seemed to arise in this new version, because I think I have seen a similar thing happen in the old version. A couple times I have gotten a fatal run error indicating that object types don't match or that a macro cannot operate on the given object type. Upon further investigation, I find that instead of the field object that is expected, the string "red" or something similarly bizarre is being feed down the pipe. This occurred within a network that is stable, being run hundreds or thousands of times with no problems of this sort.

And about the cache not being flushed properly, I want to point out my experience that when DX does run out of memory, not even "Reset Server" works to restore it. Does anyone else concur with that observation ? I always have to take the more drastic action of Disconnect from Server to restore a peace conditon. Also I seem to recall that in (much) older versions, a proper warning was given when memory was running or had run out. Now the protocol seems to be that DX just starts acting totally weird and you have to waste a bunch of time trying to figure out what the problem is.

Lastly, could the caching problem you refer to be the cause of another Most Annoying DX Behaviors ? Certain macros (Compute likely the biggest contender) get corrupted sometimes and have to be replaced with a fresh icon. Again, often there is an misguided error about object types not being compatible, etc. The problem is not limited to this condition, but it seems to occur mostly after a syntax (human) error is typed in interactively as the operational statement, and then corrected upon discovery. Again, Reset Server does nothing to flush or fix the corruption. I have to say I am pretty sure Disconnect from Server doesn't work either. Replacing the corrupted icon in the VPE is the only thing that does the trick. This has happened so many times that I have learned to be quick to replace icons before wasting too much time trying to figure out whatever else could be wrong. Another offender has been the Image macro which has on occasion not reset the camera upon updated images (yes, Reset Camera was set), although not so much recently. Maybe that has been fixed (I would suspect indirectly since I've never heard mention of it) somewhere along the way.

Back to top

IP Logged

dmoroian

Full Member

If you like CFD,
then you probably
like OpenDX!

Posts: 150

Re: Threaded OpenDX w/ better memory management
Reply #9 - 03/09/10 at 00:35:47

Hello again,
The first superficial tests didn't bring up any abnormal behaviour, so far so good.
I don't recollect ever to encounter a cache collision problem (object types don't match error). However, I have experienced the memory limitation (due to cache or some other reason), then I was forced to restart the server.
Also I have never had the problem that couldvis mentions with the necessity of replacing a module in the VPE.

I will continue the tests with bigger meshes, and let you know about the results.

Dragos

Back to top

IP Logged

cloudvis Forum Newbie Got Viz? Posts: 17	Re: Threaded OpenDX w/ better memory management Reply #10 - 04/05/10 at 09:06:24 Not able to get this due to our firewall. Is there possibly another way to obtain it ? Pretty please ? Sharon
Back to top	IP Logged

cafmike Forum Newbie Got Viz? Posts: 1	Re: Threaded OpenDX w/ better memory management Reply #11 - 04/13/10 at 20:21:52 could you use a proxy?
Back to top	Nikon D90 Body IP Logged

balden

Junior Member

Got Viz?

Posts: 24

Re: Threaded OpenDX w/ better memory management
Reply #12 - 05/27/12 at 07:06:06

I have finally found some time to experiment with this jewel.
Million thanks to Greg "GDA" Abrams for bringing "true" 64-bit to the DX world.
I managed to compile it on OpenSuSE 12.1 x86_64 after installing all "-devel" dependencies (also including flex and bison/yacc).

Before compiling, I tried to manually apply as many OpenSuSE patches as possible (taken from the OpenSuSE source RPM package), as explained here:
http://www.opendx.org/cgi-bin/forum/YaBB.pl?num=1269792099

For some reason I had to temporarily rename '/usr/bin/yes' into '/usr/bin/was_yes' because it was messing up the 'configure' step (otherwise, when checking for M_PI in math.h, a 'yes' process was spawned and would never return).

At first I tried to follow GDA's step-by-step instructions with "aclocal, autoheader, libtoolize, etc.". Seemed to go fine but 'configure' ended up complaining about missing 'Makefile.in'. I had to go back to the trunk directory and run 'autoreconf': after that, 'configure', 'make' and 'make install' worked like a charm.

Note : I ran the 'configure' with no options (except '--prefix') and I wondered whether I should have added some special options like '--enable-smp-linux' or '--with-large-arenas' (which are listed as disabled by default).

At first sight, everything seemed to work, but unfortunately, at closer look, there is an annoyance (bug?) with the folding Motif sub-menus, such as the "Mode" sub-menu of the Image window: it is empty (actually it does not unfold and there is no arrow next to the "Mode" entry), and all keyboard shortcuts of the corresponding functions (Rotate, Pan/Zoom, and so on...) are unresponsive. Other menus and shortcuts work fine though, including Ctrl-F for resetting the view and Ctrl-V for the "View Control" box, which has a working "Mode" popup menu, so all view functions can still be accessed from there, although I find it a little cumbersome.

Same thing for the sub-menus of the "Edit" menu in the VPE. Must be an issue with Motif. I think I already saw this buggy behaviour a long time ago, but I don't remember what was triggering it. If anybody knows how to fix that, that would be most welcome since it is a big showstopper at the moment. Later, I'll try to compile the official source package with the same procedure to see if I end up with the same problem. The official OpenSuSE package does not have this problem, although the Motif part in the source code should be very similar and is compiled against the same Motif library (OpenMotif 2.3.2), so it may just be an issue with compiler options or something...

Now for some (basic) testing...
-------------------------------

I have tried to disconnect from server and start a new one with 8 GB. No problem so far but I have yet to generate some big data sets to actually fill this memory and see if it still works. Commands -> Show Memory Use in the Message window confirms that there are indeed 8 GB allocated.

I have also tried the new multi-processor mode but I don't really know how to stress it though... I have disconnected from server and started a new one with "-processors 2". The message window shows 2 workers indeed. The "ExampleSMP.net" sample seems to work fine.

Now I have to find a fix for the Motif sub-menu issues and then port this version on a bigger machine and test it with really big data and more processors. I'll try to post an update in a few days or weeks.

Thanks again GDA for the great work. Long live OpenDX.

Back to top

IP Logged

balden

Junior Member

Got Viz?

Posts: 24

Re: Threaded OpenDX w/ better memory management
Reply #13 - 05/27/12 at 08:04:48

Replying to my own post...
I solved my Motif submenu issue.

In file 'src/uipp/base/CascadeMenu.C', line 47 is commented out (in the middle of a call to 'XtVaCreateManagedWidget') and apparently shouldn't be (wasn't the case in the previous 4.4.4 source tree).

Now works flawlessly. Yay!!
And the keyboard shortcuts are back as well.

Back to top

IP Logged

balden

Junior Member

Got Viz?

Posts: 24

Re: Threaded OpenDX w/ better memory management
Reply #14 - 09/04/12 at 13:57:03

Sorry I did not update this thread sooner.

I have switched to GDA's version for a few months now and it has proved highly reliable: no single regression found wrt to 4.4.4.

Also, I have since tried the multi-threaded mode and it just works as promised.
I have been able to launch a heavy volume rendering VP on 4 threads on a Core i7 M620 (dual core with hyperthreading).
Never did that before but it was really as simple as adding a "Partition" tool after importing my data, and running DX with '-processors 4'.
The 'dxexec' process was indeed reported at 400% CPU activity, and the improvement was at least by a factor 3 (although that hyperthreading thing does not work so well usually, I did not expect such a good scale-up).

I highly encourage everybody to give it try: I even think it would deserve to be officially released as 4.4.5, or even 4.5.0 considering the amount of low-level modifications.

Back to top

IP Logged