Errors Showing up with High Memory Use Programs

The following stems from a posting originally on the REALbasic Users Mailing List and I thought it was worth copying and publishing when I found myself writing the same answer to someone else recently.

RB2007 exercises your memory and virtual memory more than most programs used by casual users, other than video or image manipulation.

It is possible therefore that RB is hitting memory errors that your normal use of that machine doesn't - I suggest you check your memory, with the Apple Hardware Test (from install DVD) or something third-party like TechTool Pro.

Massive Swapfile Allocations

Check in /var/vm as to the number of swapfiles being created. If you have very little free space then maybe the OS is having trouble creating them as RB requires.

Note that if you look at VM in a tool like the Activity Monitor (applications/utilities) then you will see a very large figure for VM, eg: nearly 8GB on my machine at present. For a lot of programs, the bulk of their VM size is code mapping and so doesn't require an allocation on disk - it maps read-only segments directly from the code on disk. However, when RB runs up to huge VM allocations that seems to be writable memory - running RB is the only time I have ended up with 5 swapfiles allocated - a total of 1.25GB actual swap space in addition to the 1.25GB in RAM.

Swapfile Fragmentation on OS/X

I have read in my exploration of VM on OS/X that the OS refuses to allocate fragmented swapfiles, that is not actually the case and I just proved it.

For some reason, RB, deemed my plugins needed recompiling so I left the Mac going for the several hours it took to do so for PEF, Mach-O and Windows. I have MBS and Einhugur suites installed.

On my 1.25GB RAM PPC, my /var/vm folder went up to 8 swapfiles, a total of 2.5GB allocated, increasing mostly in a few minutes from the 128MB allocated prior to RB starting the plugin prep.

andeo-mac:/var/vm andydent$ ls -al
total 5242880
drwxr-xr-x   11 root  wheel        374 Jan 27 17:20 .
drwxr-xr-x   24 root  wheel        816 Jan 25 19:24 ..
drwx--x--x   14 root  wheel        476 Jan 25 17:22 app_profile
-rw------T    1 root  wheel   67108864 Jan 25 17:21 swapfile0
-rw------T    1 root  wheel   67108864 Jan 26 12:13 swapfile1
-rw------T    1 root  wheel  134217728 Jan 27 16:12 swapfile2
-rw------T    1 root  wheel  268435456 Jan 27 16:12 swapfile3
-rw------T    1 root  wheel  536870912 Jan 27 16:13 swapfile4
-rw------T    1 root  wheel  536870912 Jan 27 16:15 swapfile5
-rw------T    1 root  wheel  536870912 Jan 27 16:33 swapfile6
-rw------T    1 root  wheel  536870912 Jan 27 17:10 swapfile7

Using hfsdebug (*1), I found that the last four of these swapfiles were in two extents, rather than being contiguous. Previously, when I was very low on disk space and the free space was extremely fragmented, I had seen iDefrag report a massive number of extents for the swapfiles and my machine became utterly bogged down.

Something which is quite fascinating and also contrary to common web reports - the last three of these swapfiles later vanished after I had quit RB. Everything I've read says that OS/X won't reclaim swapfiles until you reboot. In this case it appears the only thing with a claim on those files was RB and so OS/X did in fact remove them when they weren't needed.

During the plugin preparation time, and the relatively small amount of time building the app, I continued trying to use the computer for some mild browsing activity. I  was running the Activity Monitor at the time on another window and could see that the machine was flat out doing IO and with a massive amount of paging activity but certainly not CPU-bound.

I know from VMS experience (*2) that page thrashing can cause a normal user-priority process to bring a multi-tasking OS to its knees and it appears that OS/X is certainly not immune. This behaviour was like the worst I've experienced with Visual Studio compiling whilst trying to use Windows (I have no RB200x comparison possible on Windows as haven't bothered buying a Windows license for it).

These results actually give me some hope that RS can do something to improve the problem - apart from the possibility that there is a significant memory leak whilst compiling there is also the hope they can improve the locality of references made by the compiler (written in c++ so they have more control over such niceties).

As I later encountered in April, 2009, you can end up with horrifically fragmented swapfiles



hfsdebug comes from

It allows you to quickly check the fragmentation of a single file, rather than using something like iDefrag which scans the entire disk.
(it also does many other reports).

andeo-mac:/var/vm andydent$ sudo hfsdebug swapfile4
 path                 = MacintoshHD:/private/var/vm/swapfile4
# Catalog File Record
 type                 = file
 file ID              = 5153585
 flags                = 0000000000000010
                      . File has a thread record in the catalog.
 reserved1            = 0
 createDate           = Sat Jan 27 16:13:20 2007
 contentModDate       = Sat Jan 27 16:13:21 2007
 attributeModDate     = Sat Jan 27 16:13:21 2007
 accessDate           = Sat Jan 27 16:13:20 2007
 backupDate           = Fri Jan  1 00:00:00 1904
 # BSD Info
 ownerID              = 0 (root)
 groupID              = 0 (wheel)
 adminFlags           = 00000000
 ownerFlags           = 00000000
 fileMode             = -rw------t
 linkCount            = 0
 textEncoding         = 0
 attrBlocks           = 0
 # Finder Info
 fdType               = 0
 fdCreator            = 0
 fdFlags              = 0000000000000000
 fdLocation           = (v = 0, h = 0)
 opaque               = 0
 # Data Fork
 logicalSize          = 536870912 bytes
 totalBlocks          = 131072
 fork temperature     = no HFC record in B-Tree
 clumpSize            = 0
 extents              =   startBlock   blockCount      % of file
                            0x262e24      0x18662        76.25 %
                            0x463c2e       0x799e        23.75 %
                        131072 allocation blocks in 2 extents total.
                        65536.00 allocation blocks per extent on an average.
 # Resource Fork
 logicalSize          = 0 bytes

My First Experience with VM Thrashing

When I was a VMS sysadmin, about 20 years ago, we had a scientific programmer write some COBOL code. She didn't realise that FORTRAN accesses its arrays in the opposite order to COBOL and so her manipulation in arrays of about 4MB of data caused massive amounts of thrashing of pages as her code was having to page in memory every few instructions due to extremely poor locality of reference.

It brought an 80-user machine to its knees for about 2 hours as the code tried to do a series of filters and sorts in-memory. Nothing I could do with permissions or job priority control would stop this - I had to rewrite the program. The replacement as a pipeline using a series of simple filter programs and use of the Sort utility program, all on sequential files on disk, ran in a few seconds.