On Unix systems, you may experience errors like the following. This document explains why this happens and how to solve this.
Not enough space in forkAndExec is normally caused by insufficient swap space. In Unix, the way Java launches a new process is first by forking the current process and then exec-ing. Therefore, for a brief period between fork and exec, there's temporarily two instances of your application server running on your system. If the application server you are running Hudson in is using 1GB virtual memory, this means you temporarily need 2GB of virtual memory, meaning more swap space.
If your Hudson is really busy and it launches a lot of processes, in theory you could be temporarily running more than 2 instances of your application server.
There are many ways to check the usage of swap space, but I usually use the "top" command, which is available on most Unix systems.
Since the issue is the lack of the virtual memory, you can resolve this issue by adding more swap space to your system. How you do this depends on OS.
First create a file to be used as a swap space, then add it as the swap. You can finally verify that the new file is active by listing all active swap files.
You will have to manually edit /etc/vfstab to make the newly added swapfile survive reboots.
First create a file to be used as a swap space, then format it properly, and finally add it as the swap. The total size of the new swap file is bs*count (in the example below it'll create a 1G swap)
You will have to manually edit the appropriate file on GNU/Linux to make the newly added swapfile survive reboots.