Main Page
From Linux Folding@Home
Contents |
[edit] Folding@Home GPU3 Windows Client on Linux Wiki
This Wiki assumes you have a general understanding of Linux and are comfortable at the command line. Most of the commands given can be used as is with minor obvious edits, but this still may be too advanced for Linux newcomers. You should also be familiar with installing and using a console version of Folding@Home.
[edit] Warning!
The writer of this Wiki and all associated parties assume no responsibility for any damages as a result of this Wiki, code, commands, or files used. While all of the following has been tested, it has only been tested on a handful of systems by a handful of users. Please back up anything of value before starting. Do not use this on a production system where crashes, data loss, or general system slowness and instability is unacceptable. Only use this guide and associated files on a system you own or have permission to use for this purpose.
[edit] Why?
The GPU version 2 client for Folding@Home is currently only available on Windows XP and Windows Vista. The Folding@Home project has stated that a GPU client for Linux and Mac OS X will be released at some point after the Windows client has stabilized. Some dedicated Folding@Home users that have compatible high end video cards that either cannot or refuse to use Microsoft Windows still wanted to donate their GPU's. See the about GPU2 page for more information.
[edit] Should We Do This?
We do not have access to the Folding@Home source code and any reverse engineering of that code would be against their terms of use. We do not want to get on anyone's bad side and we are very dedicated to this project. We do not want to submit bad results either. Many people close to the Folding@Home project know of what we have done and they have monitored a few results sent back using the wrapper from this Wiki. The only response we received so far is that they would prefer to see us not use this as the client is still not stable. They never responded saying the results that were sent back are bad.
Given that CUDA works the same on any platform and that Nvidia confirmed they use the same source code on all platforms, we feel there is a minimal issue with the data being processed incorrectly and sending back garbage results. It also looks like the Folding client itself catches many issues pretty well on its own.
[edit] How?
The CUDA drivers for Windows and Linux accept the same commands for processing data on a GPU. A CUDA program written for Windows should be able to be recompiled for Linux with no changes to the CUDA code itself. Nvidia has made CUDA very cross-platform compatible. The only exception to this is on the Windows platform CUDA can call Directx functions and on Linux CUDA can call OpenGL functions. This does not affect how Folding@Home operates as the GPU client does not use this set of functions.
So the complexity of getting a Windows only CUDA program to run in Linux was significantly reduced due to this cross-platform compatibility. When we first tried to run F@H GPU2 in Wine we saw that the client was trying to communicate with he CUDA driver through cudart.dll (later version of the client renamed this file to nvcuda.dll). The system calls for Windows are structured differently from Linux. So the solution was to create a wrapper which would convert these system calls from the nvcuda.dll library to the cudart.so library found in the Linux CUDA toolkit. This wrapper is the nvcuda.dll.so file which is a Wine specific file.
[edit] Prerequisites
[edit] Nvidia Video Card
This only works for Nvidia users who have a CUDA compatible video card. The CUDA technology first was introduced on the Nvidia G80 based cards, GeForce 8x00, and has continued support on all later Nvidia GPU generations to date.
[edit] Nvidia Binary Driver with CUDA Support
You will need to install a driver with CUDA support and the CUDA Toolkit which can be found here:
http://developer.nvidia.com/object/cuda_3_0_downloads.html
Distro specific:
Redhat (and probably the best if your distro isn't officially supported)
If you have a 64-bit Kernel you can install the 64-bit driver. Regardless of which kernel you are using you MUST install the 32-bit Toolkit. This is a compatibility issue with Wine and the CUDA wrapper.
You must also have either be running the 195 or 256 drivers from nvidia for cuda 3.0 support.
[edit] CUDA Wrapper
[edit] Wrapper
Cuda 3.0
This wrapper has a built in delay and is designed around cuda 3.0. If you are running cuda 2.x then please see the GPU2 page.
Download the wrappers
wget http://www.linuxfah.info/wrapper-3.0/cudart/cudart.dll.so ~/.wine/drive_c/windows/system32 wget http://www.linuxfah.info/wrapper-3.0/cufft/cufft.dll.so ~/.wine/drive_c/windows/system32
Wrapper Save Location
Place wrappers in your wine's window/system32 directory and run the following commands:
ln -s ~/.wine/drive_c/windows/system32/cudart.dll.so ~/.wine/drive_c/windows/system32/cudart32_30_14.dll ln -s ~/.wine/drive_c/windows/system32/cudart.dll.so ~/.wine/drive_c/windows/system32/cudart.dll ln -s ~/.wine/drive_c/windows/system32/cudart.dll.so ~/.wine/drive_c/windows/system32/nvcuda.dll ln -s ~/.wine/drive_c/windows/system32/cufft.dll.so ~/.wine/drive_c/windows/system32/cufft32_30_14.dll ln -s ~/.wine/drive_c/windows/system32/cufft.dll.so ~/.wine/drive_c/windows/system32/cufft.dll
[edit] Linking The Toolkit
You will need to tell the system that the CUDA toolkit is installed and make sure the wrapeprs can find their required dependencies. Add /usr/local/cuda/lib to /etc/ld.so.conf and run as root
ldconfig
Then run
ldd cufft32_30_14.dll ldd cudart32_30_14.dll
If you see any "not founds" then something has been setup wrong. Commonly we have seen this
user@box:~$ ldd cudart32_30_14.dll linux-gate.so.1 => (0xffffe000) libcudart.so.3 => not found libwine.so.1 => /usr/local/lib/libwine.so.1 (0xf7e4b000) libm.so.6 => /lib32/libm.so.6 (0xf7e25000) libc.so.6 => /lib32/libc.so.6 (0xf7cd6000) libdl.so.2 => /lib32/libdl.so.2 (0xf7cd2000) /lib/ld-linux.so.2 (0xf7fac000)
If this happens then your CUDA toolkit has not been installed correctly or you installed the 64-bit version. You must install the 32-bit version of the toolkit regardless of what your system is. Remember this is independent of the Nvidia driver which can be 64-bit.
Also make sure the ldconfig method used above is the same for your distro. This method is currently used for Ubuntu 8.04.
Note: The LD_LIBRARY_PATH environment variable can be used as an alternative. Set it in your login script or before starting Wine.
user@box:~$ export LD_LIBRARY_PATH=/usr/local/cuda/lib:$LD_LIBRARY_PATH
[edit] Running The Client
Download Client
Go to the link and download the Windows Folding@Home client for the console. It should be the second Windows client option.
[edit] Removing Stanford Supplied DLLs
In your gpu3 folder make sure to remove all the dll's that Stanford bundles. That should include amdcalrt.dll, cudart32_30_14.dll, cufft32_30_14.dll, amdcalcl.dll, cudart.dll, and cufft.dll.
You can do this simply by running the following inside your gpu3 directory.
rm *.dll
[edit] Running the Client
Extract the zip file into a new folder. Open a console, go to the folder you extracted the client to, and then run the client.
nice -n 19 wine Folding@home-Win32-GPU.exe
You will probably have to use the force flags, either -forcegpu nvidia_g80 for cards below fermi or -forcegpu nvidia_fermi for fermi cards.
nice -n 19 wine Folding@home-Win32-GPU.exe -forcegpu nvidia_g80
The first time you run this you should be prompted for your user information like with the other console clients. This version will accept the same console flags such as -config or -configonly. The "nice -n 19" part is to make it easier to stop the client later as your system will respond faster to user input.
To stop the client, have the console selected and press
crtl + C
SLEEPWAIT
In the gpu3 wrapper we have built in a sleepwait just like in the gpu2 wrapper. The difference is this version can be set as an environmental variable. Meaning it can be changed on the go and by anyone without having to recompile the wrapper. The default sleepwait is 150000 nanoseconds. This is too low for most people running more than one client. However it gives the best ppd if only running a single gpu3 client and no other fah clients.
The sleepwait can be adjusted to 300000, or any other number by setting the SLEEPWAIT variable in any console,
SLEEPWAIT="300000"
300000 nanoseconds seems to be optimal if running an SMP client alongside.
Auto-Mator
We have also developed a script to automate the process of setting sleepwait. What this script does is it automatically checks what percentage of the CPU the gpu client is using. If it's over 25% it increases the sleepwait, or if it's under 8% it decreases the sleepwait. If the percentage is between 8-25 it doesn't adjust the sleepwait.
Simply place auto-mater.sh into your folding directory, first make sure it's executable then simply run it.
chmod +x auto-mator*.sh ./auto-mator*.sh
[edit] Important
Please read all of this before continuing to use the client.
Client Instability
I have been noticing the Fahcore_11.exe process still running after I stop the client. After you have stopped the client check your system monitor, or top, for running processes and kill the fahcore process if it is still running. The WU seems to start back up without an issue after doing this, but if you leave this process running and try to start GPU2 again then you will make your current WU EUE as well as all other WU's the client tries to start. The client will reach a limit eventually and freeze all further activity for 24 hours, or until the client has been restarted. Please be responsible and check for this issue so we can limit the amount of EUE's sent back.
3D desktop effects (compositing) may need to be disabled while running the client. This includes Compiz-Fusion and KDE 4 special effects or anything else like it. With a 3D desktop enabled I have seen random EUE's and Xorg reset after about ten minutes of use. This may no longer be the case with Nvidia driver 177.67. After installing this driver I have had no problems keeping KDE compositing on while running the GPU client. YMMV so keep an eye out for issues.
A switch to a login console (for example with ALT+F1), a X server shutdown or a X session logout can cause the core to hang.
System Lock Up
Only applies to the first generation wrapper.
For some reason while the client is running there are two processes taking up a large portion of processing power, Xorg and Fahcore_11.exe. The fahcore is a given but we can't seem to figure out why Xorg is using so much power. The assumption is that Xorg may be waiting on the Nvidia driver to finish what it is doing. On my dual core processor each process gets 50% CPU time or one core each. This causes the system to be unusable while the client is running. Ctrl + C to stop the client makes the system usable again. There have been reports that on Quad core processors this is not so much the case.
Another option is to edit the file
client.cfg
located in your F@H client folder. Then edit the "core" line
[core] cpuusage=100
and change it to
[core] cpuusage=50
(If you do not have this file in your F@H client folder or it doesn't contain a [core] section, run the client with the "-configonly" flag, answer the questions, and say "yes" to the question "Change advanced options?," and continue to answer all questions - this will generate a config.cfg with a [core] section). This will only let the client use up to 50% CPU power. This will increase your frame time, so in my case each 1% of a WU takes ~2 minutes to process, with the processor usage limit set to 50% it doubles to 4 minutes per 1%. You can place any percentage in that option depending on the load balancing you wish to use. On a quad core you could keep the client running on one core by setting 25%. Or you could set it to 50% and run the SMP client at the same time. I currently switch it back and forth. 100% when I am away from my computer and 50% plus one uni processor client while I am using the system. This method provides a minimal hit on system usability.
KDE 4.x notes with 1st Gen Wrapper
In the KDE 4.1.1 release with Special effects enabled, I have noticed ctrl + C is not as responsive as it was in 4.0 and 4.1, sometimes taking up to five minutes to stop the client. I have been running the client at 95% CPU usage with only a few seconds hit in performance but can now instantly stop the client with ctrl + C. I have also found my background apps function better (KOrganizer Calendar, System updates). The second generation wrapper eliminates the need for this.
[edit] Overclocking
To enable GPU overclocking, you need to add the "CoolBits" option to your X11 configuration file (for example /etc/X11/xorg.conf):
Section "Device"
Identifier "Device0"
Driver "nvidia"
VendorName "NVIDIA Corporation"
Option "CoolBits" "1"
EndSection
Once enabled will you get access to the frequency settings in the NVIDIA X Server Settings:
user@box:~$ nvidia-settings
