The Mac-native version of LMMS is quite unstable and doesn't support VSTs or AUs. While I'm sure that a better version will eventually come around, as a quick fix I've decided to put the Windows version in Wine. LMMS is distributed under the GNU General Public License.
To install VSTs:
Method 1 - EXE Installer
Right click the application and choose Show Package Contents
Open Wineskin and choose Install Software, and choose the setup executable.
Install the VST to C:\Program Files\VSTPlugins (Create this folder if necessary)
Method 2 - DLL
Right click the application and choose Show Package Contents
Go to this folder and put the .dll file in it.
To install VSTs:
Method 1 - EXE Installer
Right click the application and choose Show Package Contents
Open Wineskin and choose Install Software, and choose the setup executable.
Install the VST to C:\Program Files\VSTPlugins (Create this folder if necessary)
Method 2 - DLL
Right click the application and choose Show Package Contents
Go to this folder and put the .dll file in it.