I went into the qt-everywhere-opensource-src-4.6.0/mkspecs/qws (Since I'm running on Embedded Linux) directory and chose the applicable mkspec. I then changed my .pro file to this:

linux-powerpc-g++ {
CONFIG += static
LIBS += -L../iviukbrdplugin/kbddrivers \
-liviukbrddriverplugin
}

Now I'm able to compile for the desktop and for the target without having to comment stuff out in the .pro file. Thanks Again!