There's a lot of complicating factors to consider. Lets consider the most basic: do you have access to the source code and know how to compile it?
If so, it's still hard, but doable if you have the necessary skills and code to swap the engine. Basically, you'll have to make changes to the code to harness the new engine appropriately, which likely will require a bit of understanding of physics yourself to be able to pick out where that is.
If not, you can forget about swapping out the game's physics engine, but you may be able to tweak the game's physics settings if the developers put them somewhere you can access it. Assuming they haven't hard-coded these settings, this will be in a text-based file somewhere in the game's directories, sometimes in a special archive file like a .hak or .wad that you can find expander for on the Internet. These values allow you to tweak things like the influence of gravity, the weight of various objects, momentum, ect. Of course, the engine may have limitations that you may not be able to produce ideal results no matter how much tweaking you do.
Of course, there are various legalities involved with doing either. Basically, unless they give you explicit permission to do so, you're not allowed to distribute any changes you make to their game. In fact, you probably don't want to let them know you've even been in there, lest you get slapped with some kind of reverse engineering suit.
