The problem with Ryan's utopia is the impact of greed. If everyone is looking out for number one, that opens the door to greed and from there nothing is off limits if it is in the interest of self preservation. For an interesting read on the subject look into Thomas Hobbes's "Leviathan". He talks about the structure of society and why he feels it is the way it is. The important part I'll summarize is this:
In a world without laws (which he calls the state of nature or the state of war), everyone is only going to be concerned with keeping themselves alive. This means that you will do anything you need to preserve your own life. If you have to murder a child to steal food from it, you would. Eventually, we all come to the concession though that it is in the interest of self preservation if we all agree to some limitations on our freedoms (I'll give up my right to kill you if you give up your right to kill me) because otherwise life is "solitary, nasty, brutish and short".
In Ryan's world, one focused on looking out only for one's own self interest might be tempted to slip back into this pattern. Is murder illegal in Rapture? Yes. But the splicers do it because once anarchy reigns it is in their best interest to do so, for their own personal gain. The same could be said for the smuggling in Rapture, and the reason Ryan's response was so violent against Fontaine. Ryan was looking out only for himself, so when someone started cutting into his pocket, his reaction was violent and merciless (as was Fontaine's counter reaction to Ryan, but in the interest of spoilers, I won't say what it was. Anyone who has played the game though knows exactly what I'm talking about).
There's more to be said about the counter to this, and I could throw in some John Locke as being the model for the counter example of Dr. Lamb and why that didn't work, but I'm not as well read on Locke, so I'll leave that up to you to research. (Suffice for now to say that Locke believes that we will only do what we need to survive, and nothing more. Suppose we found an apple tree, Locke would have us only taking as many apples as it takes for us to feed ourselves and pass out the rest of our neighbors so they can eat too. Hobbes would have us beating people with a fallen branch so we could keep the tree for ourselves.)