Bryan Caplan makes a post (h/t @S8mB) defending anarcho-capitalism from the criticism that private security companies would end up fighting it out until there is a winner which can rule.
The essence is that a norm would become established that such an ambition would just be insane. He makes a comparison with parliamentary democracy, where losing parties peacefully relinquish power, because nobody doubts that that is what they are supposed to do.
I think his argument is invalid, and that is based on a fundamental difference between the positions of the outvoted democrat and the security company.
The difference is that the democrat has a binary choice. He either accepts the result of the election, publicly, in which case he must step down, or he rejects it, in which case he is making it clear he is breaching the established expectations.
Private security companies in the Rothbardian sense are not forced into the same “in or out” dilemma. If one wants to protect a piece of property for one client while another wants to protect the same piece of property for another, there are infinite gradations of conflict they can resort to. The lower levels are deniable (“accidents” for example), middling levels can be justified as peaceful bargaining — “sanctions” of various kinds, and the highest levels of conflict, while still consistent with a dispute over a particular legal question, are indistinguishable from a war of conquest. The ability to vary the level of conflict in small steps allows the kind of norms that apply to democracy to be eroded or made irrelevant.
Deniability and lack of natural Schelling points are serious problems. However, they're not necessarily intractable, and you ought to look at the way that they have been handled in the absence of central sovereign authority. Historically lots of informal norms are developed and work well up to some scale. E.g., norms that work within a hunting band but become unstable when bands are under pressure to merge into a larger tribe for military security; or nomadic tribal institutions break down in a generation or two after the tribe conquers a settled agricultural civilization.
As a random example of decentralized norms to solve deniability problems, see the acceptable informal handling of nuisance strayed cattle in _Order Without Law_. Shooting a stray and putting it in the freezer is taboo, likely because allowing it opens the door to deniable thievery. Instead if you have lost patience with your neighbor's strays, you are supposed to do something like driving them someplace that doesn't benefit you, but where it will be a headache for the neighbor to recover them.
It's very hard to predict what norms will be invented, or what norms will remain stable under what conditions. E.g., I think the printing press is probably a prerequisite for the stability of norms that made parliamentary democracy usefully stable for groups as large as Industrial Revolution nation states. Even in hindsight, though, it's hard to be sure of that. In foresight it was probably impossible in 1550 even to guess that. It's fairly safe to predict, though, that as long as we're building political organizations out of humans (as opposed to e.g. silicon AIs or genetically engineered arthropods) there will be informal norms and formal mechanisms developed to work around deniability and the like. They may sometimes not work at all (much as e.g. I think the currently popular ideas of term limits and limitations on political advertising won't achieve anything like their nominal purpose). And even if they work, they may not continue to work up to a sufficiently large scale. But they might. By analogy with the printing press, today's supercheap communication and data storage seem likely to help with scaling. By analogy with gunpowder, the long-run consequences of our new military tech are likely to be hard to interpret. But the way that we are so far from the Malthusian frontier tends to make it really questionable to try to use any historical analogy to answer questions about medium-term stability.