Maybe looking at some of the details would shed some light on the scope of such an effort. (Is such an effort worth making? Separate question, addressed later. ;-)
First of all, mobs in the current batch of MMOGs already seem to be implemented as simple state machines. They just don't have many states:
- critter mob behaviors: move, fight, flight (and maybe stalk)
- NPC mob behaviors: move, fight, converse, give quest/reward, trade
Here are some possibilities for the behaviors of orcs in an orc camp:
- internal states: curious, expansive, normal, alerted, panicked, defensive-group, defensive-self, enraged, losing, terrified
- behaviors: explore, build, fiddle-with, fortify, broadcast-alert, defend-friend, defend-self, attack-nearest-enemy, lay-traps-and-retreat, flee
- interactions: send orc runners to other allied orc camps ("we're being attacked!"), mass for counterattack, counterattack nearest non-allied (orc or other enemy) population center
- roles: warrior, maintainer, builder, communicator, leader
Even with such tools, however, implementing more complex mob behavior still carries some risks:
- feedback loops -- positive feedback (in the technical sense) can be destructive
- player metagaming of complex systems
- potentially large storage requirements for retaining "state" for many mobs
- potentially significant processing power required for detecting state change conditions for many mobs
Systems-level thinking is the only hope for making something like this work. I can't imagine successfully approaching it as an isolated game feature.
Finally, there's that pesky question of whether the current very Achiever-oriented population of gamers actually wants a more dynamic world in their game. "Unwanted outcomes" look good to Explorer-types like me, but someone whose satisfaction depends on having a controlled gameplay experience is not likely to agree.