It seems you want a completely new and great way of doing dialogue now.
Quite the opposite, just like it seems you want the opposite of how I read your post. I have a long wall of text below about my thoughts on dialogue systems.
tl;dr Dialogue trees suck, keywords are the only viable option unless you go all the way and create an actual RPG chatbot, which would be completely wasted on 99% of all RPGs.
Siobhan's Wall of Text
=================
Keyword-based systems are the only viable fix at this point. The holy grail of realistic dialogue supported by NLP and AI is decades away and would represent such a game changer that it's worthy of its own genre (think Facade on steroids), rather than being reduced to a mini-game in RPGs. But anything that isn't keywords and falls short of this holy grail isn't viable for anything but highly specialized RPGs:
Suppose we take a hint from detective games and have a dialogue system where questions/replies must be unlocked by combining hints. Essentially, your journal turns into a large inventory and there is a crafting system in place to produce the dialogue entries. You could even require skill checks for crafting recipes to make sure that only very intelligent and well-spoken characters can use certain powerful dialogue entries. Sounds good, right?
No, that's no good at all. You now have all the gameplay problems of point-n-click adventures and crafting systems. You finally have a combinatorial system (which I identified as a desirable trait in an earlier post) but it's a completely unprincipled system. Crafting systems aren't mechanically predictable, out of the millions of combinations the only viable ones are those that the designers implemented. There is no emergent gameplay in this system, it's just a way of obscuring a hard-coded list of answers --- but that's what player-supplied keywords already do, they obscure the list of possible answers.
So your crafting system hasn't gained you anything over keywords except more programming work and tedious clicky-clicky combination hunting for the player. In addition, once the answers have been crafted, dialogue is once again on rails, just use what you crafted at the designated points, it'll almost always be better than the default options. For otherwise the crafting system would serve no purpose.
Now you might say: okay, I'll treat the crafted answers more like items. So the NPC says something, and then the player opens their inventory to use one of their crafted replies. That's all nice and dandy, but now you have a crafting system and an inventory system, and the two combine to give you a very rudimentary combat system. One that is probably much worse than the actual combat system in your RPG. So if dialogue is just a simplistic version of your combat system, why should anybody pick a brains-character over a brawns-character?
Alright, alright, then we'll just add more complexity to the dialogue system and that fixes it, no? No! Now you have two combat systems of comparable complexity in your game, which means that all you have successfully achieved is assimilate dialogue to combat. You've sawyerized dialogue. Best case scenario: you have a dialogue-version of Shadowrun's matrix, but without the integration of meat space and cyber space combat, which is its only selling point.
That's why keywords is the sweet spot for RPGs that aren't completely dialogue focused. Anything less is too basic to be engaging and reduces the player to a passive role, anything more is in direct competition with the other RPG systems and will end up looking bad or pointless in comparison. If you want an interesting dialogue system for a general-purpose RPG, it must:
- use keywords
- allow players to enter custom keywords
- rely on exploration to give the player ideas for useful keywords
- make exploration especially viable for characters with non-combat skills; for instance, a scholar with high history and archeology skills should learn more from visiting an ancient ruin than a barbarian
- optional: provide keyword modifiers (blunt, polite, flirtatious, etc.); even more optional: lots of points in skills like etiquette and street-wise automatically pick a modifier for you
- optional: make the outcome of (possibly modified) keywords non-deterministic so that they can be affected by skill checks (e.g. how much information the NPC supplies)
- optional: use skill checks to add more or fewer entries to the list of suggested keywords
- optional: use skill checks to determine how likely the keyword parser is to accept partial matches