React-native-navigation is an abstraction layer on top of the native implementations of navigation options on iOS and Android. react-navigation is a purely JS solution.
That means that you have more flexibility in what to do with a purely JS solution, and is far easier to setup. It also means that usually the react-native-navigation is smoother since it relies more heavily in the native libraries. However, I do not know how much of this is true nowadays with React Navigation 3.0 in which there are tons of performance improvements.
Documentation of react-navigation is very good in my opinion, and to my eyes is better than the documentation of react-native-navigation. It also has much bigger community.
What I mostly prefer about react-navigation, however, is that it’s not native. When I was doing full native iOS, I was missing some things I consider them very important, like the ability to jump from one screen to another which is not in the same navigation path as the one you are in. Or the ability to easily style and replace parts of the navigation components (like tab bars, toolbars, button fonts) in ways where the native implementation didn't give you too much “leverage”. react-native-navigation seems to try to replicate native behaviour — which by definition is a little bit more “strict” as to the things you can do.
As always, it all comes down to each dev’s preference and the application’s needs when it’s about choosing a navigation libary.