Really safe can be a bit predictable, like much of the new Soft Jazz... but on the other hand, going really outside can get annoying , like some of the more frantic Bebop and Freeform jazz. Where Louis Armstrong stayed close by, John Coltrane took it outside, and you'll either love him or hate him. The sixties freeform experiments chased away many of the more conservative jazz fans, and the remaining fans got stubborn. This contributed to the general fracturing of jazz, which was at one time VERY POPULAR. In my generation, jazz is a dirty word, because it became associated with the WWII generation, the guys we rebelled against. But this is unfair, because jazz was always the freer expression of music, uniquely American and vibrant. Those of us who followed Miles Davis into electric music could trace the growth of jazz under the surface. Improv never went away, it just got harder to find on the radio.
Jazz/Rock fusion explores loud sonic territory while showcasing expert players. It's like jazz, but with rock instrumentation, usually. We feature some great examples on our compilation disc In The Sam Hell. Latin/Jazz fusion is another rich category full of fun. Sometimes there is nothing in the music that sounds like "jazz", but it is still improv. The extended solos of Jimi Hendrix, the experimental sections of the Doors, the famous drum solos of the sixties, the guitar/keyboard explorations of the Allmond Brothers or the confusing tape experiments on the Beatles White album, these are all improv, but nobody would call them jazz. Dave Matthews plays improv alot, and he's considered Alternative Rock. The improv category is ready for a major expansion into acceptance. Or so we hope.
In our circle of musicians, we improv without playing "jazz" per se. Much of our improv is in the vocals, telling a story off the cuff that is either funny or insightful or (hopefully) both. The musicians are all playing at once, watching for the next twist, trying to catch enough of the vocals to react to them. This can be very difficult. Concentrating on a tricky time signature on your instrument while reacting to improv vocals is a mind-bender. It's especially hard for the vocalist playing an instrument. The music behind the improv vocals should sound cohesive, as if it were already planned. When done right, it sounds effortless.
We have been playing this kind of stuff for twenty years, and more. We've been searching out like-minded musicians. They are usually not the rich ones, or the ones with the coolest haircuts. But they are very interesting and independent people. And they can PLAY their instruments. When this music gets under your skin, commercial pop songs sound increasingly predictable.
It takes more patience to listen to improv, and it's not background music. It's not for everyone. But the recent success of bands that use some improv like Phish, Widespread Panic and DMB promises a new era of interest. Hermosa Records has the more intense version available, for those who care to explore.