I'm a person whose taken a lot of lessons in music and am still studying music in college at the moment. I know a lot of you shun lessons, and that's fine, but I think you owe it to yourself to check out some of this stuff I'm about to cover and find someone around you who knows what I'm talking about to help you with it because it really is very useful.
What I'm going to discuss is Chord-Tone soloing, or how to play a scale in such a way that when you improvise you play very specifically to the harmonic structure of a particular song.
I'm going to start with explaining quickly a bunch of different scales and exercises you might want to get a handle on, but of course it's not necessary to know every scale in every position to play good music, but it is absolutely necessary if you don't want to set unnecessary limits on your playing.
I think the main scales to learn (if you plan on playing any popular form of western music) would be your blues/pentatonic scales, major scale (all modes of course but the ones I'm listing are used most often), aeolian mode (natual minor), dorian mode, mixolydian mode, and your melodic and harmonic minors. It's not necessary, but it's very useful to know all of these scales up and down the neck in every position. Borrowed chords and chord subs used in jazz progressions often come from melodic minors, and bands like radiohead modulate from a major key to a melodic minor or one of its modes in almost every single song they play.
There is also 2 diminished scales, whole tone scales, hungarian minor scales, middle eastern scales, bebop, and hindu scales that are all very interesting, but they're not used that often and don't serve much purpose in most popular music. So I would suggest focusing on your major scale and it's modes first and then moving on to melodic and harmonic minors.
Once you've learned those up and down the neck then you should explore the intervallic relationships of each scale, which would mean playing each scale up and down the neck in 3rds, 4ths, 5ths, 6ths, and 7ths. Then try playing each scale around in triads, 7th chords, 9th chords, 11th chords, and 13th chord arpeggios. Once you've done that try playing each scale on just one string, and then playing all the intervals on one string, then all the arpeggios from triad all the way to 13th chords on just one string, then try two strings, three four, five and six strings at a time. Then play all your intervals in a given scale in groups of 2 strings at a time, and finally chords in groups of 3 strings, 4 strings, 5 strings, and 6 strings.
That should take you at least a few months of serious focus to work out, and at the end of it you might think it was useless, but I'll tell you how to apply it.
Take any song you like to play, and analyze the harmonic structure, meaning figure out what chords exactly are being played. So say your progression is G, Emin, Amin, D (or I-vi-ii-V: very common progression for pop music), take the G Major Scale and play a G Major Arpeggio up and down the neck, then do the same with the Emin (play Aeolian), Amin (play Dorian), and the D (play Mixolydian). Now play the G up and down the neck in G Major Seventh Chord arpeggios, then ninths, and so on. Then do the same for each chord. Now play each chord in groups of three strings up and down the whole length of the neck.
If you actually took time to do this what you've done is mapped out all the strong notes or notes of significance for this chord progression. Now try playing the song, but instead of just holding chords and strumming, play up and down the neck using the chord tones or pieces of chords you've learned from the above, making sure to change patterns and chords with the song. Now this isn't how you're going to solo of course, this is just to get you used to playing notes that are actually inside the harmony of the music, and you may be surprised at how good just this by itself actually sounds. Now, with the song or a metronome, practice playing any note in the scale you want on beats 2 and 4 (assuming it's in 4/4), but only playing chord tones or pieces of the occurring chord on beats 1 and 3. Now do the opposite, play non-chord tones from the scale on beats 1 and 3, but play chord tones on beats 2 and 4. Practice placing chord tones at different places inside subdivisions and do the same with non-chord tones.
The point of all this is not to get you to play all this exactly when you solo (though it would probably sound a lot better then what most people tend to play when they solo), but to get you to realize where the chord tones are at all times, and where you are in relation to those tones within the rhythm of the music itself. The real goal with soloing is to get yourself to a point where you can think of an idea and play it immediately afterwards, or hopefully one day at the same time, but in music, knowing how notes we play relate to the rhythm and harmonic structure is important. Most memorable and successful melodies all interact with the harmony and rhythm very specifically, most great guitar players you listen to play notes very specific to what it is the music itself is doing. It is part of the reason why their music tends to sound so much better in comparison to a lot of other mediocre guitarists. They're doing what jazz players would call "making the changes" or playing notes specific to the music being played itself.
Hendrix, for example, though he had little training, is a player whose solos and improvisations match the harmonic structures brilliantly, he nearly always makes the changes. Not to mention the way he phrases, which is another subject completely.
Below are some links to some videos by Bill Evans (brilliant jazz pianist) doing an interview where he briefly illustrates the difference between playing randomly in a scale and playing notes very specific to the music itself. If you listen, you can here the clear and distinct difference in the sounds when he just plays fast and randomly in a scale over a progression and when he plays a focused little motif that interacts with the harmony of the chords.
Hope this has helped some of you if you actually took the time to read it.
Universal Mind of Bill Evans
http://www.youtube.com/watch?v=Jm6V7bWnVpw
http://www.youtube.com/watch?v=Ihw6K7jiAS4
http://www.youtube.com/watch?v=basa3P38BRQ
http://www.youtube.com/watch?v=rKAEMkbDXfE
http://www.youtube.com/watch?v=LRuhSrLyZOs
